SHA-1 and SHA-2 share the same structure and mathematical operation as their predecessors - SHA-0 and MD5. Both SHA-0 and MD5 have been broken. This is one of the main reasons why SHA-1 is considered insecure and is being phased out for SHA-2.
NIST announced in 2007 a competition to produce the next generation NIST hash function: SHA-3.
What is the major difference in structure and mathematical operation between the previous SHA-1 and SHA-2 functions and SHA-3?
They are all hash functions. Apart from that, they are structurally quite different. The SHA family (SHA-0, SHA-1, and the SHA-2 functions such as SHA-256 and SHA-512) use the Merkle-Damgård construction, around an internal permutation which happens to be an extended Feistel network. Low-level primitives include boolean bitwise operations, and addition over 32-bit or 64-bit integers.
In SHA3, the structure is that of a sponge function, which is quite different. Moreover, the inner permutation is not a Feistel network; low-level primitives are boolean bitwise operations over 64-bit words, but not additions. This avoidance of operations that involve carry propagation helps a lot with performance on hardware implementations (FPGA, ASIC).
The difference is: All SHA-0, 1 & 2 and MD5 come under a class of algorithm called Merkle–Damgård construction, while SHA-3 falls under Sponge functions.
Merkle–Damgård construction is a method of building collision-resistant cryptographic hash functions from collision-resistant one-way compression functions.
And, Sponge functions are a class of algorithms with finite internal state that take an input bit stream of any length and produce an output bit stream of any desired length.
The main reason for the change is because a number of attacks were discovered for its predecessors, there was a fear that the very secure SHA-2 would be broken soon. So, to avoid that they chose Keccak as it was completely different from existing SHA algorithms and the AES.
While Switch is right about the difference between Merkle–Damgård and Sponge constructions, I don't believe he is correct as to NIST's reasoning.
I happened to talk to a NIST cryptographer (John Kelsey) about this. He indicated that they selected Keccak not because they distrust the SHA-2 design (Merkle–Damgård is provably secure if the compression function is collision-resistant, and we have no reason to doubt the SHA-2 compression functions). And it's not to get greater crypto-diversity (even though it does that; the designs of SHA-2 and SHA-3 are quite different). Instead, John indicated that the most attractive thing they saw in Keccak was its flexibility.
While the sponge construction can be used to create a hash function, it can also be used to create other things, such as an XOF (Extensible Output Function; essentially as hash function with an arbitrary length output). NIST has standardized SHAKE-128 and SHAKE-256 as SHA-3 based XOF's. In addition, you can do other things with a sponge construction; it's quite possible that NIST will standardize those usages as well.
