I'm reading the cornerstone paper Sequence to Sequence Learning with Neural Networks by Ilya Sutskever and Quoc Le. On the first page, it briefly mentions that:
A surprising example of the power of DNNs is their ability to sort
N N-bit numbers using only 2 hidden layers of quadratic size
Can anyone briefly outline how to sort numbers using only 2 hidden layers?
Doing some research, I found a paper which proves that sorting can be done with at most 3 layers, and that their solution is optimal if you restrict the size of the network to be polynomial w.r.t. to the number of input numbers:
Kai-Yeung Siu, Jehoshua Bruck, Thomas Kailath, Thomas Hofmeister, Depth Efficient Neural Networks for Division and Related Problems (1993), see Theorem 7 on page 955 (page 10 in the PDF).
I'd imagine one could come up with weights manually to do the job exactly, but maybe the point is to be able to train one? A couple of examples:
