Unordered Input

Question

I was just wondering what the best approach is for training a neural network (or any other machine learning algorithm) where the order of the inputs does not matter.

For example: f(x1,x2,x3,x4) = f(x2,x1,x3,x4) = f(x2,x4,x1,x3)

My current approach is to just randomize the order of every training sample (my network has 44 inputs). It kind of works but it causes the result of the validation loss to jump around every epoch a lot.

Maybe something related to embedding?

There are other questions that refer to this but with a variable number of inputs and typically in regards to a RNN. I'm talking about the simple case of a fixed number of inputs where order doesn't matter.

Thanks!

Answer 1

The suggestion of ncasas is a good one but not very clean. This ordering makes a lot of sense when it's 1-dimensional, but when you introduce more features the ordering will become more and more arbitrary. This is a problem I have come accross multiple times. This paper (https://arxiv.org/pdf/1612.04530.pdf) tries to tackle permutation equivariance, which is not exactly the same as your problem, where you would want permutation invariance. With the equivariance the output from the layer still gets shuffled if you shuffle your inputs, but the values will be the same due to the weight sharing. You could extend this idea fairly easily by using a pooling operation (max pooling or mean pooling) on the output from this permutation layer. This means you get one number output out of your layer. Repeating this with n different weight matrices you can get n outputs, which can then be fed into a normal feed forward network. Caution: This is something I just thought of based on your question and their paper, I never tried it but I don't see why it wouldn't work. In the case of sports teams you would use two lists of inputs, one for each team. Based on what you want to predict you could even fit these permutation layers into a siamese network, introducing even more weight sharing!