How to identify the distribution when its moments are specified?

Question

I was reading my course notes and I came across this statement:

If we are given a set of moments, we can identify the distribution that they came from.

My question is: how do we identify the distribution when its moments are specified?

Answer 1

This is in no way a trivial question. So I will do my best in answering and filling the gap left by the previous comments. EDIT: I've had to remove links as I don't have the reputation to post them, please just search in google what is written.

It is firstly worth stating that the comment in your course notes

If we are given a set of moments, we can identify the distribution that they came from.

Is either implicitly assuming some rather strict conditions, or just plain wrong. In general it is not possible from a set of moments to determine a probability distribution.

Under certain extra conditions this is possible, a good discussion can be found https://mathoverflow.net/questions/3525/when-are-probability-distributions-completely-determined-by-their-moments and Do moments define distributions?

in the case of where you assume a parametric form of the probability distributions one can use the method of moments (search method of moments wikipedia) to determine the parameters of the distribution, this can be extended to semi-parametric and non-parametric distributions (where the number of parameters is allowed to grow with the data). Maybe this is what your course meant? However these methods are very restrictive and are very quickly either over or under-determined. I.e you either need more moments than you have, or you need to throw away valuable moment information.

One thing I have not seen on math stackexchange, maybe because this tends to be in the realm of Physicists and not Mathematicians, is the method of maximum entropy with moment constraints. Essentially the idea is, given a set of moments and no extra information, what is the best guess I can have for a probability distribution?

The answer is to maximize the functional

$S = \int p(x)\log p(x) dx - \sum_{i}\lambda_{i}[\int p(x)x^{i} - \alpha_{i}]$

For example, if we had a mean and variance constraint this would become

$S = \int p(x)\log p(x) dx - \alpha[\int p(x) - 1] - \beta[\int p(x)x - <x>] - \gamma[\int p(x)x^{2} - <x>^{2} - \sigma^{2}]$

Which when maximized gives us the normal distribution. Great references include the wikipedia article (search principle of maximum entropy wikipedia) and Jaynes where do we stand on Maximum entropy (where do we stand on maximum entropy jaynes on google), which is an absolute must read

My own research, unifies quantum field theory with statistical mechanics under the umbrella of independence in statistics. Furthermore the method works well in Machine learning for large matrix approximations. (Search Diego Granziol arxiv).

Answer 2

I'm about 6 years late, but I was trying to get a quick answer to this and thought others might benefit.

As another physicist answering, this is a well-known problem encountered in statistical field theory, particularly with regards to the Replica trick. There, you're trying to determine P(x) given that you know $\mathbb{E}_{x \sim P}[x^n]$ for integer N

The short answer can be found on page 214 of Kardar's Statistical Physics of Fields:

This [only knowing integer-valued moments $\mathbb{E}_{x \sim P}[x^n]$ for n integer] is one of the difficulties associated with the problem of deducing a probability distribution $P(x)$ from the knowledge of its moments $\mathbb{E}_{x \sim P}[x^n]$. There is in fact a rigorous theorem that the probability distribution cannot be uniquely inferred if its nth moment increases faster than n! [N.I. Akhlezer, The Classical Moment Problem (Oliver and Boyd, London, (1965)]