2

An entropy (is Shannon sense) can be interpreted as uncertainty or missing knowledge. When the knowledge is added, the entropy decreases. Hence it can also be interpreted as information content.

However, there are discrete distributions which do not have an entropy because $-\sum_{i=1}^{\infty}p_i\log p_i$ tends to infinity (see here).

How to interpret this situation? Does non-existent or infinite entropy means that it is not possible to remove uncertainty and get complete knowledge about values of variable described by such distribution?

  • 1
    If you interpret the entropy as being the expected minimal amount of number of yes/no questions, you may get a feeling of what is happening. – P. Quinton Feb 24 '21 at 15:24

1 Answers1

1

A discrete distribution (with infinite outcomes) can have an infinite entropy. I don't see much to interpret here. Actually, one might argue that, on the contrary, the surprising thing is that some (most) discrete distributions with infinite outcomes have finite entropy, i.e. they produce a finite amount of information, so they can be described (in average) with a finite number of bits.

I think that the most natural example of infinite entropy is a continuous distribution (say, uniform on $[0,1]$), which can be regarded as the limit of a discrete uniform distribution for increasingly large number of values. It's not surprising that a real number in $[0,1]$ has infinite entropy, because to describe it you need an infinite number of bits (think of its binary fractional representation). Put in other way, in a real number in $[0,1]$ you can code an arbitrarily big amount of information.

leonbloy
  • 66,202
  • I think there is a slight misunderstanding. In case of continuous distribution, the entropy is calculated as $-\int_a^b f(x)\ln f(x)dx$. For continuous uniform distribution $f(x) = \frac{1}{b-a}$. Since $f(x)$ is constant and therefore $\ln f(x)$ is also constant, entropy is $\ln(b-a)$. This is finite number. –  Feb 25 '21 at 08:30
  • @MartinVesely No, that's the differential entropy, which is very different from the Shannon entropy (it doesn't correspond to the average amount of bits of information, or the average bits needed to encode it - and it can be zero or even negative). https://math.stackexchange.com/questions/2552895/uniformly-random-number-on-0-1-has-zero-entropy/2553036#2553036 https://math.stackexchange.com/questions/2880612/comparing-differential-entropy-of-normal-distribution-to-shannon-entropy/2880947#2880947 – leonbloy Feb 25 '21 at 10:34
  • Thanks. I clarified my question. I am asking about discrete distributions. –  Feb 26 '21 at 07:24