3

Entropy is defined as $$H(x) = E_{x\sim p(x)}[ - \log p(x)]$$ and $- \log p(x) \geq 0$ so it makes sense that the expectation is always non-negative, here is a proof.

However, Wikipedia says the entropy of a normal distribution is $\tfrac{1}{2} \ln(e2\pi\sigma^2 $), which means that the entropy can be negative for some values e.g. $\sigma = 0.01$ but how can be this be the case and how can it be interpreted?

monk
  • 147
  • 3
    The differential entropy in not the (traditional / Shannon) entropy. https://math.stackexchange.com/questions/2552895/uniformly-random-number-on-0-1-has-zero-entropy/2553036#2553036 https://math.stackexchange.com/questions/2880612/comparing-differential-entropy-of-normal-distribution-to-shannon-entropy/2880947#2880947 https://math.stackexchange.com/questions/1398438/differential-entropy/1398471#1398471 – leonbloy Oct 31 '22 at 12:12

3 Answers3

4

You're either confusing $p(x)$ with probability $P(E)$, the probability mass function $P(x)$ or the cumulative distribution function $F(x)$ which are all constrained to be $\in [0, 1]$.

However, $p(x)$ denotes the probability density function (pdf), which can take any non-negative value.

Thus, $-\log (p(x))$ can indeed be a negative quantity.

whoisit
  • 4,028
  • 1
  • 8
  • 29
2

As your second link says, the entropy of a continuous distribution with density $f$ can be negative, as $f$ may be greater than $1$ (though $\int_\mathbb{R} f$ must still be $1$) and thus $-\log f$ may be negative. The continuous case is a bit less well-behaved than the discrete one in general; it's also not invariant under a change of variables (though there is a reasonable formula for a constant rescaling factor).

anomaly
  • 26,475
2

$-\ln p(x)$ is not necessary nonnegative.

For example, if $p(x)$ is a density of $N(0, 0.01)$ then $p(x) \ge x$ for some $x$.

Sergei Nikolaev
  • 1,471
  • 4
  • 12