$\mathrm {H} (X)=\mathbb {E} [\log \frac{1}{p(X)}]$
Is Shannon entropy always nonnegative? I have heard it can be negative for some continuous distributions. A counterexample or a proof of nonnegativity would be appreciated.
Asked
Active
Viewed 1,083 times
-1
-
2Consider the continuous uniform distribution over $[0,1/2]$. – sudeep5221 Nov 18 '22 at 22:10
-
1How about searchig for "entropy negative" in this site? – leonbloy Nov 19 '22 at 12:40
1 Answers
1
It is always nonnegative for discrete distributions. For continuous distributions (then called sometimes differential entropy) it can indeed be negative. You can find a standard proof at the first chapter of Thomas and Cover.
Yuval
- 41