Basic understanding of entropy

Question

I calculated the entropy of a Gaussian having say SIGMA2=0.01 ? The formula 1/2 ​log(2π σ^2​) +1/2. It gives a negative number. But I don't think entropy can be negative. Am I doing something wrong?

Answer 1

Note that there are different concepts of entropy:

For Discrete Distributions

This is what one typically has in mind when talking about entropy. It is defined by

$$H(X) = - \sum_xp(x)\log p(x)$$

and this entropy is non-negative

Differential Entropy

This is defined in a similar way than the entropy for discrete distributions:

$$h(X) = - \int f(x)\log f(x) dx$$

Note:

1. Unfortunately, it does not share all properties with the entropy for discrete distributions. One example: it can be negative.
2. This is the one you used.

Limiting density of discrete points

This is the corrected version of the differential entropy that makes it a natural extension of the entropy for discrete distributions.