Comparing Differential Entropy of Normal Distribution to Shannon Entropy

Question

The (differential) entropy of the multivariate normal distribution is given by:

$$H(\underline{X}) = \frac12 \ln(|2 \pi e \Sigma|)$$

Does the Shannon entropy:

$$ H(\underline{X})=−p(\underline{x}) \sum_{\underline{x}\in \underline{X}}\log_2 p(\underline{x}) $$

approach this in the limit as e.g. $\; p(\underline{x}) \rightarrow 0 \,$?

If not, are they related or comparable at all? From personal coding experiments, they do appear to become similar, but only for the standard normal: $$ \underline{X} \sim \mathcal{N}(0,1)$$

But I am not aware of any derivation explaining the relationship?

Answer 1

The result for the differential entropy of the multivariate normal distribution simply arises from its definition:

$$h(X)= - \int f(X) \log(f(X))dX $$

are they related or comparable at all?

They are not comparable. The Shannon entropy of a continuous variable is infinite.

Think of the case of a uniform rv on $[0,a]$. Its differential entropy is $h(X)=\log(a)$. The Shannon entropy, for a discrete uniform is $H(X) = \log(N)=\log(a / \Delta x)= \log(a) - \log(\Delta x)$ Then $H(X) \to +\infty$ as $\Delta x \to 0$. (This agrees with the concept of the Shannon entropy as average information: the information you can code in a real number over any interval is infinite).

Basically the same relationship exists for the Gaussian or any well behaved continuous density.

At most, you can think the differential entropy as the "residual" of the Shannon entropy, what is left apart from the (tending to infinite) discretization term.