For a Gaussian random variable of unitary variance, the entropy is given by $H = 0.5\log(2 \pi e)$. Also, the mutual information (MI) between correlated Gaussian random variables is given by $MI = -0.5\log(1 - \rho^2)$, where $\rho$ denotes correlation. With these equations in mind, I can think of values for $\rho$ with which the MI is larger than the joint entropy between the two random variables. Am I missing something here or the MI may indeed be larger than the joint entropy?
Asked
Active
Viewed 421 times
1 Answers
1
For a Gaussian random variable of unitary variance, the entropy is given by $H=0.5 \log(2 \pi e)$.
Not quite. That's the differential entropy, which is a very different thing from the Shannon entropy. You cannot compare in a meaningful mutual information with differential entropies.
In particular, calling $h()$ the differential entropies, it's true that
$$I(X;Y) = h(X) - h(X|Y)$$
but it's not true that $h(X|Y) \ge 0$. Hence, the mutual information can be larger that the (differential) entropy of any variable (or the joint two variables).
leonbloy
- 66,202