4

I need a hint on how to start solving the following problem.

Entropy of a discrete variable X is $H(X) = −\sum_{x\in \{x:P(X=x)>0\}}P(X=x)logP(X=x)$. Let $f:R → R$ be any function.\

a) Show that entropy of a discrete variable X is greater than or equal to entropy of a discrete variable f(X).\

b) Show that equality occurs if and only if function f is injective on {x : P(X = x) > 0}

Novak Djokovic
  • 877
  • 6
  • 9

1 Answers1

3

$$\begin{align}&-\sum_{y : P(f(X) = y) > 0} P(Y=y) \log P(Y = y)\\ &= -\sum_{y : P(f(X) = y) > 0} \left\{\left(\sum_{x \in f^{-1}(y) : P(X=x) > 0} P(X = x)\right) \log \left(\sum_{x \in f^{-1}(y) : P(X=x) > 0} P(X=x)\right)\right\}.\end{align}$$ It then suffices to show $-\left(\sum_{x \in f^{-1}(y) : P(X=x) > 0} P(X = x)\right) \log \left(\sum_{x \in f^{-1}(y) : P(X=x) > 0} P(X=x)\right) \le - \sum_{x \in f^{-1}(y) : P(X=x) > 0} P(X=x) \log P(X=x)$ for each $y$, since then the above would be bounded by $$\le - \sum_{y : P(f(X)=y)>0}\sum_{x \in f^{-1}(y) : P(X=x) > 0} P(X=x) \log P(X=x) = - \sum_{x : P(X=x)>0} P(X=x) \log P(X=x).$$

The unverified claim reduces to showing that the function $g(x) = - x \log x$ is sub-additive, i.e. $g(a+b) \le g(a) + g(b)$.

angryavian
  • 93,534