Assume we have a function
$$H(x) = -\sum_{i=1}^n x_i \log x_i$$ where $0 < x_i < 1$ $\forall i$ (this is the Shannon entropy if you are familiar with it).
I am reading a paper in which the authors stated the following approximation $$H(x) = -\sum_{i=1}^n x_i \log x_i \approx \sum_{i=1}^n x_i (1-x_i)$$ Can anyone tell me the intuition behind this approximation?
$\log (1-y) = - \sum_{n=1}^\infty \frac{y^n}{n}, |y|<1$.
If $0<x<1$, then $0<1-x<1$, let $y=1-x$,
$\log(1-(1-x))=\log x = -\sum_{n=1}^\infty \frac{(1-x)^n}{n}$.
$-\log x = \sum_{n=1}^\infty \frac{(1-x)^n}{n}$.
Dropping higher order terms: $\log x \approx 1-x$.
The function $x\log(x)$ has a zero at $x=1$, and it sort of has a zero at $x=0$ in that $$\lim_{x\to0^+}x\log(x)=0$$ And there are no other zeros. Just based on that, it is reasonable to approximate $$x\log(x)\approx cx(x-1)$$ They go with $c=1$, but depending on what they are after, there could be better approximations from other values of $c$. Like $c=1.5$ seems better just from comparing graphs: desmos.com/calculator/yaxpkjyxcv.
