1

We are working on finding differential entropy of Dirac delta function $\delta(x)$, that is $$H(X)= - \int_{-\infty}^{\infty} \delta(x) \ln \delta(x) \, \mathrm{d}x$$ We found answer that $H(X)= -\infty$ by defining \begin{align*} \delta_{\Delta}(x)= \begin{cases} 1/\Delta & -\Delta/2\leq x\leq \Delta/2\\ 0 & \text{otherwise} \end{cases} \end{align*} and computing ($0 \ln 0=0$ is pre-defined for computing entropy) $$H(X)= \lim_{\Delta \rightarrow 0} - \int_{-\infty}^{\infty} \delta_{\Delta}(x) \ln \delta_{\Delta}(x) \, \mathrm{d}x= \lim_{\Delta \rightarrow 0} \ln \Delta= -\infty$$

Later, we had a different way of thinking, but we are not sure if it is correct and why. Since delta function can be seen as $\delta(x)= \infty$, if $x=0$, and $\delta(x)= 0$, if $x \neq 0$, then the entropy function becomes \begin{align*} H(X)= - \int_{0}^{0} \infty \ln \infty \, \mathrm{d}x= -\infty \end{align*} where for $x \neq 0$, $0 \ln 0=0$.

The second way of thinking gives the same result but we are not sure if this is valid mathematically and why.

Intuitively, we think this is invalid with the following reasons: 1) basically integral of a single point is $0$; 2) $\delta(x)$ is not an elementary function, $\delta(0)$ can be seen as $\infty$ does not strictly mean $\delta(0)= \infty$. But still feel something is missing.

We are seeking for explanation of this. All helps are appreciated.

PrincessEev
  • 50,606
Jay
  • 11
  • 1

1 Answers1

1

The Dirac delta is not a function in the usual sense. It is called a "generalized function" or "functional": it does not send numbers to numbers, but instead functions to numbers. Specifically, it is defined by the rule (for "sufficiently nice" functions $f$) $$ \delta(f) := f(0) $$ (Specifically, we typically use $f \in \mathcal{C}_c^\infty(\mathbb{R})$.) This is sometimes written using inner product notation, $$ \langle \delta,f \rangle = f(0) $$ and, knowing the $L^2(\mathbb{R})$ inner product we thus sometimes write $$\begin{align*} \int_{\mathbb{R}} f(x) \delta(x) \, \mathrm{d}x &= f(0) \\ \int_{\mathbb{R}} \delta(x) \, \mathrm{d}x &= 1 \end{align*}$$ The "rule" that $$ \delta(x) \stackrel{\text{"defined"}}{=} \begin{cases} 0, & x \ne 0 \\ +\infty, & x = 0 \end{cases} $$ is more a heuristic and hand-waving than it is anything rigorous (and that would be the best interpretation of it).

This has probably been addressed many times on this site, and is mentioned in the Wikipedia article for the Dirac $\delta$.

You can define it via a limit as your first attempt discusses (discussed some here, alternatives here & here to cement my point), so long as you pay mind to what the $\delta$ distribution really is, i.e. what it acts upon and the topology it's expected to respect.

I would also take issue with the claim that $$ - \int_{0}^{0} \infty \ln \infty dx \stackrel ? = -\infty $$ since you are integrating on a set of measure zero. I think you could just as easily make the argument the quantity is zero. In fact, that is the convention used in Measure & Integral (the textbook my school prefers for measure theory and Lebesgue integration): that $0 \cdot \infty = 0$.

I recognize this is already hinted at in $(1)$ in your post, but I figured it merited pointing out again.

PrincessEev
  • 50,606