I am looking at a setting where I have a random variable $X$ and two other random variables that are derived from $X$, $Y = f(X)$ and $Z =g(X)$. What I want to do is compute the mutual information between $Y$ and $Z$.
I thought of starting with the full probability density of the three variables and then marginalizing things as needed:
$$p(X, Y, Z) = p(Z | Y, X) p(Y | X) p(X).$$
Since $Y$ and $Z$ are determined by $X$, I have that $p(Z|Y,X) = \delta(Z-g(X))$ and $p(Y|X) = \delta(Y - f(X))$, giving me $$p(X, Y, Z) = \delta(Z-g(X)) \delta(Y - f(X)) p(X).$$
Using the properties of the delta Dirac I get the following PDF's for $Y$,$Z$, and $Z$ and $Y$ $$p(Y=y) = \sum_{f(x_i)=y}\frac{p(X=x_i)}{|f'(x_i)|},$$ $$p(Z=z) = \sum_{g(x_j)=z}\frac{p(X=x_j)}{|g'(x_i)|},$$ $$p(Y=y, Z=z) = \sum_{f(x_i)=y}\sum_{g(x_j)=z}\frac{\delta(x_i-x_j)p(X=x_i)}{|f'(x_i)||g'(x_j)|}.$$
Everything seems right up to here, but I run into problems when I try to compute the mutual information
$$MI(Y,Z) = \int_{-\infty}^\infty\int_{-\infty}^\infty p(y,z)\log\left(\frac{p(y,z)}{p(y)p(z)}\right)dydz.$$
This happens because I get a term of the form $$\int_{-\infty}^\infty \delta(x)\log(\delta(x))dx,$$ which is not well defined (when I approximate the delta function with step functions, I get a divergent function).
At what point am I doing a wrong step? (If you need more of the intermediary steps, I can provide them).
