Conditional Expectation of Gaussian Random Vector of length n

Question

I am trying to prove the following:

Let $(X_1,\dots,X_n)$ be a Gaussian vector with mean 0 and covariance matrix B. Find the distribution of $E(X_1\mid X_2,\dots,X_n).$

I know in general for two Gaussian r.v. $X_1$ and $X_2$ we can show that $f_{X_1|X_2} (x_1|x_2) = \frac{1}{\sigma_{X_1} \sqrt{2 \pi (1-\rho^2)}}\exp\frac{(-(x_{1}- \rho(\sigma_{X_1}/ \sigma_{X_2})x_2)^2)}{2 \sigma^{2}_{X_1}(1- \rho^{2})}$ where $\rho$ is the correlation and

$E(X_{1}|X_{2})=\int x_{1} f_{X_1|X_2} (x_1|x_2)dx_{1}. $

How can I generalize this for a gaussian random vector of size $n$? Can I conclude on the distribution of $E(X_1|X_2,...X_n)$ based on the form of $f_{X_1|X_2,\dots,X_n} (x_1|x_2,\dots,x_n)$? For example, in the case of 2 Gaussian r.v we produce a normal r.v with variance $2 \sigma^{2}_{X_1}(1- \rho^{2})$ and mean $\rho(\sigma_{X_1}/ \sigma_{X_2})x_2$.

Answer 1

Partition the column vector $X:=(X_1, X_2,\ldots, X_n)^T$ into subvectors $X_a$ and $X_b$: $$ X = \left(\begin{matrix}X_a\\X_b\end{matrix}\right) $$ (in your case $X_a=X_1$ is univariate) and correspondingly partition the mean vector $\mu$ and covariance matrix $\Sigma$ of $X$: $$ \mu = \left(\begin{matrix}\mu_a\\ \mu_b\end{matrix}\right) $$ $$ \Sigma=\left(\begin{matrix}\Sigma_{a,a}&\Sigma_{a,b}\\\Sigma_{b,a}&\Sigma_{b,b}\end{matrix}\right)$$ Assume that $\Sigma_{b,b}$ is invertible. To find the conditional expectation $E(X_a\mid X_b)$, first find a matrix $C$ of constants such that $Z:=X_a- C X_b$ is uncorrelated with $X_b$. For this to be true we demand $$ 0= \operatorname{cov} (Z, X_b)=\operatorname{cov} (X_a - CX_b, X_b)=\Sigma_{a,b}-C\Sigma_{b,b}, $$ which yields $$ C=\Sigma_{a,b}\Sigma_{b,b}^{-1}. $$ Therefore $$ \begin{align} E(X_a\mid X_b)&=E(Z + C X_b\mid X_b)\\ &=E(Z\mid X_b) + CX_b\\ &\stackrel{(*)}=E(Z) + CX_b\\ &= E(X_a) + C(X_b - E(X_b))\\ &= \mu_a + \Sigma_{a,b}\Sigma_{b,b}^{-1}(X_b - \mu_b),\tag1 \end{align} $$ where in (*) we use the fact that $Z$ and $X_b$ are uncorrelated multivariate Gaussian random vectors and therefore are independent. From (1) we see that $E(X_a\mid X_b)$ is a linear combination of the elements of the Gaussian vector $X_b$, and therefore also is Gaussian.

Answer 2

First, it's follows very quickly that the conditional distribution is normal.

$X = (X_1, ..., X_n) \sim N(0, B)$

So the joint pdf. is $f_X(x) = const_1. e^{-\frac{1}{2}x'B^{-1}x}$, where $const_1=\frac{1}{(2\pi)^{n/2}B^{1/2}}$.

We then have -

$$ \begin{array}{rcl}f_{X_1|X_2,...,X_n}(x_1|x_2, ..., x_n) &=&\frac{f_{X_1,X_2,...,X_n}(x_1,...,x_n)}{f_{X_2,...,X_n}(x_2,...,x_n)}\\ &=&\frac{const_1.e^{-\frac{1}{2}x'B^{-1}x}}{f_{X_2,...X_n}(x_2,...,x_n)}\\ &=&const_2.e^{-\frac{1}{2}x'B^{-1}x} \end{array}$$

Note 1: $const_2=\frac{const_1}{f_{X_2,...,X_n}(x_2,...,x_n)}$ is actually determined by $x_2,...,x_n$. But it is useful to think of it as a constant as far as the pdf of $x_1$ is concerned.

Note 2: Since this is a pdf, $const_2(x_2,...,x_n)$ is such that the total integral is 1. This is useful, and enables us to loose track of it as we can recover this anytime by just integrating this over $x_1$, since $const_2 = 1/\int_{x_1}e^{-\frac{1}{2}x'Bx}$.

Say $B^{-1}=D=\left(\begin{matrix}D_{11}&D_{12}&...&D_{1n}\\D_{21}&D_{22}& ...& D_{2n}\\&&...&\\ D_{n1}&D_{n2}&...&D_{nn}\end{matrix}\right)$.

Then note that $x'B^{-1}x=x_1^2D_{11}+2x_1 \sum_{r=2}^n D_{1r}x_r + \sum_{r=2}^n \sum_{s=2}^n x_r x_s D_{rs}$. This shows that the distribution of $X_1$ is indeed normal.

Moreover, we can also read off the distribution's mean and variance -

$$\begin{array}{rcl}x'B^{-1}x&=&x_1^2D_{11}+2x_1 \sum_{r=2}^n D_{1r}x_r + \sum_{r=2}^n \sum_{s=2}^n x_r x_s D_{rs}\\ &=&\frac{(x_1-\mu)^2}{\sigma^2} + c \end{array}$$

Where $\sigma = 1/\sqrt{D_{11}}$, $\mu=-\frac{\sum_{r=2}^n D_{1r}x_r}{\sqrt{D_{11}}}$.

The full distribution $f_{X_1|X_2,...,X_n}(x_1|x_2,...,x_n)$ is thus -

$$\begin{array}{rcl} f_{X_1|X_2,...,X_n}(x_1|x_2,...,x_n) &=&const_2.e^{-\frac{(x_1-\mu)^2}{2\sigma^2} + c}\\ &=&const_2.e^c.e^{-\frac{(x_1-\mu)^2}{2\sigma^2}}\\ \end{array}$$

So the conditional pdf is univariate normal $N\left(-\frac{\sum_{r=2}^n D_{1r}x_r}{\sqrt{D_{11}}}, \frac{1}{D_{11}}\right)$, where $D=B^{-1}$.