Is the expectation of an invertible matrix invertible?

Question

Let $X \in \mathbb{R}^{k}$ be an $k$-dimensional random variable, such that $XX' \in \mathbb{R}^{k \times k}$ is invertible.

Does it always follow that $\mathbb{E}[XX']$ is invertible?

---

Let's assume that $\mathbb{E}[X^2_i] < \infty$ for each $i=1,\dots,k$.

I am then thinking of $\mathbb{E}[XX']$ as a $k \times k$ matrix filled with entries like $$ \begin{pmatrix} \mathbb{E}[X_1^2] & \mathbb{E}[X_1 X_2] &\dots &\mathbb{E}[X_1X_k] \\ \mathbb{E}[X_2 X_1] &&& \\ \vdots &&&\\ \mathbb{E}[X_kX_1] & \dots & & \mathbb{E}[X^2_k] \end{pmatrix} $$ However it's not clear to me that this matrix is always invertible, especially if we don't know the marginal and joint distributions so that the values inside can be anything.

This comes up in statistics in the context of linear regression, where some authors assume $XX'$ is invertible but then later on simply write $\mathbb{E}[XX']^{-1}$ where its implied that it exists.

Answer 1

From the comments, here is an educated guess at what the situation is. In a text that you're trying to understand, the author considers a (mean-zero?) random variable $X \in \Bbb R^k$ and a $k \times n$ matrix that I will call $\mathbf X$, whose columns consist of $n$ observed values of $X$. Importantly, we have $n \geq k$. The question is: if it is known that $\mathbf{XX}'$ is invertible (which is only possible for $n \geq k$) almost surely, why can we conclude that $\Bbb E[XX']$ is invertible?

Here's an answer: the matrix $\mathbf C:=\frac 1{n-1} \mathbf{XX}'$ is the sample covariance, which is an unbiased estimate of the true covariance. That is, we have $$ \mathbb E[\mathbf C] = \mathbb E\left[\frac 1{n-1}\mathbf{XX}'\right] = \mathbb E[XX']. $$ We are given that $\mathbf{XX}'$ is invertible almost surely, hence $\mathbf C = \frac 1{n-1} \mathbf{XX}'$ is invertible almost surely. It suffices to use this to show that $\mathbb E[\mathbf C]$ is invertible.

Towards that end, it helps to note that $\mathbf C$, like the covariance $\mathbb E[\mathbf C]$, is positive semidefinite. Thus, $\mathbf C$ fails to be invertible if and only if there exists a non-zero vector $v$ for which $v' \mathbf C v = 0$, and in either case we always have $v' \mathbf C v \geq 0$.

Let $v$ be any non-zero vector. Because $\mathbf C$ is invertible with probability $1$, $v' \mathbf C v$ is positive with probability $1$, which implies that $\mathbb E[v'\mathbf C v] > 0$. However, we have $$ 0 < \mathbb E[v'\mathbf C v] = v' \mathbb E[\mathbf C] v. $$ Because this holds for every non-zero vector $v$, we can conclude that the positive semidefinite matrix $\mathbf E[\mathbb C]$ is invertible.