If $X$ is not full rank, are $X^TX$ or $X^TX + \lambda I_p$ invertible?

Question

Suppose $X$ is a $n \times p$ matrix, where $rank(X) < p$. Since $X$ is not full rank, then it is not invertible.

I'm trying to understand whether functions of $X$ are invertible:

$X^TX$
$X^TX + \lambda I_p$ ($\lambda > 0$ is some scalar, and $I_p$ is a $p \times p$ identity matrix)
$X(X^TX + \lambda I_p)^{-1}X^T$

My intuition is that 1) is NOT invertible, but 2) IS invertible. Given that 2) is invertible, however, 3) is NOT invertible.

Is this correct? Can anyone help me understand why this is?

Hm, I've not thought about that..would the answers be different for $n < p$ vs. $n \geq p$? — Adrian, Jul 11 '20 at 16:52
To be sure to have 2) invertible, you need $\lambda$ to be strictly positive — Damien, Jul 11 '20 at 16:54
In my understanding, answer is not different if n is greater or lower than p — Damien, Jul 11 '20 at 16:55

Clarinetist · Accepted Answer · 2020-07-11T17:19:32.987

1

$\DeclareMathOperator{\rank}{rank}$

$1$ is not invertible, and it is because $\rank X = \rank X^TX$. See Prove that $\text{rank}(X^TX)=\text{rank}(X)$.

$2$ is invertible, as long as $\lambda > 0$. See When is $\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}$ invertible?.

$3$ is known as the hat matrix for ridge regression. Thanks to Brian Borchers. If $n > p$,

$$\rank X(X^TX + \lambda I)^{-1}X^T \leq \min (\rank X, \rank (X^TX + \lambda I)^{-1}X^T)$$

We know that $\rank X <p$ and $(X^TX + \lambda I)^{-1}X^T$ is a $p \times n$ matrix. Thus $\rank (X^TX + \lambda I)^{-1}X^T \leq \min(n, p) = p$ and it must be the case that

$$\rank X(X^TX + \lambda I)^{-1}X^T\leq \min (\rank X, \rank (X^TX + \lambda I)^{-1}X^T) < p < n$$

hence $X(X^TX + \lambda I)^{-1}X^T$ (an $n \times n$ matrix) is not invertible.

edited Jul 11 '20 at 17:19

answered Jul 11 '20 at 16:54

Clarinetist

20,278
10
72
137

assuming that $n > p$, then $X(X^{T}X+\lambda I)^{-1}X^{T}$ has size $n$ by $n$ but its rank is $<=p$, so it's rank deficient. Use the fact that $\mbox{rank}(AB)\leq \min(\mbox{rank}(A),\mbox{rank}(B))$ – Brian Borchers Jul 11 '20 at 17:02
In 3), you are multiplying a full rank matrix with matrices which are not full rank. So the result cannot be full rank – Damien Jul 11 '20 at 17:02
@BrianBorchers Thank you so much! – Clarinetist Jul 11 '20 at 17:07
It's also interesting to investigate the case where $n$ is less than the rank of $X$. – Brian Borchers Jul 11 '20 at 17:20
@BrianBorchers n < rk(X) is impossible no? – Stockfish Jul 12 '20 at 17:09
Sorry- I meant $n \leq p$. – Brian Borchers Jul 12 '20 at 19:01

If $X$ is not full rank, are $X^TX$ or $X^TX + \lambda I_p$ invertible?

1 Answers1