$M = A N A^t,$ where $A$ not necessarily square, $M, N$ symmetric. If I know $N^{-1}$ exists, is there any easy way to compute $M^{-1}?$

Question

Suppose I have $M = A N A^t,$ where $A$ is not necessarily square, and $M, N$ are symmetric. If I know the inverse of $N,$ is there any particularly easy way to compute the inverse of $M?$ (and yes, I know that $M$ might not be invertible - "compute the inverse" means also "tell me if there is not one").

For background: I am using Newton's method to minimize a function $f$ which can be written as $f = g \circ h,$ with $h$ linear. I can compute and invert the Hessian of $g$ by hand. $A$ is the Jacobian of $h.$

Answer 1

Here's one way to approach the problem.

I assume that $A$ is of size $m \times n$ with $m < n$. If $A$ has rank less than $m$, then $M$ cannot be invertible. Otherwise, there exists a matrix $B$ such that the matrix $\binom AB$ is invertible. We note that $$ \pmatrix{A\\B}N\pmatrix{A\\B}^T = \pmatrix{ANA^T & ANB^T\\B^TNA & B^TNB}. $$ We have $$ \left[\pmatrix{A\\B}N\pmatrix{A\\B}^T\right]^{-1} = \pmatrix{A\\B}^{-T}N^{-1}\pmatrix{A\\B}^{-1}, $$ which means that it suffices to use the inverse of this large matrix to find the inverse of its submatrix. As is discussed here, this can be done with the help of the Woodbury formula.

---

In the case that $N$ is positive definite, we can write $$ M = AN^{1/2}[N^{1/2}]^TA^T = (AN^{1/2})(AN^{1/2})^T. $$ From there, if $AN^{1/2} = U \Sigma V^T$ is an SVD, then we can write $$ M^{-1} = [U \Sigma \Sigma^T U^T]^{-1} = U [\Sigma \Sigma^T]^{-1} U^T. $$ Of course, $[\Sigma \Sigma^T]^{-1}$ is the inverse of a square diagonal matrix.

In fact, it is not important to have $N^{1/2}$ be the positive definite square root of $N$ here, it suffices to replace $N^{1/2}$ with $L$ from the Cholesky decomposition $LL^T = N$. Because $N$ is a rank-one update of a diagonal matrix, its Cholesky decomposition is relatively inexpensive to compute