Eigendecomposition of a real, symmetric matrix with distinct and nondistinct eigenvalue

Question

My question is motivated by this question of uniqueness of SVD of a matrix, especially its first answer.

So to start with, assume that $A$ be an $m\times m$ real symmetric matrix with distinct non-zero eigenvalues. So this means it's orthogonally diagonalizable, i.e. $\exists D$ diagonal with distinct entries (the eigenvalues of $A$) and an orthogonal matrix $U$ so that $A=UDU^T.$

My question is: how unique can $U$ be? The answer I'm getting, motivated by the first answer in the above link (the step about commuting matrices with a diagonal matrix with distinct eigenvalues) is that if there are two such $U's$ say $U_1,U_2,$ then $U_2^{-1}U_1$ is a diagonal matrix with $\pm 1$ on the diagonal. Below is the proof, but I'm just requesting you to check its correctness and the resulting conclusion.

Let $A=U_1DU_1^T=U_2DU_2^T \implies U_2^TY_1 D = D U_2^TU_1 \implies U_2^TU_1$ commutes with the diagonal matrix $D$ that has distinct entries. This means $U_2^TU_1= U_2^{-1}U_1$ must be diagonal, say $D'.$ But since $U_1,U_2$ are orthogonal, so is $D',$ and this means $D'$ has only $\pm 1$ on the diagonal. From here, we conclude that any two orthogonal change of basis $U_1,U_2$ of a real, symmetric matrix with distinct eigenvalues must be related by $U_1=D'U_2,$ where $D'$ is a diagonal matrix with $\pm 1$ on the diagonal.

Remark: this is connected with SVD because to have SVD of an $m\times n$ matrix $M$, first we look at the orthogonal eigenvectors $\{v_i:1\le i \le m\}$ of the $m\times m$ matrix $MM^T,$ that form an orthogonal matrix $V$, that orthogonally diagonalizes $MM^T: MM^T=V\Sigma V^T, \Sigma$ being the diagonal matrix consisting of the singular values of $M,$i.e. the eigenvalues of $MM^T.$

Question 1: In above, the assumption of distinct eigenvalues was crucial, as seen in the counterexample of $A:=I_m,$ where the orthonormal change of basis $U$ can be any orthogonal matrix. This brings me to the question: if $A$ has repeated eigenvalues or one or more zero eigenvalues, how unique is its eigendecomposition: if $A=U_1DU_1^T=U_2DU_2^T,$ how are $U_1,U_2$ related? If we assume here that $rank(A)=r \le m,$ then $\exists \{u_1 \dots u_r\}$ orthonormal that spans the eigenspace of $A, Au_i=\lambda_i u_i, \lambda_i\ne 0.$ But how unique is the chose of this orthonormal eigenbasis $\{u_1 \dots u_r\}$ in case $\lambda_i \ne 0$ are all distinct? It seems here we can't apply the commuting matrix with a diagonal matrix argument above, hence my question.

Question 2: For the $(m-r)$ dimensional eigenspace of $A$ corresponding to the zero eigenvalue, we can choose any orthonormal eigenbasis we like and this'll create a non-uniqueness for sure. So is there a result that tells us how non-unique can the orthogonally diagonalizing matrix $U$ be?

Here's my attempt: assuming $\{u_{r+1}, u_{r+2}\dots u_m\}$ form an orthonormal basis for the eigenspace for the zero eigenvalue, we can write $A=UDU^T, U:=[u_1, u_2 \dots u_r, u_{r+1}\dots u_m].$ And if two such $U'$s exist, then we'll still end up with $U_2U_1^{T}$ commutes with $D, $ but then it seems that only an $r \times r$ submatrix of $U_2U_1^{T}$ will be diagonal with $\pm 1,$ but I'm not sure about this part. Any thoughts?

Answer 1

This is equivalent to understanding how unique an orthonormal basis of eigenvectors for $A$ is. The answer is the following: if we write $V_{\lambda} = \ker(A - \lambda)$ for the eigenspaces of $V$, then $\mathbb{R}^m$ is the orthogonal direct sum

$$\mathbb{R}^m \cong \bigoplus_{\lambda} V_{\lambda}$$

of the eigenspaces of $A$, and on each eigenspace $V_{\lambda}$, $A$ acts (by definition) by multiplication by the scalar $\lambda \in \mathbb{R}$. If $A$ has distinct eigenvalues, each of these eigenspaces is $1$-dimensional so has a unique orthonormal basis up to sign (since $\mathbb{R}$ has exactly two orthonormal bases, namely $\{ 1 \}$ and $\{ -1 \}$); in this case $A$ itself has a unique orthonormal basis up to sign (and permutations), which gives your first result.

If the eigenvalues of $A$ have multiplicity then the eigenspaces $V_{\lambda}$ are no longer $1$-dimensional. In this case a choice of orthonormal basis of eigenvectors of $A$ is exactly the same thing as a choice of orthonormal basis of each eigenspace $V_{\lambda}$, and we have no more uniqueness than this. Concretely, this means that the difference between two orthogonal matrices $U, U'$ satisfying $A = UDU^T$ takes the form of an arbitrary block orthogonal matrix whose blocks correspond to the multiplicities of the eigenvalues of $A$; for example, if $D$ is diagonal with entries $6, 6, 6, 5, 5, 4$ then the corresponding block orthogonal matrices have the form

$$\begin{bmatrix} U_3 & 0 & 0 \\ 0 & U_2 & 0 \\ 0 & 0 & U_1 \end{bmatrix}$$

where $U_i$ is an $i \times i$ orthogonal matrix.

The same analysis applies to SVD but with "eigenspace" replaced by "singular space"; namely, the "right singular spaces" are the eigenspaces of $A^{\dagger} A$ and the "left singular spaces" are the eigenspaces of $A A^{\dagger}$, and a choice of SVD is a compatible choice of orthonormal bases of each of the singular spaces. For more on this see here.