2

Suppose we have a matrix $$ A = \begin{bmatrix} 3 & 0\\0 & -2\end{bmatrix}$$ According to theory, the eigenvectors of $A A^T$ form $U$ and the eigenvectors of $A^T A$ form $V$. Then, solving

$$\det(AA^T - \lambda I)=0$$

I find $\lambda_1=9$ and $\lambda_2=4$. And solving $AA^Tu_1 = 9u_1$ and $AA^Tu_2 = 4u_2$ I find that the eigenvectors of $AA^T$ are $u_1=[1,0]$ and $u_2=[0,1]$, this way obtaining

$$U = \begin{bmatrix}1 & 0\\0 & 1\end{bmatrix}=I_{2\times2}$$

So far so good. But now if I compute $A^TA = AA^T$ yielding the same equations for finding the eigenvectors and I end up with the erroneous solution that $V=I_{2\times2}$ when in fact it should be

$$\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$$

I do not want a solution by inverting $A$ to find $V$. Please, point me where in the calculation above I missed a $-1$. Thank you.

Rodrigo de Azevedo
  • 23,223
alienflow
  • 349
  • 1
    $[0, -1]$ is also an eigenvector of $AA^T=A^TA$. SVD says there exists orthogonal matrices, but they might not be unique. For example you can select $U=-I$ and $V=\operatorname{diag}{ -1, 1 }$. ā€“ obareey Oct 12 '21 at 13:10

2 Answers2

1

The matrices $A^TA$ and $AA^T$ have the same eigenvalues, so we need not perform the eigen-decomposition twice. For example, let $AA^T$ has eigenvalue $\lambda$ and the corresponding eigenvector $v$, then we have,

$AA^Tv=\lambda v \implies (A^TA)(A^Tv)=\lambda (A^Tv)$

i.e., $A^TA$ also has the same eigenvalue $\lambda$ and the eigenvector $A^Tv$ (needs to be normalized).

Since, you already found eigenvalues for $AA^T$ as $9,4$ and the corresponding (left singular) eigenvectors as $\begin{bmatrix}1 \\ 0\end{bmatrix}$ and $\begin{bmatrix}0 \\ 1\end{bmatrix}$ respectively,

The eigenvalues for $A^TA$ will be $9, 4$ and the corresponding unnormalized eigenvectors will be $A^T\begin{bmatrix}1 \\ 0\end{bmatrix}=\begin{bmatrix}3 & 0 \\ 0 & -2\end{bmatrix}\begin{bmatrix}1 \\ 0\end{bmatrix}=\begin{bmatrix}3 \\ 0\end{bmatrix}$ and $A^T\begin{bmatrix}1 \\ 0\end{bmatrix}=\begin{bmatrix}3 & 0 \\ 0 & -2\end{bmatrix}\begin{bmatrix}0 \\ 1\end{bmatrix}=\begin{bmatrix}0 \\ -2\end{bmatrix}$ $\implies$ the corresponding normalized (right singular) eigenvectors are $\begin{bmatrix}1 \\ 0\end{bmatrix}$ and $\begin{bmatrix}0 \\ -1\end{bmatrix}$.

Since we have $A^TA=V\Sigma^2V^T$ and $AA^T=U\Sigma^2U^T$

$\implies$ we have $\Sigma=\begin{bmatrix}\sqrt{\lambda_1} & 0 \\ 0 & \sqrt{\lambda_2} \end{bmatrix}=\begin{bmatrix}3 & 0 \\ 0 & 2 \end{bmatrix}$, $V=I_{2\times 2}=\begin{bmatrix}1 & 0 \\ 0 & 1 \end{bmatrix}$ and $U=\begin{bmatrix}1 & 0 \\ 0 & -1 \end{bmatrix}$,

so that $U\Sigma V^T=\begin{bmatrix}1 & 0 \\ 0 & -1 \end{bmatrix}\begin{bmatrix}3 & 0 \\ 0 & 2 \end{bmatrix}\begin{bmatrix}1 & 0 \\ 0 & 1 \end{bmatrix}^T=\begin{bmatrix}3 & 0 \\ 0 & -2 \end{bmatrix}=A$

Sandipan Dey
  • 2,249
  • Okay, but this approach of computing $v$: $(^)(^)=(^)$ does not work if $\lambda = 0$ because the eigenvector you get is the zero vector. What would you do in that case to obtain a $v$ (eigenvector of $\lambda = 0$ of $A^TA$)? ā€“ alienflow Oct 13 '21 at 04:23
  • If there is a zero eigenvalue, it means both $A^TA$ and $AA^T$ are singular. We need to first find the null space of $AA^T$, i.e., solve the linear homogeneous system of equations $AA^Tv=0$ to find the corresponding nonzero eigenvector $v$ (a non-trivial $vā‰ \vec{0}$ solution will exist since the matrix is not full-rank) and then use $A^Tv$ to compute the corresponding eigenvector of $A^TA$. Since in the approach we are not using any assumption of non-zero eigenvalues, the same approach should work ā€“ Sandipan Dey Oct 13 '21 at 06:53
0

Let ${\bf e}_1, {\bf e}_2 \in {\Bbb R}^2$ be the (unit) vectors of the canonical basis for ${\Bbb R}^2$. Note that

$$ {\bf A} = \begin{bmatrix} 3 & 0\\0 & -2\end{bmatrix} = 3 \, {\bf e}_1 {\bf e}_1^\top + (-2) \, {\bf e}_2 {\bf e}_2^\top = 3 \, \underbrace{\left( \pm {\bf e}_1 \right) \left( \pm {\bf e}_1 \right)^\top}_{\color{gray}{\text{same signs}}} + |-2| \, \underbrace{\left( \pm {\bf e}_2 \right) \left(\mp {\bf e}_2 \right)^\top}_{\color{gray}{\text{opposite signs}}} $$

Thus, the singular values of a real diagonal matrix are the absolute values of the entries on the main diagonal, whereas the left and right singular vectors are $\pm 1$ ("signed") combinations of the vectors of the (same) canonical basis.

Related: Calculating SVD by hand: resolving sign ambiguities in the range vectors.

Rodrigo de Azevedo
  • 23,223