0

I'm trying to get a deeper understanding of why SVD is not necessarily unique and how different SVDs of a particular matrix are related to each other. I have a geometric understanding of what singular values and represent.

How are different Singular Value Decompositions of a particular matrix, $A$, related to each other? Can some matrices have infinitely many SVDs? Can they have infinitely many different singular vectors?

Ahmir_r4
  • 49
  • 6

1 Answers1

6

Short version: The singular values are always unique. If you think of the left and right singular vectors as the eigenvectors of $X X^T$ and $X^T X$ respectively, all the non-uniqueness of SVD boils down to the lack of uniqueness of choices of orthonormal bases for the eigenspaces of $X X^T$ and $X^T X$. However, these eigenspaces themselves are unique.

Long answer: SVD written in matrix form $X = U \Sigma V^T$ is highly non-unique if $X$ has low rank, because there are many choices of left and right singular vectors corresponding to the singular value $0$ (we can pick any orthonormal basis of the nullspace of $X$ and of $X^T$). To get something more unique it's better to write SVD in the form

$$X = \sum_i \sigma_i u_i v_i^{\ast}$$

where the sum runs over nonzero singular values only, which is unique in the following sense: if we sort the singular values in decreasing order as is standard, and the nonzero singular values are distinct, then the pair consisting of the left and right singular vectors $u_i, v_i$ is unique up to a sign in the real case, and up to a phase in the complex case. That is, we always have the freedom to pick a scalar $c$ satisfying $\| c \| = 1$ and replace the pair $(u_i, v_i)$ with $(c u_i, c v_i)$, but that's it (if the singular values are distinct). In particular the rank-$1$ operator $u_i v_i^{\ast}$ is unique.

So over $\mathbb{R}$ a generic matrix has finitely many SVDs which differ only by signs on the singular vectors, and this is a very minor difference. Over $\mathbb{C}$ every nonzero matrix has infinitely many SVDs, which generically differ by phases on the singular vectors, which still isn't a big deal.

If any singular value $\sigma_i$ has multiplicity then we again have more freedom than this: we can again pick any orthonormal bases of the corresponding eigenspaces of $XX^T$ and $X^T X$ for the corresponding singular vectors.

To write down a unique version of SVD we can do the following. Suppose $\sigma_1 > \sigma_2 > \dots$ is a list of the singular values of $X$ ignoring multiplicity. Associated to each $\sigma_i$ is a left singular subspace $U_i = \text{ker}(X X^T - \sigma_i^2)$ and a right singular subspace $V_i = \text{ker}(X^T X - \sigma_i^2)$ with the properties that

  • the $U_i$ give an orthogonal direct sum decomposition of the codomain (by the spectral theorem),
  • the $V_i$ give an orthogonal direct sum decomposition of the domain (by the spectral theorem again), and
  • $\frac{X}{\sigma_i}$ restricts to an isometry from $V_i$ to $U_i$ for each $i$. This is a direct computation: if $X^T Xv = \sigma_i^2 v$ then $(XX^T)(Xv) = \sigma_i^2 (Xv)$ so $v \in V_i \Rightarrow Xv \in U_i$, and $\langle Xv, Xv \rangle = \langle v, X^T Xv \rangle = \langle v, \sigma_i^2 v \rangle = \sigma_i^2 \langle v, v \rangle$.

Write $\varphi_i$ for the linear map which is equal to this isometry on $V_i$ and zero on its orthogonal complement (edit: these are partial isometries). Then

$$X = \sum_i \sigma_i \varphi_i$$

and this decomposition is unique. Writing down an SVD in the usual form from here amounts to choosing orthonormal bases of the left and right singular subspaces such that $\varphi_i$ sends one to the other.

Qiaochu Yuan
  • 468,795