64

This may be a trivial question yet I was unable to find an answer:

$$\left \| A \right \| _2=\sqrt{\lambda_{\text{max}}(A^{^*}A)}=\sigma_{\text{max}}(A)$$

where the spectral norm $\left \| A \right \| _2$ of a complex matrix $A$ is defined as $$\text{max} \left\{ \|Ax\|_2 : \|x\| = 1 \right\}$$

How does one prove the first and the second equality?

Rodrigo de Azevedo
  • 23,223
mathemage
  • 850
  • 1
  • 7
  • 15
  • What are your thoughts on the second equality? – Git Gud Nov 30 '13 at 11:35
  • @GitGud Oh, does that mean singular values are defined in this way, i.e., the second equality is a definition? – mathemage Nov 30 '13 at 12:23
  • It's not a definition. It's a direct consequence of the definition. Can you see this? – Git Gud Nov 30 '13 at 12:26
  • http://math.stackexchange.com/questions/482170/2-norm-vs-operator-norm – Prahlad Vaidyanathan Nov 30 '13 at 17:11
  • @GitGud One normally can't if one does not know about the Courant-Fischer's characterisation. – Algebraic Pavel Dec 01 '13 at 02:37
  • Courant-Fischer is not needed. The singular values of $A$ are the square roots of the eigenvalues of $A^*A$ and since the function $\sqrt{, \cdot, }\colon[0, +\infty[\to [0, +\infty[$ is increasing, the second equality follows imediately. – Git Gud Dec 01 '13 at 08:07
  • This is only true if the matrix is symmetric. Otherwise, we could have our maximum eigenvalue be less than the spectral norm: https://math.stackexchange.com/questions/2719868/operator-norm-is-equal-to-max-eigenvalue – information_interchange Mar 27 '20 at 19:30

2 Answers2

52

First of all, $$\begin{align*}\sup_{\|x\|_2 =1}\|Ax\|_2 & = \sup_{\|x\|_2 =1}\|U\Sigma V^Tx\|_2 = \sup_{\|x\|_2 =1}\|\Sigma V^Tx\|_2\end{align*}$$ since $U$ is unitary, that is, $\|Ux_0\|_2^2 = x_0^TU^TUx_0 = x_0^Tx_0 = \|x_0\|_2^2$, for some vector $x_0$.

Then let $y = V^Tx$. By the same argument above, $\|y\|_2 = \|V^Tx\|_2 = \|x\|_2 = 1$ since $V$ is unitary. $$\sup_{\|x\|_2 =1}\|\Sigma V^Tx\|_2 = \sup_{\|y\|_2 =1}\|\Sigma y\|_2$$ Since $\Sigma = \mbox{diag}(\sigma_1, \cdots, \sigma_n)$, where $\sigma_1$ is the largest singular value. The max for the above, $\sigma_1$, is attained when $y = (1,\cdots,0)^T$. You can find the max by, for example, solving the above using a Lagrange Multiplier.

Martin Sleziak
  • 56,060
yulunz
  • 621
49

Put $B=A^*A$ which is a Hermitian matrix. A linear transformation of the Euclidean vector space $E$ is Hermite iff there exists an orthonormal basis of E consisting of all the eigenvectors of $B$. Let $\lambda_1,...,\lambda_n$ be the eigenvalues of $B$ and $\left \{ e_1,...e_n \right \}$ be an orthonormal basis of $E$. Denote by $\lambda_{j_{0}}$ to be the largest eigenvalue of $B$.

For $x=a_1e_1+...+a_ne_n$, we have $\left \| x \right \|=\left \langle \sum_{i=1}^{n}a_ie_i,\sum_{i=1}^{n}a_ie_i \right \rangle^{1/2} =\sqrt{\sum_{i=1}^{n}a_i^{2}}$ and $Bx=B\left ( \sum_{i=1}^{n}a_ie_i \right )=\sum_{i=1}^{n}a_iB(e_i)=\sum_{i=1}^{n}\lambda_ia_ie_i$. Therefore:

$\left \| Ax \right \|=\sqrt{\left \langle Ax,Ax \right \rangle}=\sqrt{\left \langle x,A^*Ax \right \rangle}=\sqrt{\left \langle x,Bx \right \rangle}=\sqrt{\left \langle \sum_{i=1}^{n}a_ie_i,\sum_{i=1}^{n}\lambda_ia_ie_i \right \rangle}=\sqrt{\sum_{i=1}^{n}a_i\overline{\lambda_ia_i}} \leq \underset{1\leq j\leq n}{\max}\sqrt{\left |\lambda_j \right |} \times (\left \| x \right \|)$

So, if $\left \| A \right \|$ = $\max \left\{ \|Ax\| : \|x\| = 1 \right\}$ then $\left \| A \right \|\leq \underset{1\leq j\leq n}\max\sqrt{\left |\lambda_j \right |}$. (1)

Consider: $x_0=e_{j_{0}}$ $\Rightarrow \left \| x_0 \right \|=1$ so that $\left \| A \right \|^2 \geq \left \langle x_0,Bx_0 \right \rangle=\left \langle e_{j_0},B(e_{j_0}) \right \rangle=\left \langle e_{j_0},\lambda_{j_0} e_{j_0} \right \rangle = \lambda_{j_0}$. (2)

Combining (1) and (2) gives us $\left \| A \right \|= \underset{1\leq j\leq n}{\max}\sqrt{\left | \lambda_{j} \right |}$ where $\lambda_j$ is the eigenvalue of $B=A^*A$

Conclusion: $$\left \| A \right \| _2=\sqrt{\lambda_{\text{max}}(A^{^*}A)}=\sigma_{\text{max}}(A)$$

user1868607
  • 6,243
An Khuong Doan
  • 2,155
  • Why is $A^*A$ a unitary matrix? For $A$ as a zero matrix or a general real diagonal matrix, this doesn't have to be unitary, right? – mathemage Nov 30 '13 at 16:29
  • 2
    That's my mistake. That should be Hermite. Got fixed! – An Khuong Doan Nov 30 '13 at 16:58
  • I think I understood the proof except for one part: in the "iff" statement, why does the reverse implication hold? For any unitary $U$ and diagonal non-Hermitian $\Lambda$ the matrix $B = U^\dagger \Lambda U$ has eigenvectors forming orthonormal basis (namely, columns of $U$). However, $B$ is not Hermitian since $\Lambda$ is not. This is a counterexample, right? – mathemage Dec 01 '13 at 19:09
  • How can we say (1)? It seems a jump between (1) and previous statement. – sleeve chen Jan 20 '15 at 23:55
  • 2
    And (2) is still blur to me: how can we say $$\left | A \right | \geq \left \langle x,Bx \right \rangle=\left \langle e_{j_0},B(e_{j_0}) \right \rangle$$ – sleeve chen Jan 21 '15 at 00:01
  • 1
    @sleevechen for (1) we can say it because $\left | A \right |$ is the maximum of all the elements in the set, and it was just shown that each element is $\leq $ the largest singular value times $\left | x \right | = 1$. For (2), since $\left | A \right |$ is the maximum, it must be $\geq$ than any particular element, namely $\left | A {e_j}_0 \right |=\left | A{e_j}_0\right |=\left \langle A{e_j}_0,A{e_j}_0 \right \rangle=\left \langle {e_j}_0,A^*A{e_j}_0 \right \rangle=\left \langle {e_j}_0,B{e_j}_0 \right \rangle$ – ignoramus Jun 08 '16 at 11:40
  • Is there any point in writing the modulus around the eigenvalue? Because A*A is positive semi-definite (please correct me if I'm wrong). – Arkya Feb 14 '17 at 16:59
  • 1
    @AnKhuongDoan, Could you explain why on page 16 in http://www.math.ucla.edu/~wotaoyin/summer2016/4_proximal.pdf it says: "note: the spectral norm returns the square root of the max eigenvalue of $ {A}^{T} A $, may not equal the max singular value for asymmetric matrices."? – Royi Sep 25 '17 at 08:17
  • The statement is misleading; for any matrix A, the spectral norm is the max singular value. And we get this max singular value from exactly finding the eigenvalues of $A^TA$ , which are the same as $AA^T$. Although this may be different for complex matrices – information_interchange Mar 28 '20 at 04:27