Misunderstanding small detail in proof - svd analysis of gradient descent

Question

Does gradient descent converge to a minimum-norm solution in least-squares problems?

In this wonderful answer, the writer writes a proof that says to which value gradient descent will converge.

I'm trying to understand a simple detail.

It is implied that if $A = U\Sigma V^T$ and $y = V^Tx$, then $(I-A^TA)^kx = (I-\Sigma^T\Sigma)^ky$ and I am struggling more than I should be to understand why that is.

Shouldn't it be $ (I-V\Sigma^T\Sigma V^T)^kVy$?

I assume you're referring to my answer. Thank you for the praise. — Rodrigo de Azevedo, Jul 15 '20 at 19:49
I am. Thank you for the great answer. I will try to make full sense of it. — Oria Gruber, Jul 15 '20 at 19:52
By the way, I contacted the corresponding author of the paper on arXiv and she confirmed that, indeed, the claim in the paper is incomplete. — Rodrigo de Azevedo, Jul 15 '20 at 19:53
Why is it incomplete? because of the issue I raised or is there something else at play? — Oria Gruber, Jul 15 '20 at 20:17
If the initial state ${\rm x}_0$ is orthogonal to the null space of $\rm A$ then gradient descent will indeed converge to the least-norm solution. The authors forgot to mention the orthogonality requirement. — Rodrigo de Azevedo, Jul 15 '20 at 20:25
Ah I knew that. I thought the proof was to show what it converges to, regardless of additional assumptions. Thank you so much. — Oria Gruber, Jul 15 '20 at 20:29

score 2 · Accepted Answer · answered Jul 15 '20 at 19:47

What you are trying to show is false, but you can get something close then you get the desired matrix $(I-\Sigma^T\Sigma)^k$ by considering the iteration this definition is used to construct.

$$ \begin{aligned} (I-A^TA)^kx &= (I-V\Sigma^T\Sigma V^T)^kx \\ &= (VV^T-V\Sigma^T\Sigma V^T)^kx := (VDV^T)^kx, \end{aligned} $$ where $D = I-\Sigma^T\Sigma$. $$ (VDV^T)^kx = VD^kV^Tx = VD^ky. $$

I believe this is enough to proceed with the proof in the answer. Since you are forming an iteration between the $y$'s, you multiply both sides of the $x$ iteration by $V^T$, which cancels out the $V$, which yields the desired result.

Misunderstanding small detail in proof - svd analysis of gradient descent

1 Answers1