I'm having a bit of trouble proving the following:
Let $A \in \mathbb{R}^{m\times n}$ and $b \in \mathbb{R}^{m}$ and define $f: \mathbb{R}^n \to \mathbb{R}$ by $f(x) = ||Ax - b||^2$. How can I show that $x_*$ is a global minimizer for $f$ if and only if $x_*$ solves $A^TAx = A^Tb$?
I would appreciate any help proving this!
To show it you basically need two component
to show the least square solution is a local minimizer. To show it we could show the gradient vanishes. Since the gradient $$ \partial_x f(x)= 2A^TAx-2A^Tb $$ This is satisfied directly at $x_*$
to show the $f(x)$ is a convex function. We can show the Hessian is constant and positive definite $$ \partial_{xx} f(x)= 2A^TA $$ Thus $f(x)$ is convex, and the critical point $x_*$ is also a local minimizer.
For convex function all local minimizer is also global minimizer, QED.
