2

Consider the matrix vector equation $Ax=b$ with $$ A = \begin{pmatrix} 1 & 2\\ 2 & 4 \end{pmatrix}, \quad b= \begin{pmatrix} 4\\ 8 \end{pmatrix}. $$ Since the second row of matrix $A$ is a multiple of the first, $A^{-1}$ does not exist and the system has infinitely many solutions. The set of solutions is a line in $\mathbb{R}^2$ which satisfy $$\label{abc} x_1 + 2 x_2 = 4. $$ However, calculating the Pseudo-Inverse $A^+$ of $A$ (using python or similar) is get a unique solution $x^+ = (0.8, 1.6)^T$.

As far as I understood correctly, the Pseudo-Inverse is an explicit way of calculating the least squares solution given by $$ x^+ = \arg\min_x \frac{1}{2} || Ax - b||_2^2. $$ This can be seen by using the singular value decomposition of $A$.

But obviously, all points satisfying $x_1+2x_2=4$ minimize the norm, hence there are infinitely many least squares solutions.

Why does the Pseudo-Inverse then give me a unique solution $x^+=(0.8, 1.6)^T$?

Thanks in advance.

Mittens
  • 46,352
stish
  • 51
  • if $Ax=b$ has a solution, then $x_*=A^+b$ is a particular solution and has the smallest $|;|_2$-norm. See this posting, for example, for more details. – Mittens Dec 30 '22 at 09:04
  • Pseudo-Inverse is used if there is no unique solution i.e. when a solution does not exist or when there are infinitely many solutions. When a solution does not exist, pseudo-inverse gives you a result that minimise L2 norm of the error i.e. roughly what you have in your post. When there are infinitely many solutions, pseudo-inverse gives you solution with the lowest L2 norm. – acat3 Dec 30 '22 at 09:22
  • Thank ou for the link. If I understood that right, my solution satisfies $|| (0.8, 1.6)^T ||_2 \leq || x^\prime ||_2$ for all other $x^\prime$ that lie on the line $x_1 + 2x_2 =4$. But doesn't this mean that $x^+$ is not the solution of the least squares, but rather of some regularized version $x^+ = \arg\min_x \frac{1}{2} || Ax-b||_2^2 + ||x||_2$? Because it no only minimized the error, but also the $||\cdot||_2$-norm. – stish Dec 30 '22 at 09:32
  • @Sim: The regularized problem is not equivalent to the original mean squared problem. $x_+=A^+b$ is indeed a solution to $\operatorname{arg.min}|Ax-b|2$, and amongst all solutions to such minimization problem, $x+$ has minimal norm. – Mittens Dec 30 '22 at 09:41

1 Answers1

2

In general, if $A\in \operatorname{mat}(\mathbb{R}^m,\mathbb{R}^n)$ and $b\in\mathbb{C}^n$, $x_*=A^+b$, where $A^+$ is the Moore-Penrose pseudo inverse, solves two problems simultaneously:

$$x_*=\operatorname{arg.min}\{\|x\|_2: x=\operatorname{arg.min}\|Ax-b\|_2\}$$ where $\|\;\|_2$ is the standard quadratic Euclidean norm, that is, if $x'$ is such that $\|Ax'-b\|_2=\min\{ \|Ay-b\|_2:y\in\mathbb{R}^n\}$, the $\|x_+\|_2\leq\|x'\|_2$.

Mittens
  • 46,352
  • 1
    Thank you, I think this cleared my confusion as it was not clear to me that the Moore-Penrose inverse solves the two minimization problems simultaneously. – stish Dec 30 '22 at 10:21