Issue for applying penalty method to find the minimum distance between point $y$ and $a^Tx=b$ in $\mathbb{R}^n$

Question

According to Approach0, this question is new.

I am practicing problems, no solutions, for my optimization course (not homework, just extra practices).

Problem:

Apply penalty method to find the minimum distance between point $y$ and $a^Tx=b$ in $\mathbb{R}^n$

My attempt:

Penalty function $$\phi(x,c)=\|x-y\|^2+c(a^Tx-b)^2$$Thus$$\frac{\partial\phi}{\partial x}=2x-2y+2c(a^Tx-b)a=2x-2y+2c(aa^T)x-2(cb)a$$So by setting it equals to $0$, we get $$\begin{align} (I+c(aa^T))x&=y+(cb)a\\ \implies x&=(I+c(aa^T))^{-1}(y+(cb)a)\\ &=(\frac{1}{c}I+aa^T)^{-1}(\frac{y}{c}+b\times a)\\ &\to b(aa^T)^{-1}a\tag{As $c\to\infty$}\quad\quad(1) \end{align}$$ However the optimal solution is $b(aa^T)^{-1}a$ from my calculation, but it has no relation with point $y$. And I am not sure if $(1)$ is correct. Thanks!

Are you sure you correctly took the derivative of $||x-y||^2 = (x^T-y^T)(x-y)$? — Greg Martin, Sep 25 '24 at 07:40
I think I got it correct, from here, replace $x$ by $x-y$, then it is $2(x-y)$ — lIlIlIlIlIlIlIlIlI, Sep 25 '24 at 07:43
@user317176 I am not sure if it is correct, because I don't have solutions. Could you please explain more about it? — lIlIlIlIlIlIlIlIlI, Sep 25 '24 at 07:48
It is the generalization of the distance formula from Algebra II. — user317176, Sep 25 '24 at 07:49
@user317176 Ah yes, I nearly forgot it. However I want to solve it using penalty method. — lIlIlIlIlIlIlIlIlI, Sep 25 '24 at 07:51
@YvesDaoust Why? $(a^Tx)a=\begin{pmatrix}a_1(a_1x_1+\cdots+a_nx_n)\\vdots\ a_n(a_1x_1+\cdots+a_nx_n)\end{pmatrix}=\begin{pmatrix}a_1^2x_1+\cdots+a_na_1x_n\\vdots\ a_na_1x_1+\cdots+a_n^2x_n\end{pmatrix}=\begin{pmatrix}a_1^2&\cdots&a_na_1\\vdots&&\vdots\ a_na_1&\cdots&a_n^2\end{pmatrix}x=(aa^T)x$. Here is the details. — lIlIlIlIlIlIlIlIlI, Sep 25 '24 at 09:16

River Li · Accepted Answer · 2024-12-15T15:10:52.973

The issue is that it is not true to say $$\left(\frac{1}{c}I + aa^\top\right)^{-1} \left(\frac{1}{c}y + ba\right) \to b(aa^\top)^{-1}a, \quad \mathrm{as}\,\, c \to \infty,$$ since $aa^\top$ is not invertible when $n\ge 2$. For example, set $$a = \begin{pmatrix} -1\\ 3 \end{pmatrix}, \quad y = \begin{pmatrix} 1\\ 2 \end{pmatrix}, \quad b = 4. $$ We have $$\left(\frac{1}{c}I + aa^\top\right)^{-1} \left(\frac{1}{c}y + ba\right) = \begin{pmatrix} \frac{11c+1}{10c+1}\\[4pt] \frac{17c+2}{10c+1} \end{pmatrix} \to \begin{pmatrix} \frac{11}{10}\\[4pt] \frac{17}{10} \end{pmatrix}$$ which is the desired result.

The right way to take limit is given as follows. We have \begin{align*} x &= (I + c aa^\top)^{-1}(y + cba)\\ &= \left(I - \frac{caa^\top}{1 + ca^\top a }\right)(y + cba)\\ &= y + cba - \frac{ca^\top y}{1 + ca^\top a }a - \frac{c^2ba^\top a}{1 + ca^\top a } a\\ &= y + \frac{cb}{1 + ca^\top a }a - \frac{ca^\top y}{1 + ca^\top a }a \\ & \to y + \frac{b}{a^\top a}a - \frac{a^\top y}{a^\top a}a, \quad \mathrm{as}\,\, c \to \infty. \end{align*}

By the way, we can obtain the solution $x(c)$ without calculus. We have $$\phi(x,c) = x^\top x + y^\top y - 2y^\top x + cx^\top aa^\top x + cb^2 - 2cb a^\top x$$ $$= x^\top(I + caa^\top)x - 2(y + cba)^T x + y^\top y + cb^2$$ $$= (x - x_0)^\top (I + caa^\top)(x - x_0) - x_0^\top (I + caa^\top) x_0 + y^\top y + cb^2$$ $$\ge - x_0^\top (I + caa^\top) x_0 + y^\top y + cb^2$$ where $x_0 = (I + caa^\top)^{-1} (y + cba)$.

Issue for applying penalty method to find the minimum distance between point $y$ and $a^Tx=b$ in $\mathbb{R}^n$

1 Answers1