Solving for a matrix from its quadratic form

Question

I have a set of vectors that I am trying to predict from another set of vectors using a matrix $W$. To find this matrix, I decide I want to minimize the $\ell^2$ norm of the error, e.g.:

$$ \text{find} \min_W \|y - Wx\|_2 \\ x,y \in \mathbb{C}^N \quad W\in \mathbb{C}^{N \times N} $$

Where $x$ and $y$ are respective vectors from their sets, and each pair is (I hypothesize) related to eachother by $W$. I start by expanding this out:

$$ \min_W \left[ (y - Wx)^H (y - Wx) \right] \\ = \min_W \left[ (y - Wx)^H y - (y - Wx)^H Wx \right] \\ = \min_W \left[ (y^H y - y^H Wx)^H - (x^H W^H y - x^H W^H Wx)^H \right] \\ = \min_W \left[ y^H y - x^H W^H y - y^H W^H x + x^H W^H W x \right] $$

Where $(\cdot)^H$ denotes Hermitian transpose. I want to take the derivative with respect to $W$ and set it equal to zero, but I'm having a hard time with the derivative as I'm not sure exactly how that works with Hermitian transposes. Taking a look at this page, it looks like I might be in trouble (e.g. I'm going about this the wrong way), but I wanted to pick your brains to see if you all had any ideas on how to move forward from here.

Thank you all!

EDIT Michael C. Grant has pointed out that this question is underdetermined, and he is right, if I only have a single $x$ and $y$. However I have a set of $x$ and $y$ vectors that I assume are related to eachother by $W$ (And since right now I am working with simulations, I can generate as many pairs of $x$'s and $y$'s as I want). This is a kind of "system identification" problem, where I have input-output correspondences and I'm trying to understand how the math is derived.

Answer 1

Yes, I'd suggest that you are in trouble, but not for the reason you may think. :-) You can certainly take the derivative of that expression with respect to $W$.

But more fundamentally, your problem is underdetermined: you have $2N$ complex quantities $x$ and $y$ that you are using to construct a matrix $W$ with $N^2$ complex quantities.

If $x\neq 0$, the minimum value of the norm is zero, and any matrix that satisfies $Wx=y$ is going to attain this minimum value. For instance, $W=(x^Hx)^{-1}yx^H$ will do the trick. But you can add any other matrix $V$ satisfying $Vx=0$ to $W$ and achieve the same result.

If $x=0$, then the minimum value of the norm is $\|y\|$, and any $W$ will achieve it.

EDIT: OK, now we see the problem. What you really want to do is something like this:

$$\begin{array}{ll}\text{minimize}_W & \| Y - W X \|_F\end{array}$$ where $X,Y\in\mathbb{C}^{N\times M}$, with $M\geq N$. This is actually a standard least squares problem, though it may not seem so since it's expressed in matrix form. Using Kronecker products, we can write this as $$\begin{array}{ll}\text{minimize}_W & \| \mathop{\textrm{vec}}(Y) - (X^H\otimes I) \mathop{\textrm{vec}}(W) \|_2\end{array}$$ where $\otimes$ denotes the Kronecker product, and $\mathop{\textrm{vec}}$ stacks the columns of its matrix argument on top of one another. The normal equations for this least squares problem are $$ \begin{aligned} &(X^H\otimes I)^H (X^H\otimes I) \mathop{\textrm{vec}}(W) = (X^H\otimes I)^H \mathop{\textrm{vec}}(Y) \\ &\qquad \Longrightarrow\quad (X\otimes I) (X^H\otimes I) \mathop{\textrm{vec}}(W) = (X\otimes I) \mathop{\textrm{vec}}(Y) \\ &\qquad \Longrightarrow\quad (XX^H\otimes I) \mathop{\textrm{vec}}(W) = (X\otimes I) \mathop{\textrm{vec}}(Y) \\ &\qquad \Longrightarrow\quad WXX^H = YX^H \end{aligned} $$ Therefore, if $XX^H\in\mathbb{C}^{N\times N}$ is invertible---and note that's an important condition that is not guaranteed to be satisfied even if $M\geq N$---then the solution is $$W = YX^H(XX^H)^{-1}$$ The similarity to the single-vector case is not too surprising. If $XX^H$ is singular, then $$W=YX^H(XX^H)^\dagger$$ is a minimizer, where $(XX^H)^\dagger$ is the pseudoinverse of $XX^H$.