Derivative of conjugate transpose of matrix

Question

Building off of my previous question, I am trying to derive the normal equations for the least squares problem:

$$ \min_W \|WX - Y\|_2 \\ W \in \mathbb{C}^{N \times N} \quad X, Y \in \mathbb{C}^{N \times M} $$

The intuitive way of viewing this problem is that I am trying to predict a vector $y$ (of length $N$) from a corresponding $x$ vector using a matrix $W$, and to estimate $W$ I have multiple ($M$, to be precise) realizations of $x$ and $y$ packed into matrices $X$ and $Y$.

I am trying to define this in terms of a least-squares problem and derive the normal equations myself, but I'm running into issues in taking the derivative. To spell it out explicitly, I can re-state the above equation as:

$$ \min_W (WX - Y)^H (WX - Y) \\ = \min_W (X^H W^H - Y^H) (WX - Y) \\ = \min_W (X^H W^H W X - Y^H W X - X^H W^H Y + Y^H Y) $$

Now typically I would take the derivative with respect to $W$, set it equal to zero, and solve for $W$. However, my matrix calculus is rusty and everything I know is basically summed up on this webpage, where it explicitly states:

Note that the Hermitian transpose is not used because complex conjugates are not analytic.

Now, because of Michael C. Grant's answer I feel there must be some way of doing this, but I am at a loss as to how. Thank you all in advance!

Answer 1

In the first version of my answer I overlooked the fact that you defined the $L_2$ error in a way that is not suitable for matrices. For vectors it would have been OK, but if your complex error is $E=WX-Y$, then $E^HE$ is not the error that you want to minimize, because $E^HE$ is a matrix (if $E$ were a vector, then $E^HE$ would be a scalar and you could minimize it). The appropriate definition of the (scalar) error measure is given by the Frobenius norm

$$\epsilon=\|E\|_F^2=\|WX - Y\|_F^2=\\ =\text{trace}\left ( E^HE\right)=\text{trace}\left ( (WX-Y)^H(WX-Y)\right)=\\ =\text{trace}\left ( X^HW^HWX - X^HW^HY - Y^HWX +Y^HY\right)$$

Now we can apply the trick of taking the conjugate complex derivative (reference) w.r.t. $W^H$:

$$\frac{\partial\epsilon}{\partial W^H} = WXX^H - YX^H$$

Setting the derivative to zero gives the solution

$$W=YX^H(XX^H)^{-1}$$

assuming $(XX^H)^{-1}$ exists.

EDIT: just an additional simple example to show how the trick with the conjugate derivative works when solving for an unknown complex vector (instead of a matrix). In this case we get a well-known result:

Assume we have an overdetermined system of complex linear equations

$$Ax=b$$

where $A$ is $m\times n$, $m>n$, $x$ is $n\times 1$, and $b$ is $m\times 1$. Minimizing the squared error

$$\epsilon=(Ax-b)^H(Ax-b)=x^HA^HAx - x^HA^Hb-b^HAx+b^Hb$$

by taking the derivative w.r.t. $x^H$ and equating it with zero gives

$$A^HAx-A^Hb=0$$

which leads to the well-known Moore-Penrose pseudoinverse:

$$x=(A^HA)^{-1}A^Hb$$

EDIT 2: Let's have a look at two little examples showing that it is correct to regard a complex variable $z=z_R+iz_I$ as constant when taking the derivative w.r.t. $z^*$. The conjugate derivative is defined as (ref.)

$$\frac{\partial f}{\partial z^*}=\frac{1}{2}\left [\frac{\partial f}{\partial z_R} +i\frac{\partial f}{\partial z_I} \right]$$

For $f=z$ we get

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R+iz_I)}{\partial z_R} +i\frac{\partial (z_R+iz_I)}{\partial z_I} \right]=\frac{1}{2}(1+i^2)=0$$

For $f=zz^*$ we have

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R^2+z_I^2)}{\partial z_R} +i\frac{\partial (z_R^2+z_I^2)}{\partial z_I} \right]=\frac{1}{2}(2z_R+2iz_I)=z$$