0

Frobenius norm differentiation $X\in R^n$

\begin{align} f(x) &= \tfrac12\|X^TX-A\|^2_F=\tfrac12\langle X^TX-A,X^TX-A \rangle \\[1ex] df(x) &= \tfrac12d\bigl(\langle X^TX-A,X^TX-A \rangle\bigl)=\tfrac12\langle d(X^TX-A),d(X^TX-A) \rangle\\ & = \tfrac 42\bigl(\bigl\langle \langle X,dX\rangle \langle X,dX \rangle\bigr\rangle\bigr) =2(X^Tdx)^TX^Tdx=2dxXX^Tdx=2(X^TXdx)^Tdx \end{align}

Where $\langle x,y \rangle$ is the scalar multiplication.

I don't actually know how to do matrix/vector/norm differentiation. What I come up with is the above. Please, can you help me to differentiate the expression?

Bernard
  • 179,256
user13
  • 1,739

1 Answers1

2

The Frobenius product is defined as $$A:B = {\rm Tr}(A^TB)$$ The properties of the trace allow terms in such a product to be rearranged in various ways. $$\eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ A:BC &= AC^T:B \;=\; B^TA:C \;=\; etc \\ }$$ This product makes it easy to calculate differentials and gradients involving the Frobenius norm. $$\eqalign{ \phi &= \tfrac{1}{2}\|M\|_F^2 = \tfrac{1}{2}M:M \\ d\phi &= \tfrac{1}{2}\big(M:dM+dM:M\big) = M:dM \\ }$$ For your current problem, set $M=(X^TX-A)$ $$\eqalign{ d\phi &= M:dM \\ &= M:(X^TdX+dX^TX) \\ &= XM:dX + MX^T:dX^T \\ &= XM:dX + XM^T:dX \\ &= X(M+M^T):dX \\ \frac{\partial\phi}{\partial X} &= X(M+M^T) \\ \\ }$$ Another general rule to learn is the differential of a product, $$d(A\star B) = dA\star B + A\star dB$$ where $\star$ can be nearly any product you are likely to encounter, e.g. Frobenius, Kronecker, Hadamard, Dyadic, Matrix, Tensor.

And the quantites $(A,B)$ can be any scalar, vector, matrix, or tensor pair which are dimensionally compatible with said product. But you must maintain their relative order, i.e. $A$ before $B$.

greg
  • 40,033
  • So, $X^TdX=X$ and $dX^TX=X^TdX^T$? Did I understand it right? If so can yo explain why they are equal to those values? – user13 Oct 04 '19 at 21:10
  • The rule for rearranging a Frobenius product says $M:X^TdX=XM:dX$. This is not the same as saying $X^TdX=X$, which is absolutely false. – greg Oct 04 '19 at 21:37