How is $\nabla_X \operatorname{Tr}\left\{ X^T A X B\right\} = AXB + A^TXB^T$?

Question

Given the matrices $A$ and $B$,

$$ \nabla_X \operatorname{Tr} \left\{ X^T A X B \right\} = AXB + A^TXB^T $$

How?

Answer 1

With implicit summation over repeated indices,

$$\frac{\partial\operatorname{tr}X^T AXB}{\partial X_{ij}}=A_{lm}B_{nk}\frac{\partial}{\partial X_{ij}}(X_{lk}X_{mn})=A_{lm}B_{nk}(\delta_{il}\delta_{jk}X_{mn}+X_{lk}\delta_{im}\delta_{jn}),$$where $\delta_{rs}$ is the Kronecker delta ($1$ if $r=s$, $0$ is $r\neq s$). The right-hand side is $$A_{im}X_{mn}B_{nj}+A_{li}X_{lk}B_{jk}=(AXB+A^T XB^T)_{ij}.$$

Answer 2

Let $A=\left(a_{ij}\right)_{i,j}$ be the entry-wise form of the matrix $A$, with $a_{ij}$ being the $\left(i,j\right)$-th entry of $A$.

With this notation, $\text{tr}\left(X^{\top}AXB\right)$ can be written as $$ \text{tr}\left(X^{\top}AXB\right)=\sum_{j,k,l,m}x_{jm}a_{jk}x_{kl}b_{lm}=\sum_{j,k,l,m}a_{jk}b_{lm}x_{jm}x_{kl}. $$ Thus \begin{align} \frac{\partial}{\partial x_{rs}}\text{tr}\left(X^{\top}AXB\right)&=\frac{\partial}{\partial x_{rs}}\left(\sum_{j,k,l,m}a_{jk}b_{lm}x_{jm}x_{kl}\right)\\ &=\sum_{j,k,l,m}\frac{\partial}{\partial x_{rs}}\left(a_{jk}b_{lm}x_{jm}x_{kl}\right)\\ &=\sum_{j,k,l,m}a_{jk}b_{lm}\frac{\partial}{\partial x_{rs}}\left(x_{jm}x_{kl}\right)\\ &=\sum_{j,k,l,m}a_{jk}b_{lm}\left(\frac{\partial x_{jm}}{\partial x_{rs}}x_{kl}+x_{jm}\frac{\partial x_{kl}}{\partial x_{rs}}\right)\\ &=\sum_{j,k,l,m}a_{jk}b_{lm}\left(\delta_{jr}\delta_{ms}x_{kl}+x_{jm}\delta_{kr}\delta_{ls}\right)\\ &=\sum_{j,k,l,m}a_{jk}b_{lm}\delta_{jr}\delta_{ms}x_{kl}+\sum_{j,k,l,m}a_{jk}b_{lm}x_{jm}\delta_{kr}\delta_{ls}\\ &=\sum_{k,l}a_{rk}b_{ls}x_{kl}+\sum_{j,m}a_{jr}b_{sm}x_{jm}\\ &=\sum_{k,l}a_{rk}x_{kl}b_{ls}+\sum_{j,m}a_{jr}x_{jm}b_{sm}\\ &=\left(AXB+A^{\top}XB^{\top}\right)_{r,s}. \end{align}

Answer 3

Let

$$f (\mathrm X) := \mbox{tr} \left( \mathrm X^\top \mathrm A \,\mathrm X \,\mathrm B \right)$$

Hence, the directional derivative of $f$ in the direction of $\rm V$ at $\rm X$ is

$$\begin{array}{rl} \displaystyle\lim_{h \to 0} \dfrac{f (\mathrm X + h \mathrm V) - f (\mathrm X)}{h} &= \mbox{tr} \left( \mathrm V^\top \mathrm A \,\mathrm X \,\mathrm B \right) + \mbox{tr} \left( \mathrm X^\top \mathrm A \,\mathrm V \,\mathrm B \right)\\ &= \mbox{tr} \left( \mathrm V^\top \mathrm A \,\mathrm X \,\mathrm B \right) + \mbox{tr} \left( \mathrm B \,\mathrm X^\top \mathrm A \,\mathrm V \right)\\ &= \langle \mathrm V, \mathrm A \,\mathrm X \,\mathrm B \rangle + \langle \left( \mathrm A^\top \mathrm X \,\mathrm B^\top \right)^\top,\mathrm V \rangle\\ &= \langle \mathrm V, \mathrm A \,\mathrm X \,\mathrm B \rangle + \langle \mathrm A^\top \mathrm X \,\mathrm B^\top,\mathrm V \rangle\\ &= \langle \mathrm V, \mathrm A \,\mathrm X \,\mathrm B \rangle + \langle \mathrm V, \mathrm A^\top \mathrm X \,\mathrm B^\top \rangle\\ &= \langle \mathrm V, \mathrm A \,\mathrm X \,\mathrm B + \mathrm A^\top \mathrm X \,\mathrm B^\top \rangle\end{array}$$

where the cyclic property of the trace and the Frobenius inner product were used. Thus,

$$\nabla_{\mathrm X} f (\mathrm X) = \color{blue}{\mathrm A \,\mathrm X \,\mathrm B + \mathrm A^\top \mathrm X \,\mathrm B^\top}$$

---

matrix-calculus scalar-fields gradient

Answer 4

Frobenius inner product, or matrix version of inner product, is the best notational tool to use to prove identities involving derivatives of traces like this. Instead of starting with limit definition of directional derivative as in Rodrigo de Azevedo's answer, we can take derivative directly into Frobenius inner product.

Defining the Frobenius inner product $$f=X\cdot AXB={\rm Tr}(X^TAXB),$$ we have $$\begin{align} df&=dX\cdot AXB+X\cdot A\,dX\,B\tag{1}\\ &=AXB\cdot dX+A^TXB^T\cdot dX\tag{2}\\ &=(AXB+A^TXB^T)\cdot dX\tag{3},\\ \end{align}$$ which in turn implies $$\frac{\partial f}{\partial X}=AXB+A^TXB^T.\tag{4}$$ In the above derivation, (1) is product rule of derivative, (2) is Hermitian (commutative in real) property of Frobenius inner product and cyclic property of trace, (3) is linearity of inner product and finally (4) is expansion from derivative wrt an element to derivative wrt a matrix.