The gradient of an element wise matrix function

Question

Let $F = F[A]$ represent an element wise function $F$ applied to matrix $A$. Here $F_{ij} = f(A_{ij})$ where $f$ is a scalar function. I would like to derive an expression for $\frac{\partial F}{\partial A}$.

My strategy was to use summation notation:

$$\frac{\partial F_{ij}}{\partial A_{pq}} = \frac{\partial f}{\partial A_{ij}} \frac{\partial A_{ij}}{\partial A_{pq}} $$

$$\frac{\partial F_{ij}}{\partial A_{pq}} = \frac{\partial f}{\partial A_{ij}} \delta_{ip} \delta_{jq}$$

I know there should be 4th order tensor result but the implied sum is throwing me off. I am not too familiar with matrix manipulation when there are tensors of order 3 and higher so I did not try to construct a differential.

Any walkthroughs/strategies would be much appreciated!

Answer 1

Given a function $f=f(\alpha)$ of a scalar argument $\alpha$, its derivative $f'(\alpha)=\frac{df}{d\alpha}$ can be used to write the differential as $$\eqalign{ df &= f'\,d\alpha \\ }$$ Applying these functions element-wise to a matrix argument $A$ defines the matrices $$\eqalign{ F &= f(A) \\ F' &= f'(A) \\ }$$ Write matrix differential using the element-wise product $\odot$ (aka the Hadamard product). $$\eqalign{ dF &= F'\odot dA \\ }$$ Introduce a sixth-order tensor ${\cal S}$ with components $$\eqalign{ {\cal S}_{ijklmn} &= \begin{cases} 1\quad{\rm if}\;(i=k=m)\;{\rm and}\;(j=l=n) \\ 0\quad{\rm otherwise} \end{cases} \\ }$$ Note that this tensor is unchanged under permutation of its the three index-pairs, so $${\cal S}_{ij\,kl\,mn} = {\cal S}_{kl\,mn\,ij} = {\cal S}_{ij\,mn\,kl}$$ The Hadamard product between two matrices can now be written in index notation. $$\eqalign{ C &= A\odot B \quad&\implies\quad &C_{kl} = A_{ij}{\cal S}_{ijklmn}B_{mn} \\ &= A:{\cal S}:B \quad&&\big({\rm Double\,Dot\,Products}\big) \\ }$$ Apply this to the differential relationship and solve for the gradient $\Gamma$ (a fourth-order tensor). $$\eqalign{ dF &= F':{\cal S}:dA \\ \frac{\partial F}{\partial A} &= F':{\cal S} \;\triangleq\; \Gamma \\ &= {\cal S}:F' \quad\big({\rm Permute\,Indices\,of\,}{\cal S}\big) \\ \frac{\partial F_{ij}}{\partial A_{kl}} &= {\cal S}_{ijklmn}\,F'_{mn} \;=\; \Gamma_{ijkl} \\ \\ }$$ NB: This answer uses a convention wherein repeated indices are (implicitly) summed over.