Showing a function is Frechet Differentiable？

Question

I just started learning the Frechet Derivatives. So I have a function $H:\mathbb{R}^{N\times n}\to\mathbb{R}^{N\times n}$, i.e. $U^T\in\mathbb{R}^{N\times n}$ and $$H(U^T)=GW\times (F(U))^T+S\times U^T+C$$ with $G,W,S\in \mathbb{R}^{N \times N}$ are two matrices of size $N\times N$, $F(\cdot)\in \mathbb{R}^n\to\mathbb{R}^n $ is a nonlinear function which maps each column vetor of $U$ to the corresponding column vector of $F(U)$, and $C\in\mathbb{R}^{N\times n}$.

My question is what property should the nonlinear unknown function $F(\cdot)$ satisfy to ensure the function $H(\cdot)$ is Frechet differentiable? What does the Frechet derivative matrix looks like? What should I start?Thank you!

Answer 1

I'll rewrite the definition of $H$ as $$ H(X) = GW F(X^T)^T + SX + C. $$ Let's assume that $F$ is Frechet differentiable at a particular point $X^T$, so that $$ F(X^T + \Delta X^T) = F(X^T) + F'(X^T) \Delta X^T + e(\Delta X), $$ and the error term $e(\Delta X)$ satisfies $$ \lim_{\Delta X \to 0} \frac{\|e(\Delta X)\|}{\| \Delta X \|} = 0. $$ Notice that \begin{align} H(X + \Delta X) &= GW F(X^T + \Delta X^T)^T + SX + S\Delta X + C \\ &= GW \left( F(X^T) + F'(X^T) \Delta X^T + e(\Delta X) \right)^T + SX + S \Delta X + C \\ &= \underbrace{GWF(X^T)^T + SX + C}_{H(X)} + \underbrace{GW(F'(X^T) \Delta X^T)^T + S \Delta X}_{H'(X) \Delta X} + \underbrace{GW e(\Delta X)^T}_{\text{small}}. \end{align}

Comparing this with the equation $$ H(X + \Delta X) \approx H(X) + H'(X) \Delta X $$ suggests that $H$ is differentiable at $X$ and that $H'(X)$ is the linear transformation defined by $$ \tag{1} H'(X) \Delta X = GW(F'(X^T) \Delta X^T)^T + S \Delta X. $$ To prove that this is true, we only need to show that $$ \tag{2} \lim_{\Delta X \to 0} \frac{\| GW e(\Delta X)^T \|}{ \| \Delta X \|} = 0 $$ To establish (2), let $L$ be the linear transformation defined by $$ L(v) = GW v^T. $$ Then \begin{align} \frac{\| GW e(\Delta X)^T \|}{ \| \Delta X \|} &= \frac{\| L(e(\Delta X)) \|}{\| \Delta X \|} \\ &\leq \frac{\| L \| \|e(\Delta X) \|}{\| \Delta X \|} \end{align} which approaches $0$ as $\Delta X \to 0$.

In order to reach the conclusion that $H$ is differentiable at $X$, we needed to assume that $F$ is differentiable at $X^T$.

I don't see a simpler way to express $H'(X)$, but maybe somebody else will.

Answer 2

They gave you the hand-wavy proof above. I am giving you the high-end proof now.

Recall from basic differential calculus:

Basic differentiation algebra: the derivative acts linearly $(f+ \alpha g)'(x) = f'(x) + \alpha g'(x)$; the derivative of constant functions is zero; and the derivative of continuous linear functions are themselves $f'(x) \cdot h = f(h)$ whenever $f$ is linear and continuous.

Chain rule: if $g$ and $f$ are two functions defined on open subsets of normed vector spaces such that $f$ is differentiable at $x$ and $g$ is differentiable at $f(x)$ then the composite function $g \circ f$ is differentiable at $c$ and its derivative is the composite of the derivatives $$(g \circ f)'(x) = g'(f(x)) \circ f'(x).$$ Abridged proof. Write $y = f(x)$ and $f(x + h) = f(x) + \underbrace{f'(x) \cdot h + o(h)}_k$ and $$g(y + k) = g(y) + g'(y) k + o(k) = g(y) + g'(y) \cdot f'(x) \cdot h + \underbrace{g'(y) o(h) + o(k)}_{o(k)}. \square$$

To your exercise. The function $H$ is differentiable at every $U$ where the function $F$ is differentiable as well.

Proof. The functions $\varphi:V \mapsto GW V^\intercal$ and $\psi = U \mapsto SU^\intercal$ are linear while the function $U \mapsto C$ is contant. Therefore, the function $H = \varphi \circ F + \psi + C$ will be differentiable at all points where $F$ is differentiable (by the chain rule) and its derivative is simply $$H'(U) = \varphi'(U) \circ F'(U)^\intercal + \psi'(U) = \varphi \circ F'(U) + \psi.$$

If you are dealing with finite dimensional vector spaces, find bases of each so that (by denoting $[ \cdot ]$ the matrix represantion) we get $$[H'(U)]=[\varphi][F'(U)]^\intercal + [\psi]. \square$$

Ammend. If the function $\varphi$ is invertible then the differentiability of $H$ implies that of $F$ for we can write $F = \varphi^{-1} \circ (H - \psi - C),$ and $\varphi^{-1}$ being linear and continuous, it is differentiable. $\square$