Let $F$ be an arbitrary real square matrix, $C=F^{\sf T}F$, and $F_2 = FF$. Is there any way to express $$\frac{\partial F}{\partial C}$$ or $$\frac{\partial F}{\partial F_2}$$ analytically?
If it helps the answer can be specialized to $3\times3$ matrices. The case of $1\times1$ matrices is solvable by substitution, but I don't see the basic idea how to generalize this.
Part 1. $\frac{\partial F}{\partial C}$ is defined only if $F$ is a function of $C$, that is if we can define a bijection $F\rightarrow C=F^TF$. Note that $C$ is a symmetric $\geq 0$ matrix ($C\in S⁺$).
The simplest method is to consider the function $f:C\in S⁺\rightarrow \sqrt{C}\in S⁺$.
More generally, we can choose an orthogonal matrix $O$ (fixed) and consider the function $g:C\in S⁺\rightarrow O\sqrt{C}$.
These functions have a derivative when $C$ is invertible (cf. my post in Derivative (or differential) of symmetric square root of a matrix)
$Df_C(H)=\int_0^{\infty}e^{-t\sqrt{C}}He^{-t\sqrt{C}}dt$
$Dg_C(H)=ODf_C(H)$ where $H$ is a symmetric matrix.
EDIT 1.
Part 2. For the second function $F^2$, let $Z=\{C\in GL_n(\mathbb{R}); spectrum(C)\subset \mathbb{R}^+\setminus\{0\}\}$. Then we can define a function $f:C\in Z\rightarrow f(C)\in Z$ with $f(C)^2=C$. Indeed, there is a unique matrix $X\in Z$ s.t. $X^2=C$.
EDIT 2. Let $Z^+=\{C\in Z; \text{the eigenvalues of } $C$ \text{ are distinct}\}$. Since $Z^+$ is open, its tangent space is $M_n(\mathbb{R})$ and, again, the derivative of $f:C\in Z^+\rightarrow f(C)\in Z^+$ is:
$Df_C(H)=\int_0^{\infty}e^{-t\sqrt{C}}He^{-t\sqrt{C}}dt$ where $H\in M_n(\mathbb{R})$.