@Dunham was faster than me. I agree with the essential. Yet, the inequality to prove is
$F(A)\leq \sum_j |\lambda_j|^2$ where $spectrum(A)=(\lambda_j)_j$. In particular, if $A$ is nilpotent, then $F(A)=0$, and consequently, $F$ is not a norm.
Proof. Case 1. $spectrum(A)\subset \mathbb{R}$. (as Dunham). For every $\epsilon >0$, there is a $M\in GL(\mathbb{R})$ s.t. $MAM^{-1}=D+T$ where $D=diag(\lambda_i)$ and $T$ is strictly upper triangular and $||T||<\epsilon$ where $||.||$ is a matricial norm invariant by transposition. Note that $trace(DD^T)=\sum_j \lambda_j^2$; after it's easy.
Case 2. Otherwise, we may assume, by Case 1, that $A$ has only non-real eigenvalues of the form $(a_j\pm ib_j)_j$; then (cf. the standard reference given by Asaf Shachar), for every $\epsilon >0$, there is $M\in GL(\mathbb{R})$ s.t.
$MAM^{-1}=diag(U_1,\cdots,U_k)+T=\Delta+T$ where $U_j=\begin{pmatrix}a_j&b_j\\-b_j&a_j\end{pmatrix}$ and $T$ is strictly upper triangular and $||T||<\epsilon$ where $||.||$ is a matricial norm invariant by transposition. Note that $trace(\Delta\Delta^T)=\sum_j (a_j^2+b_j^2)=\sum_j|\lambda_j|^2$. After, it's easy. $\square$
EDIT 1. Proposition. $F(A)= \sum_j |\lambda_j|^2$.
Proof. It remains to show that if $spectrum(B)=(\lambda_j)_j$, then $\sum_j |\lambda_j|^2\leq tr(B^TB)$. There is a unitary matrix $U$ s.t. $B=UTU^*$ where $T$ is complex upper triangular. Then $tr(B^TB)=tr(UT^*U^*UTU^*)=tr(T^*T)\geq \sum_j |T_{j,j}|^2=\sum_j |\lambda_j|^2$. $\square$
EDIT 2. Proposition. The lower bound is reached IFF $A$ is diagonalizable over $\mathbb{C}$.
Proof. If $A$ is diagonalizable over $\mathbb{C}$, then $A$ is similar over $\mathbb{R}$ to $D$ (Case 1) or $\Delta$ (Case 2) and we are done.
Otherwise, let $M\in GL_n(\mathbb{R})$ and $B=MAM^{-1}$ (cf. EDIT 1). Since $T$ cannot be diagonal, $tr(B^TB)=tr(T^*T)> \sum_j |T_{j,j}|^2=\sum_j |\lambda_j|^2$. $\square$