1

I have a symmetric positive definite matrix $L\in\mathbb{R}^{n\times n}$ and its Cholesky factor $G\in\mathbb{R}^n$ such that $L=GG^T$. Called $\mathrm{vec}:\mathbb{R}^{n\times n}\to\mathbb{R}^{n^2}$ the vectorization operator, and $K_{n}\in\mathbb{R}^{n^2\times n^2}$ the commutation matrix, i.e. the permutation matrix realizing $\mathrm{vec}(M)=K\mathrm{vec}(M^T)$ for $M\in\mathbb{R}^{n\times n}$, I know I can get $$ \frac{\partial \mathrm{vec}(L)}{\partial \mathrm{vec}(G)} = (I_{n^2}+K)(G\otimes I_n), $$ where $I_k\in\mathbb{R}^{k\times k}$ stands for the identity matrix. My goal would be to compute the derivative of the vectorization of the Cholesky factor $G$ with respect to the original matrix $L$. However, the matrix $I_{n^2}+K$ is singular, since $K$ has $-1$ as an eigenvalue.

Is there a way to compute the derivative $$ \frac{\partial \mathrm{vec}(G)}{\partial\mathrm{vec}(L)}? $$

Dadeslam
  • 876
  • 8
  • 22

1 Answers1

2

$ \def\o{{\tt1}} \def\f{\frac12} \def\h{\odot} \def\k{\otimes} \def\BR#1{\Big[#1\Big]} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\Diag#1{\op{Diag}\LR{#1}} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\D{\Delta} $Let $P$ denote a matrix whose elements are all $\tt1$ below the diagonal, $\tfrac12$ on the diagonal, and $0$ above the diagonal, e.g. $$ P=\left[ \begin{array}{cccc} \f & 0 & 0 & 0 \\ \o & \f & 0 & 0 \\ \o & \o & \f & 0 \\ \o & \o & \o & \f \\ \end{array} \right]$$ This $P$ matrix can disentangle the symmetrized product of lower triangular matrices $(A,B)$ $$\eqalign{ \def\AT{A^{-T}} \def\A{A^{-1}} \def\BT{B^{-T}} \def\B{B^{-1}} F = AB^T + BA^T &\qiq &B = A\,\BR{P\h\LR{\A F\AT}} \\ &&({\rm assuming\ }\A\ {\rm exists}) }$$ This has obvious applications to your problem $$\eqalign{ \def\GT{G^{-T}} \def\G{G^{-1}} L &= GG^T \\ dL &= G\,dG^T + dG\,G^T \qquad\; \{{\rm differential\ of}\ L\}\\ dG &= G\,\BR{P\h\LR{\G\,dL\:\GT}} \\ \vc{dG} &= \LR{I\k G}\,\vc{P\h\LR{\G\,dL\:\GT}} \\ &= \LR{I\k G}\,\Diag{\vc P}\,\vc{\G\,dL\:\GT} \\ &= \LR{I\k G}\,\Diag{\vc P}\,\LR{\G\k\G}\:\vc{dL} \\ \grad{\vc G}{\vc L} &= \LR{I\k G}\,\Diag{\vc P}\,\LR{\G\k\G} \\ }$$

In the above, $\h$ is the Hadamard product and $\k$ is the Kronecker product.

greg
  • 40,033
  • Very interesting approach, thanks! Is there any reference (like books) where I can read about these ideas/techniques. I've never seen them, since every source I read where the vec operator is considered, only very simple properties are presented. – Dadeslam Jan 05 '24 at 14:28
  • 2
    @Dadeslam I can't recall the journal article (several years ago) where I originally saw the $P$ matrix. As for book recommendations, the standard text is probably Magnus and Neudecker's Matrix Differential Calculus, although personally I prefer Hjorungnes's Complex-Valued Matrix Derivatives.

    For looking up formulas, if you cannot find it in Petersen and Pedersen's Matrix Cookbook then consult Bernstein's Matrix Mathematics: Theory, Facts, and Formulas

     – greg Jan 05 '24 at 17:15
  • Amazing, thank you again! – Dadeslam Jan 05 '24 at 17:37