17

I use The Matrix Cookbook by Kaare Brandt Petersen and Michael Syskind Pedersen to solve many problems (mostly matrix derivatives). In most cases, I just map the problem to one of the formula and solve it but I cannot derive the formula by myself easily (I may prove the given formula is correct).

Since I do not have access to the book when I am taking test, I am wondering how others perform these kinds of calculation without a reference book. Is this book just considered as a reference or I should study the book and try to drive the formula by myself?

Does anyone have an insight on how to get better in matrix calculus (specially derivative with respect to vector or matrix) or how should I study such books? Thanks in advance.

Rodrigo de Azevedo
  • 23,223
Crimson
  • 1,141
  • Could you be specific about a type of calculation you are referring to? Perhaps giving an example problem? – TomGrubb Aug 02 '17 at 01:45
  • @ThomasGrubb Thanks for your comment. As an example last time for solving the following problem I used formula 119 in the book. Without that I had no idea how to solve it quickly. $E_D(\widetilde{ W}) = \frac{1}{2} Tr{(\widetilde{X}\widetilde{W}- T)^T(\widetilde{X}\widetilde{W}- T)}\quad\quad$

    Setting the derivative with respect to $\widetilde{W}$ to zero $\widetilde{W} = (\widetilde{X}^T\widetilde{X})^{-1}\widetilde{X}^TT$.

     – Crimson Aug 02 '17 at 01:49
  • This example is from Pattern Recognition and Machine Learning book chapter 4.1, page 185, I was trying to do the calculation by myself. – Crimson Aug 02 '17 at 01:59
  • 3
    The best way to memorize a formula is understand its proof. – Surb Aug 05 '17 at 12:14
  • Thanks for your insight, Surb. – Crimson Aug 07 '17 at 01:49
  • 1
    Did you find a good book for studying this subject? I am trying to find the $\frac{ \partial A V }{ \partial X}$, where both $A$ and $V$ depend on $X$. I've studied "vector calculus", in terms of div/grad/curl, but have never taken the derivative of a matrix product with respect to a matrix before. Any recommendation for good books for learning this subject would be appreciated. – Joe Feb 02 '20 at 21:53

2 Answers2

10

I have the same problem!
I found two things help. 1) Knowing the basics and why the work the way they do. 2) Being able to prove things for myself.

I found 3 more references, that may be helpful. One of them helped me understand the basics specifically differentiation with respect to a vector and Hessian a lot better. https://www.sfu.ca/~haiyunc/notes/matrix_calculus.pdf It's short sweet and very nicely done reminder of the basics. The 2nd one I have not started reading... Matrix Calculus By Sourya Dey. Finally, the second reference also cites Matrix Calculus in Wikipedia and Wikipedia is almost always good https://en.wikipedia.org/wiki/Matrix_calculus

user96265
  • 201
8

The Matrix Cookbook is only a list of formulas without proof. It absolutely does not learn to understand the theory - and the practice - of derivation. Better, you can have a look at the solutions given in this website. Roll up your sleeves, my friend.

Let $g(U)=1/2tr(U^TU),f(W)=1/2tr((XW-T)^T(XW-T))$.

If $K$ has same dimensions as $U$, then $Dg_U(K)=1/2(tr(K^TU)+tr(U^TK))=tr(K^TU)$.

We deduce that, if $H$ has same dimensions as $W$, then $Df_W(H)=tr((XH)^T(XW-T))=tr(H^TX^T(XW-T))$.

If, for every $H$, $Df_W(H)=0$, then $X^T(XW-T)=0$, that implies

$W=(X^TX)^{-1}X^TT$ (if $X^TX$ is invertible).

  • Absolutely agree. The principle (the product of two first-order terms is negligible for first-order approximation) is so simple that I always wonder why anyone would need that "cookbook" for calculating derivatives. – user1551 Aug 05 '17 at 14:47
  • 30
    @user1551 Am afraid I would have to disagree with your statement, calculating matrix derivatives is an involving one for those who are not well versed in mathematics, having something like a cookbook saves occasional users time. Your statement is remotely analogous to saying why should there be more accessible programming languages like Python when C is so simple. – chutsu Mar 25 '18 at 17:14