I am trying to prove that for a symmetric matrix $A$, the following is true: $∂(x^TA x)/∂x = 2A$ .
Part 1: First I tried to verify this for a small example: $$ A = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{12} & a_{22} & a_{23} \\ a_{13} & a_{23} & a_{33} \end{bmatrix} \quad \text{(symmetric matrix)} $$
$$ x = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} $$
Step 1: Calculate $x^T A x$ First, calculate $Ax$: $$ Ax = \begin{bmatrix} a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \\ a_{12}x_1 + a_{22}x_2 + a_{23}x_3 \\ a_{13}x_1 + a_{23}x_2 + a_{33}x_3 \end{bmatrix} $$
Now, multiply $x^T$ by $(Ax)$: $$ x^T A x = \begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix} \begin{bmatrix} a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \\ a_{12}x_1 + a_{22}x_2 + a_{23}x_3 \\ a_{13}x_1 + a_{23}x_2 + a_{33}x_3 \end{bmatrix} $$
Step 2: Expand $x^T A x$ $$ \begin{align} x^T A x &= x_1(a_{11}x_1 + a_{12}x_2 + a_{13}x_3) + x_2(a_{12}x_1 + a_{22}x_2 + a_{23}x_3) + x_3(a_{13}x_1 + a_{23}x_2 + a_{33}x_3) \\ &= a_{11}x_1^2 + a_{12}x_1x_2 + a_{13}x_1x_3 + a_{12}x_2x_1 + a_{22}x_2^2 + a_{23}x_2x_3 + a_{13}x_3x_1 + a_{23}x_3x_2 + a_{33}x_3^2 \end{align} $$
Step 3: Calculate $∂(x^T A x)/∂x$
Find the partial derivative with respect to each $x_i$:
$$ \frac{\partial(x^T A x)}{\partial x} = \begin{bmatrix} \frac{\partial(x^T A x)}{\partial x_1} \\ \frac{\partial(x^T A x)}{\partial x_2} \\ \frac{\partial(x^T A x)}{\partial x_3} \end{bmatrix} $$
calculate each component:
$$ \begin{align} \frac{\partial(x^T A x)}{\partial x_1} &= 2a_{11}x_1 + a_{12}x_2 + a_{13}x_3 + a_{12}x_2 + a_{13}x_3 \\ &= 2a_{11}x_1 + 2a_{12}x_2 + 2a_{13}x_3 \\ \frac{\partial(x^T A x)}{\partial x_2} &= a_{12}x_1 + a_{12}x_1 + 2a_{22}x_2 + a_{23}x_3 + a_{23}x_3 \\ &= 2a_{12}x_1 + 2a_{22}x_2 + 2a_{23}x_3 \\ \frac{\partial(x^T A x)}{\partial x_3} &= a_{13}x_1 + a_{23}x_2 + a_{13}x_1 + a_{23}x_2 + 2a_{33}x_3 \\ &= 2a_{13}x_1 + 2a_{23}x_2 + 2a_{33}x_3 \end{align} $$
Step 4: Combine the results $$ \frac{\partial(x^T A x)}{\partial x} = \begin{bmatrix} 2a_{11}x_1 + 2a_{12}x_2 + 2a_{13}x_3 \\ 2a_{12}x_1 + 2a_{22}x_2 + 2a_{23}x_3 \\ 2a_{13}x_1 + 2a_{23}x_2 + 2a_{33}x_3 \end{bmatrix} $$
Factor out 2 $$ \frac{\partial(x^T A x)}{\partial x} = 2\begin{bmatrix} a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \\ a_{12}x_1 + a_{22}x_2 + a_{23}x_3 \\ a_{13}x_1 + a_{23}x_2 + a_{33}x_3 \end{bmatrix} = 2Ax $$
This produces the desired result: $$ \frac{\partial(x^T A x)}{\partial x} = 2Ax = 2A\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} $$
Part 2: Now, I am trying to prove this in general.
I can recognize that the equivalence between matrix multiplications and summations
$$x^T A x = \sum_{i=1}^n \sum_{j=1}^n x_i a_{ij} x_j$$
$$\frac{\partial (x^T A x)}{\partial x_k} = \frac{\partial}{\partial x_k} \sum_{i=1}^n \sum_{j=1}^n x_i a_{ij} x_j$$
Using the product rule:
$$\frac{\partial (x^T A x)}{\partial x_k} = \sum_{i=1}^n \sum_{j=1}^n \left(\frac{\partial x_i}{\partial x_k} a_{ij} x_j + x_i a_{ij} \frac{\partial x_j}{\partial x_k}\right)$$
Here is where I got stuck.
I am having trouble manipulating these matrix derivatives to prove this identity. Can someone please help me?
