Ridge regression objective function gradient

Question

Considering ridge regression problem with given objective function as:

$$f (W) = \|XW - Y\|_F^2 + \lambda \|W\|_F^2$$

Having convex and twice differentiable function results into:

$$\nabla f (W) = 2 \lambda W + 2X^T(XW - Y)$$

And finding its roots. My question is: why is the gradient of $\|XW - Y\|_F^2$ equal to $2X^T(XW - Y)$?

The initial problem is empirical risk minimization: $\min_{w, b} F(w, b) = \frac{1}{m} \sum_{i = 1}^{m} (wx_i^T + b + y_i)^2)$, so they represent it as matrix-valued function, where $W=(w_1, ..., w_d, b)^T$ and $X$ has additional ones column. — Artem, Mar 02 '19 at 19:33
I am not sure I understand how it transits from $n \times p$ to $m \times p$ dimensions. — Artem, Mar 02 '19 at 19:41
I don't understand how they differentiate over $W$ in $||XW - Y||^2$ part and they get exactly $2X^T(XW - Y)$. — Artem, Mar 02 '19 at 19:50
Use the definition of the Frobenius norm and write the function in terms of the trace. Take a look at this answer. Then compute the directional derivative in terms of the Frobenius inner product. Finally, extract the gradient. — Rodrigo de Azevedo, Mar 02 '19 at 19:59

score 1 · Accepted Answer · answered Mar 05 '19 at 23:24

Rewrite

$$||XW-Y||^2_2 = \left[XW-Y\right]^T\left[XW-Y\right]=\left[W^TX^T-Y^T\right]\left[XW-Y\right]=W^TX^TXW-Y^TXW-W^TX^TY+Y^TY$$

Hence, the total derivative on the right-hand side gives

$$dW^TX^TXW+W^TX^TXdW-2dW^TX^TY=2dW^T\left[X^TXW-X^TY \right]=dW^T\left[2X^T(XW-Y) \right].$$

The expression in the bracket is the gradient of the initial expression with respect to $W$. Note, that I used the fact that the transpose of a scalar is a scalar.

Ridge regression objective function gradient

1 Answers1