23

I met this equation frequently in Guass-Newton optimizations. But I dont understand why the left and right side of the equation can be equal.

Lets say the Jacobian is $2$ by $2$

and Hessian is $$\begin{bmatrix}\frac{\partial^2f_1}{\partial^2 x_1 } & \frac{\partial^2f_1}{\partial^2 x_2 } \\ \frac{\partial^2f_2}{\partial^2 x_1 } & \frac{\partial^2f_2}{\partial^2 x_2 }\end{bmatrix}$$

But the right hand side of the equation is $$J^TJ =\begin{bmatrix} \left( \frac{\partial f_1}{\partial x_1} \right)^2 + \left( \frac{\partial f_2}{\partial x_1} \right)^2 & \frac{\partial f_1}{\partial x_1}\frac{\partial f_1}{\partial x_2} +\frac{\partial f_2}{\partial x_1}\frac{\partial f_2}{\partial x_1}\\ \frac{\partial f_1}{\partial x_1}\frac{\partial f_1}{\partial x_2} +\frac{\partial f_2}{\partial x_1}\frac{\partial f_2}{\partial x_1} &\left( \frac{\partial f_1}{\partial x_2} \right)^2 + \left( \frac{\partial f_2}{\partial x_2} \right)^2\end{bmatrix} $$

Why can these two be equal as presented in papers?

Siong Thye Goh
  • 153,832
user7487638
  • 333
  • 2
    See here: "The Gauss–Newton method is obtained by ignoring the second-order derivative terms [in the computation of the Hessian]." – angryavian Jul 07 '17 at 00:28

2 Answers2

29

The quadratic model based on the true Hessian is derived from truncating a Taylor series of the objective function as a whole, whereas the quadratic model based on the gauss-Newton hessian is based on truncating a Taylor series of the residual.

Starting with the optimization problem: $$\min_x \frac{1}{2} ||y-f(x)||^2,$$ Consider taking a Taylor series of $f$: $$f(x)=f(x_0)+J(x-x_0)+\text{higher order terms}.$$ The approximate optimization problem formed by truncating the Taylor series, $$\min_x \frac{1}{2} ||y-f(x_0)-J(x-x_0)||^2,$$ has Hessian $J^TJ$.

In general this is not exactly equal to the true Hessian, owing to potential second order cross reactions between other terms in the Taylor series of the residual, but they are equal when $y=f(x_0)$.

Nick Alger
  • 19,977
1

Two simple examples where we can compare, plot, $H = f''(x)$ and $J^2 = f'(x)^2$, are lines and quadratics in 1d:
Lines: $ \qquad \ f(x) = a x, \qquad J(x) = a, \ \ \ \ \ J^2(x) = a^2, \ \ \ \ \ \ H(x) = 0$
Quadratics: $ \ f(x) = a x^2 /2, \ \ \ \ J(x) = ax, \ \ \ J^2(x) = a^2 x^2, \ \ \ H(x) = a$

So for lines, $J^2$ is close to $H$ iff $a$ is small; for quadratics, if $ax^2 \approx 1, f(x) \approx \, ^1/_2 $.

In $n$ dimensions, $H$ can be rotated to a sum of independent components $a_i (x_i - b_i)^2$ . Some of these may have $H_i \approx J_i^2$, some not.

denis
  • 961