2

The second order Taylor approximation for a function $f(\mathbf{x}^*)$ is presented to me as

$$f(\mathbf{x}^* + h\mathbf{y}) = f(\mathbf{x}^*) + h \nabla f(\mathbf{x}^*)^T \mathbf{y} + \dfrac{1}{2} h^2 \mathbf{y}^T \nabla^2 f(\mathbf{x}^*) \mathbf{y} + O(h^3)$$

But from what I understand, shouldn't there be a Hessian matrix in the third term? Or is the Laplacian in the third term somehow the Hessian?

I would greatly appreciate it if people would please take the time to clarify this.

The Pointer
  • 4,686

1 Answers1

2

Yes, in general it should be the Hessian matrix and not the Laplacian.

uniquesolution
  • 18,982
  • 1
    For a local minimum, we require $\nabla^2 f(\mathbf{x})$ to be positive semidefinite. And we know that, if a matrix $\mathbf{A}$ is positive definite in the function $f(\mathbf{x}) = \mathbf{x}^T \mathbf{A} \mathbf{x}$, then $f$ has a unique global minimum. So, given the resemblance here, it seems that $\mathbf{y}^T \nabla^2 f(\mathbf{x}^*)\mathbf{y}$ is required? [...] – The Pointer Mar 11 '20 at 15:45
  • 1
    [...] The quadratic approximation of a twice-differentiable function $f$ at $\mathbf{x}_0$ is $$f(\mathbf{x}) \approx f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0)^T(\mathbf{x} - \mathbf{x}_0) + \dfrac{1}{2}(\mathbf{x} - \mathbf{x}_0)^T \mathbf{H}_0 (\mathbf{x} - \mathbf{x}_0),$$ so if we let $\mathbf{y} = (\mathbf{x} - \mathbf{x}_0)$, then it seems evident that $\nabla^2 f(\mathbf{x}^*)$ has the role of the Hessian matrix, so to speak? – The Pointer Mar 11 '20 at 15:45
  • 1
    Actually, I think the term $\dfrac{1}{2} h^2 \mathbf{y}^T \nabla^2 f(\mathbf{x}^) \mathbf{y}$ might be* the Hessian matrix. We know that, at a minimum, the first derivative must be zero. Neglecting the higher order terms and rearranging, we get $$\dfrac{1}{2} h^2 \mathbf{y}^T \nabla^2 f(\mathbf{x}^) \mathbf{y} = f(\mathbf{x} + h \mathbf{y} - f(\mathbf{x}^) \ge 0,$$ which is the definition of a positive semidefinite matrix. The structure of the term $\dfrac{1}{2} h^2 \mathbf{y}^T \nabla^2 f(\mathbf{x}^*) \mathbf{y}$ matches that of Hessian matrix https://en.wikipedia.org/wiki/Hessian_matrix – The Pointer Mar 11 '20 at 15:59
  • 2
    So it seems that you have clarified everything for yourself, right? – uniquesolution Mar 11 '20 at 17:17