0

Consider $$f(x,y,k) = kx-y$$ and its level set $f(x,y,k) = 0$

Now for example assume that I have found a point on this level set $(9,81,9)$, how can I then estimate $k$ in $(8,76,k)$? This translates to if we know $81/9=9$, how can we calculate $76/8 = ?$

I can calculate gradient $$\nabla f = [k,-1,x]$$

To remain on surface (locally), I shall go perpendicular to gradient. So $$[\Delta_x,\Delta_y,\Delta_k][k,-1,x]^T = 0$$

Now this lets us derive a wealth of algorithms which we can iterateively take small steps:

  1. Take step along a line in the tangent plane.
  2. Make correction in gradient direction to come back to surface.

But the problem I hit is when I want to go one step further into higher order models.

For example if I want to use a second order polynomial model...

All second order partial differentials are zero!

How can I modify $f$ or my algorithm to somehow add information of higher order than $1$ ?

Own work:

I've been thinking I can do $f \to f^3$. This is sure to generate a higher order power series expansion everywhere, right?

mathreadler
  • 26,534
  • 1
    How do you say that all second derivatives are $0$? You have the quadratic term $kx$. In particular, implicitly differentiating this term will lead to nonzero terms for $\partial^2 k/\partial x^2$ and $\partial^2 k/\partial x\partial y$. – Ted Shifrin Oct 27 '20 at 22:58
  • @TedShifrin of course you are right. I must have been still asleep. In fact maybe this is one of the simplest examples which still does have a non-zero Hessian. – mathreadler Oct 28 '20 at 07:11

1 Answers1

1

As @TedShifrin writes, of course we do get second order partial derivatives:

$$\frac{\partial^2}{\partial k \partial x} f = \frac{\partial^2}{\partial x \partial k} f = 1$$

So we get a non-zero, but constant Hessian:

$${\bf H}\{f\} = \begin{bmatrix}0&0&1\\0&0&0\\1&0&0\end{bmatrix}$$

Perhaps even one of the simplest and sparsest Hessians possible.

We can now go on to devise (for example) a second order polynomial approximation as per this question:

$$f(\mathbf{x}^* + h\mathbf{y}) = f(\mathbf{x}^*) + h \nabla f(\mathbf{x}^*)^T \mathbf{y} + \dfrac{1}{2} h^2 \mathbf{y}^T \nabla^2 f(\mathbf{x}^*) \mathbf{y} + O(h^3)$$

If we explicitly write $\bf g$ and $\bf H$ for gradient and Hessian:

$$f(\mathbf{x}^* + h\mathbf{y}) = f(\mathbf{x}^*) + h {\bf g}^T \mathbf{y} + \dfrac{1}{2} h^2 \mathbf{y}^T {\bf H} \mathbf{y} + O(h^3)$$

Now we can use this however we see fit to pick out a better direction for estimation.

mathreadler
  • 26,534