Linkage between Lagrange multipliers and eigenvectors/eigenvalues?

Question

I am a Freshmen Engineering Student, and this past Semester I took an Intro Multivariable Calculus Course, where we covered everything up to Lagrange Multipliers and Space Curves but not things like Surface and Line Integrals. At the same time, I also took an Intro to Linear Algebra Course that covered everything up to Determinanats, Eigenvectors/Eigenvalues and threw in Orthogonalization and Gram-Schmidt Orthonormalization.

The set up for Eigenvalues is the solution to:

A$\vec x$ = $\psi$$\vec x$

Simultaneously, we where taught that the method to find Lagrange Multipliers with 1 constraint was the solution to:

$\nabla$$f(x, y)$ = $\lambda$$\nabla$$g(x, y)$

resulting in 3 equations:

$\frac{\partial f}{\partial y}$ = $\frac{\partial g}{\partial y}$

$\frac{\partial f}{\partial x}$ = $\frac{\partial g}{\partial x}$

$g(x, y) = c$

Where $g(x, y) = c$ is the appropriate Level Curve given by the restraint conditions. With these equations, your unknowns are the appropriate $(x, y)$ and the corresponding $\lambda$. My Multi Professor then said, "From here it is as simple as solving the given system of equations and checking the whether or not they are Maximum or Minimums." Myself and a friend of mine, whom is in the same Linear Class as I, immediately wondered if there is a way to use Linear Algebra to solve the system of equations, as we realized that the set up of:

$\nabla$$f(x, y)$ = $\lambda$$\nabla$$g(x, y)$

looks remarkable similar to:

A$\vec x$ = $\psi$$\vec x$

We realized quickly that if we consider the Linear Transformation Matrix:

A: $\mathbb{R} ^ 3 \mapsto \mathbb{R} ^ 3$

$\nabla$$f(x, y)$ = A $\cdot$ $\nabla$$g(x, y)$

Then we can set up the entire Lagrange Multiplier Process as:

A $\cdot$ $\nabla$$g(x, y)$ = $\psi$$\nabla$$g(x, y)$

and then from there we can solve for the Eigenvalues, which will be our Lagrange Multiplies, and our Eigenvectors, which will be our Critical Points. Now this seemed all well and good, and we confirmed with our Linear Professor that this set-up was valied, but we came upon a major questions that left all 3 of us stumped. Our Linear Professor told us that he would dig into it things a little more, as he started spewing math things that where way above my friend and I's head, and I figured this is a good place to ask them.

How would we find the Transformation Matrix from $f(x, y)$ to $g(x, y)$. This entire process hinges upon finding that Transformation Matrix, and assuming that you can find said Matrix and assuming that is is a valid Linear Transformation, then how would one go about finding it?

Answer 1

For your equation $\nabla f(x, y) = \mathbf{A} \nabla g(x, y)$, the important thing to realize is that not all pairs of functions $f, g$ are going to have a matrix $\mathbf{A}$ that solves this equation. This means that your method of finding the Lagrange multipliers is not going to be fully general, but that's okay, there are plenty of methods in math that only apply to special cases.

So what conditions do $f$ and $g$ have to satisfy in order for there to exist an $\mathbf{A}$ that satisfies $\nabla f(x, y) = \mathbf{A} \nabla g(x, y)$? First let's consider the simpler 1 dimensional case:

$$ f'(x) = ag'(x) $$

For a given $g$, we can pick an arbitrary constant $a$ that we want as the coefficient in the above equation. Then, by the fundamental theorem of calculus:

$$ f(x) = C + \int_0^x ag'(t) dt = C + ag(x) $$

Where $C$ is an arbitrary constant. So the general solution in the 1d case is that we get to pick two constants, $a$ and $C$, and then $f(x) = ag(x) + C$.

Now for the case in two dimensions. This time we'll pick an arbitrary matrix $\mathbf{A}$. As in the previous case, we should be able to find $f(x, y)$ by somehow integrating $\mathbf {A}\nabla g(x, y)$. One way of doing this is to think of some curve $\gamma$ from the origin to $(x, y)$. Then we can do a line integral like this:

$$ f(x, y) = C + \int_\gamma (\mathbf{A}\nabla g(x, y)) \cdot d\vec l $$

This gives us a value for $f(x, y)$, but we should remember that not all vector fields are the gradient of a function. Just because we can write down this integral doesn't mean it will have the same value for all paths $\gamma$ from $(0,0)$ to $(x,y)$. But in order for $f$ to be defined, we need it to give the same value for all paths.

The condition for a vector field $\vec v(\vec x)$ to be the gradient of a function is that $\frac{\partial v_i}{\partial x_j} = \frac{\partial v_j}{\partial x_i}$ for all $i,j$. In this case, that means that we need:

$$ \frac{\partial (a_{xx}g_x+a_{xy}g_y)}{\partial y} = \frac{\partial (a_{yx}g_x+a_{yy}g_y)}{\partial x} $$

$$ a_{xx}g_{xy}+a_{xy}g_{yy} = a_{yx}g_{xx}+a_{yy}g_{xy} $$

This is a second order linear PDE for $g$. Just to give an example, if we have:

$$ \mathbf{A} = \begin{pmatrix} 0 & c^2 \\ 1 & 0 \end{pmatrix} $$

Then we get that $g$ must satisfy a wave equation:

$$ g_{xx} = c^2g_{yy} $$

Other choices of $\mathbf{A}$ give other equations that $g$ must satisfy, but in general we see that for any given value of $\mathbf{A}$, there are only some functions $g$ that will work. One interesting case is what happens when we take $\mathbf{A} = a\mathbf{I}$ for some constant $a$. In that case, we get:

$$ g_{xy} = g_{xy} $$

This is automatically satisfied by all "nice" $g$, and so the constraint doesn't do anything. Therefore in this case we get that:

$$ f(x, y) = ag(x,y) + C $$

similar to the 1-d case.