2

I learned that the notation $\alpha I \preceq \nabla ^2 F $ means that $\langle \alpha x, x\rangle \leq \langle \nabla^2 F x, x\rangle$.

  1. What is this called? I know that $\nabla^2 F \succeq 0$ is means that $F$ is positive semidefinite. What would we call $\alpha I \preceq \nabla ^2 F $ which I guess is a weaker version?

  2. Is this related to $\nabla F$ being Lipschitz continuous? It roughly looks like an upper/lower bound on $\nabla \cdot \nabla F$, but I don't understand the vector case where we work with inner products.

  3. I saw in a paper that $\alpha I \preceq \nabla ^2 F $ implied that $\langle \nabla F(x) - \nabla F(y) , \; x-y \rangle \geq \alpha \|x-y\|^2_2$. I can see why $\langle \nabla^2 F(x-y) , \; x-y \rangle \geq \alpha \|x-y\|^2_2$ from the definition, but I don't know why this would apply to $\nabla F(x) - \nabla F(y)$ on the left hand side.

Rodrigo de Azevedo
  • 23,223
blue_egg
  • 2,333

1 Answers1

2

For #3:

Fix $x$ and $y$. Define $g(t) = F(y+t(x-y))$. Then $g'(t) = \langle \nabla F(y+t(x-y)), x-y\rangle$ and $g''(t) = \langle (\nabla^2 F(y+t(x-y))) x-y, x-y \rangle$.

Then $$\langle \nabla F(x) - \nabla F(y), x-y\rangle = g'(1)-g'(0).$$ By the mean value theorem, there exists some $c \in [0, 1]$ such that the above equals $$g''(c) = \langle (\nabla^2 F(y+t(x-y))) (x-y), x-y \rangle \ge \alpha \|x-y\|^2$$ where the last bound follows from the definition of $\alpha I \preceq \nabla^2 F$.

angryavian
  • 93,534
  • Thanks a lot for the answer. I still have a question about it. In the last line, I'd expect that we use something like $\langle \nabla^2 F (x-y) , x-y\rangle \geq \alpha |x-y|^2$. Why does the inequality work with the $y+t(x-y)$? – blue_egg Sep 15 '23 at 13:47
  • and do you mean $\nabla^2F(y+t(x-y))(x-y)$ rather than $\nabla^2F(y+t(x-y))x-y$? – blue_egg Sep 15 '23 at 13:56
  • 1
    @900edges Thanks, I have fixed my typo. In the first sentence of your question, you wrote $\langle \alpha x, x \rangle \le \langle \nabla^2 F x, x \rangle$, but it is ambiguous where the Hessian is evaluated. I am assuming that the inequality holds for any place the Hessian is evaluated, that is $\langle \alpha x, x \rangle \le \langle \nabla^2 F(u) x, x \rangle$ for all $x$ and $u$. Having a global bound on the Hessian is a common assumption for these kind of results. – angryavian Sep 15 '23 at 15:50