3

Assume that $\mathbf{J} = \{ J \}_{ij}$ are centered independent standard Gaussian with variance $1/n$ random variables for $i, j = 1, \dots, n$. Why is the function $\|\mathbf{J}\|_{\infty}$ then $1/n$-Lipschitz w.r.t to the Euclidean norm, that is, $$ \left| \sup_{\|u\| = 1} \langle u, \mathbf{J}u \rangle - \sup_{\|v\| = 1} \langle v, \mathbf{J}v \rangle \right| \leq \frac{1}{n} \|u - v\|_2 . $$ Thus, it has a sub-Gaussian tail which is $$ P(\|\mathbf{J}\|_{\infty} - E[\|\mathbf{J}\|_{\infty}] \geq x) \leq e^{-n x^2 / 2}. $$

The original statement is as follows.

Moreover, it is easy to see that $$ \|\mathbf{J}\|_\infty = \sup_{\|u\| = 1} \langle u, \mathbf{J} u \rangle $$ has sub-Gaussian tail. Indeed, is is a Lipschitz function of the Gaussian entries of $\mathbf{J}$ (with respect to the Euclidean norm) with constant bounded by $N^{-1/2}$ so that by Herbst argument (see [24, 25]), we have the concentration inequality $$ P(\|\mathbf{J}\|_\infty \geq E[\|\mathbf{J}\|_\infty] + x) \leq e^{-\frac{1}{2} N x^2} . $$

Jendrik Stelzner
  • 17,081
Hermi
  • 1,488

1 Answers1

1

$\|\mathbf{J}\|_{\infty}$ can be considered as a function of $\{\mathbf{J}_{i,j}\}_{i,j\in [n]^2}$, denoted it by $f(\mathbf{J}_{1,1},\dots,\mathbf{J}_{n,n})$. Let $\mathbf{J}_{vec}=(\mathbf{J}_{1,1},\dots,\mathbf{J}_{n,n})\in \mathbb{R}^{n^2}$ represents the unrolled version of $\mathbf{J}$. Based on Theorem 5.8 in lecture note [1], if we manage to show that $\|\mathbf{J}\|_{\infty}$ is $1$-Lipschitz in the sense that $$ \|\mathbf{J}\|_{\infty} - \|\mathbf{J'}\|_{\infty} \leq \| \mathbf{J}_{vec}- \mathbf{J'}_{vec} \|_2 $$

Then, we are done. Consider

$$ \sup_{||u||=1}<u, \mathbf{J}u>-\sup_{||v||=1}<v, \mathbf{J'}v> \\ \leq \sup_{||u||=1}<u, \mathbf{J}u>-<u, \mathbf{J'}u> $$ Then, $\sup_{||u||=1}<u, \mathbf{J}u>-<u, \mathbf{J'}u>=\sup_{||u||=1}<u, (\mathbf{J}-\mathbf{J'})u> = \|\mathbf{J}-\mathbf{J'}\|_2$

where $\|\mathbf{J}-\mathbf{J'}\|_2$ is the matrix $\ell_2$ norm. Then, by a standard result (for a proof see [2]) we have $\|\mathbf{J}-\mathbf{J'}\|_2 \leq \|\mathbf{J}-\mathbf{J'}\|_F$ where $\|\mathbf{J}-\mathbf{J'}\|_F$ is the Frobenius norm. Finally note that $\|\mathbf{J}-\mathbf{J'}\|_F = \| \mathbf{J}_{vec}- \mathbf{J'}_{vec} \|_2$

[1]https://www.stat.cmu.edu/~arinaldo/Teaching/36755/F16/Scribed_Lectures/LEC0914.pdf

[2] Why is the Frobenius norm of a matrix greater than or equal to the spectral norm?

MMH
  • 587
  • Thanks. Why do we have $\sup_{||u||=1}<u, (\mathbf{J}-\mathbf{J'})u> = |\mathbf{J}-\mathbf{J'}|2?$ Isn't it $\sup{||u||=1}<u, (\mathbf{J}-\mathbf{J'})u> = |\mathbf{J}-\mathbf{J'}|_\infty?$ – Hermi May 10 '22 at 02:30
  • Sorry, from the definition of $||\infty$, $|A|\infty=\sup_{|x|\infty=1}|Ax|{\infty}$. – Hermi May 10 '22 at 02:36
  • @Hermi you are right. Why this definition of $|A|\infty=\sup{|x|\infty=1}|Ax|{\infty}$ is equivalent with the definition in the question? – MMH May 10 '22 at 17:53
  • I do not know... the author says "it is easy to see that $|J|_\infty=\sup <x, Jx>$. Also, I think $|A|_2=\sup <Ax, Ax>$? – Hermi May 10 '22 at 23:37