2

Given a set of normalized vectors $\mathbf{x}$ and $\mathbf{y}$ of length $N$, with each entry independently sampled from $\mathcal{N}(0,1)$ before being divided by the vector norm.

By running simulations I got this empirical result:

$$var(\mathbf{x}^T\mathbf{y})=\frac{1}{N}$$

Can this be proved?

pear
  • 23

3 Answers3

4

This can be treated in analogy to Expected value of inner product of uniformly distributed random unit vectors.

By rotational symmetry, we can assume one of the vectors to be a fixed unit vector, say, $\mathbf e_1$. The expected value of the dot product with this vector is $0$ by symmetry. Thus the variance is the expected value of the square. The sum of the squares of the $n$ components is $1$ due to normalization, so by symmetry the expected value for each component must be $\frac1N$.

joriki
  • 242,601
  • If I understand you correctly, gaussian sampling is not necessary. As long as each element in the vector is sampled independently from the same distribution, $f(\mathbf{x})$ would be invariant under permutations of the coordinates. – pear Aug 13 '18 at 02:40
  • @pear: Yes, that's right. But see the last comment I just added under YC_Xu's answer. The permutation symmetry is enough to show that $(\mathbf e_j^\top\mathbf y)^2$ is equidistributed over $j$, but for its expected value to be the variance of $\mathbf x^\top\mathbf y$, we also need inversion symmetry (to allow us to replace the variance by the expectation of the square and to replace $\mathbf x$ by $\mathbf e_1$). So Gaussian sampling isn't necessary, but the distribution of the elements of the vector must be symmetric about the origin. – joriki Aug 13 '18 at 04:56
  • @pear: For instance, the distribution could be $P(x_i=+1)=P(x_i=-1)=\frac12$, which results in uniform sampling from the corners of a hypercube. – joriki Aug 13 '18 at 04:59
  • So by permutation symmetry $E(y_i^2)=\frac{1}{N}$, and $E(y_iy_j)=0$ comes from $f(y)=f(-y)$. Yet is the inversion symmetry necessary for making $E(y_iy_j)=0$ (or some weaker conditions might suffice)? – pear Aug 13 '18 at 07:53
  • @pear: There are certainly other distributions that lead to $E(y_iy_j)=0$, but I can't think of any interesting conditions that would ensure this other than inversion symmetry. – joriki Aug 13 '18 at 10:04
1

The solution provided by joriki is great. One of the sufficent conditions is the uniformity (as joriki points out, the invariance under interchange of the coordinates is the name of the game) : The normalized gaussian vectors are uniformly distributed on a sphere. The intuitive explanation is that when you generate a gaussian vector, the probability density of this point is a function of sum of squares, which is exactly the same for all the points that is on the same sphere. When you do the normalization, this sphere is projected to the standard sphere $\left| r \right|=1$. So the normalization process is in fact projecting all the sphere onto the standard sphere. Since gaussian vectors are distributed uniformly on each sphere, their projection should be uniform, too.

If you want to verify the solution with a little bit calculus, you can use the sphere axis. Because the points is on the sphere, you have $n-1$ degrees of freedom and the dot product is the cosine, correspondingly.

$$\frac{2\pi^{n/2}}{\Gamma(n/2)}\int_{0}^{\pi}\cdots\int_{0}^{\pi}\int_{0}^{2\pi}\cos^{2}(\phi_{1})sin^{n-2}(\phi_{1})sin^{n-3}(\phi_{2})\cdots sin(\phi_{n-2})d\phi_{1}\cdots d\phi_{n-1} \\=1-\frac{\int_{0}^{\pi}sin^{n}xdx}{\int_{0}^{\pi}sin^{n-2}xdx}=\frac{1}{n}$$

YC_Xu
  • 43
  • That's not true, actually. Uniformity isn't necessary; the key feature here is the symmetry among the coordinates. You could have an arbitrary distribution function $f(\mathbf x)$; as long as it's invariant under interchange of the coordinates, by symmetry the expected value of the square of each coordinate is $\frac1N$ because the normalized vector has square $1$. As an extreme case, you could take a discrete uniform distribution on the vertices of a cube or an octahedron, which is about as far from uniform as it gets. – joriki Aug 10 '18 at 09:09
  • Also, I'm not sure why you call my solution "intuitive" -- I'm sure you meant it in a nice way :-), but I'd be interested to know which part of it you don't consider rigorous. – joriki Aug 10 '18 at 09:13
  • And one final remark: You can get displayed equations by enclosing them in double instead of single dollar signs. That makes them a lot easier to read, especially when you have integrals nested in fractions and the like. – joriki Aug 10 '18 at 09:14
  • I'm sorry that i've misused the languange...I meant your answer were concise and insightful. One thing i do not understand about your comment is that in your example, octahedron for instance, the variance is not $\frac{1}{3}$ as expected (the dimension is 3). But i guess it is true that uniformity is not necessarily required for this conclusion. And what does "invariant under interchange of the coordinates" mean? Do you mean rotational invariance? @joriki – YC_Xu Aug 10 '18 at 12:37
  • No, I mean invariance under permutations of the coordinates, e.g., $f(x_1,x_2,x_3)=f(x_2,x_1,x_3)=f(x_2,x_3,x_1)=\cdots$. I do think the variance is $\frac13$ for the octahedron. At least if it's aligned with the coordinate axes. I think it should be $\frac13$ even in different alignments, but I haven't checked that. – joriki Aug 10 '18 at 12:44
  • Yes, you are right. @joriki – YC_Xu Aug 11 '18 at 07:18
  • Actually, we also need invariance under coordinate inversions, not just permutations. (The octahedron and cube both have that.) The expectation of the dot product has to be zero in order for the variance to be just the expectation of its square. Also, I should have mentioned that for the proof we need the fact that if the squares of the components are equidistributed in one coordinate system, then they're equidistributed in all rotated coordinate systems. (If $E[a^2]=E[b^2]$ and $E[ab]=0$, then $E[(a\cos\phi+b\sin\phi)^2]=E[a^2]=E[b^2]$.) That allows us to replace $\mathbf x$ by $\mathbf e_1$. – joriki Aug 13 '18 at 04:50
0

With $u_i , v_i\stackrel{iid}{\sim} \mathcal{N}(0,1)$

\begin{align} var(\mathbf{x}^T\mathbf{y})&=\mathbb{E}[(\mathbf{x}^T\mathbf{y})^2]-(\mathbb{E}[\mathbf{x}^T\mathbf{y}])^2\\ &=\mathbb{E}\big[\frac{(\sum_i u_iv_i)^2}{(\sum_i u_i^2)(\sum_i v_i^2)}\big]\\ &=\mathbb{E}\big[\frac{\sum_{i}u_i^2v_i^2+\sum_i\sum_j^{j\neq i}u_i u_j v_i v_j}{(\sum_i u_i^2)(\sum_i v_i^2)}\big]\\ &=N \mathbb{E}\big[\frac{u_1^2 v_1^2}{(\sum_i u_i^2)(\sum_i v_i^2)}\big]\\ &=N \mathbb{E}\big[\frac{u_1^2}{\sum_{i}u_i^2}\big]\mathbb{E}\big[\frac{v_1^2}{\sum_{i}v_i^2}\big]\\ &=\frac{1}{N} \end{align}

The second equality follows $\sum_i E[x_i]E[y_i] = 0$, the forth equality requires $E[x_i x_j]=0$. These are satisfied by the standard gaussian distribution. Also we used the permutation symmetry and the independence between $x_1^2$ and $y_1^2$.

pear
  • 23