9

Let $F(\bar{x})=\prod_{i=1}^{n}f(x_i)$ and $G(\bar{x})=\prod_{i=1}^{n}g(x_i)$, where $f(x)$ and $g(x)$ are probability density functions, and $\bar{x}=(x_1,\ldots,x_n)$. The variational distance between $F$ and $G$ is: $$V(F,G)=\int |F(\bar{x})-G(\bar{x})|d\bar{x}$$ Can we write it in terms of the variational distance between $f$ and $g$?

I know we could do such if it was KL divergence: $$D_{KL}(F,G)=n D_{KL}(f,g).$$ Do we have such a simplification for variational distance as well?

Albert
  • 359
  • 7
    No, we don't. The TV distance simply doesn't tensorize well. A saving grace, however, is the fact that TV is easily bounded by many $f$-divergences (e.g., Pinsker's inequality says $V \le \sqrt{2 D_{KL}}.$), which is many times sufficient to solve the problem at hand. Alternately one has the choice to 'metrise' the problems using some other divergence. KL, $\chi^2,$ Jensen-Shannon and Hellinger are popular choices, with the last two both being symmetric in their arguments, and being 'close to' actual norms, if that is important. – stochasticboy321 Sep 14 '18 at 02:49

1 Answers1

4

The property of the KL divergence, is that it has a "chain rule" which leads to tensorization i.e. a property in one dimension holding in multiple dimensions.

The TV distance, while being convenient because it's a distance (and having a nice formula in simple spaces) does not tensorize the way the divergences do. This is because, by Monge-Kantorovich duality (which is what Blackbird used in his answer in the statement above) the TV distance is a minimum under some coupling, of the size of the set where the coupled random variables differ. Certainly, assuming independence we get the trivial bound derived by Blackbird. However, because of the fact that there could be more "structure" to the couplings, and this structure could potentially be very complicated as the dimension increases (essentially, you have more coordinates to couple and play with), there is more "between the components" than there is in the components themselves. This is why one would expect just trivial bounds for the TV distance in larger dimensions.

However, with MK Duality, we convert the Bobkov-Gotze theorem (a generalization of Pinsker) to a "transportation cost inequality", which does tensorize. So in that case, bounds on the individual TV distances do give bounds on the big TV distance. Marton's theorem is the one I know here : find it in Van Handel's notes chaptet 4 section 4 : https://web.math.princeton.edu/~rvan/APC550.pdf

Yes, in general it is very important to choose which distance you work with when you are dealing with inequalities. The fact that TV doesn't tensorize serves to emphasize that.

Sarvesh Ravichandran Iyer
  • 77,817