Union Bound of two events?

Question

I am trying to understand the assumption proof of Theorem 2(Page -$7$) in the paper "A Universal Law of Robustness via isoperimetry" by Bubeck and Sellke.

Inequality 1

\begin{align} \mathbb{P}\left(\frac{1}{n} \sum_{i=1}^{n}\left(f\left(x_{i}\right)-\mathbb{E}[f]\right) z_{i} \geq \frac{\epsilon}{8}\right) \leq 2 \exp \left(-\frac{\epsilon^{2} n d}{9^{4} c L^{2}}\right) \end{align}

Since we assumed that the range of the functions is in $[-1,1]$ we have $\mathbb{E}[f] \in[-1,1]$ and hence:

Inequality 2

$$ \mathbb{P}\left(\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n} \mathbb{E}[f] z_{i} \geq \frac{\epsilon}{8}\right) \leq \mathbb{P}\left(\left|\frac{1}{n} \sum_{i=1}^{n} z_{i}\right| \geq \frac{\epsilon}{8}\right) $$

By Hoeffding's inequality, the above quantity is smaller than $2 \exp \left(-n \epsilon^{2} / 8^{3}\right)$ (recall that $\left.\left|z_{i}\right| \leq 2\right)$.

Thus we obtain with an union bound:

Inequality 3

$$ \begin{aligned} \mathbb{P}\left(\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n} f\left(x_{i}\right) z_{i} \geq \frac{\epsilon}{4}\right) & \leq|\mathcal{F}| \cdot \mathbb{P}\left(\frac{1}{n} \sum_{i=1}^{n}\left(f\left(x_{i}\right)-\mathbb{E}[f]\right) z_{i} \geq \frac{\epsilon}{8}\right)+\mathbb{P}\left(\left|\frac{1}{n} \sum_{i=1}^{n} z_{i}\right| \geq \frac{\epsilon}{8}\right) \\ & \leq 2|\mathcal{F}| \cdot \exp \left(-\frac{\epsilon^{2} n d}{9^{4} c L^{2}}\right)+2 \exp \left(-\frac{n \epsilon^{2}}{8^{3}}\right) \end{aligned} $$

---

I am not getting how Union bound is getting happened using Ineq 1 and 2. Can anyone help me with that how they able to reach last inequality?

Answer 1

Assume there are three events $E_{7}$,$E_{8}$ and $E_{9}$ such as $E_{7} := \underbrace{\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n}\left(f\left(x_{i}\right)-\mathbb{E}[f]\right) z_{i}}_{A} \leq \frac{\epsilon}{8}$

and $E_{8} := \underbrace{\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n} \mathbb{E}[f] z_{i}}_{B} \leq \frac{\epsilon}{8}$

and $E_{9} := \underbrace{\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n} f\left(x_{i}\right) z_{i}}_{C}\leq \frac{\epsilon}{4}$

For simplicity I can write

\begin{align} A - \frac{\epsilon}{8} \leq 0 \end{align} \begin{align} B - \frac{\epsilon}{8} \leq 0 \end{align}

Adding Eq $19$ and $20$

\begin{align} A + B \leq \frac{\epsilon}{4} \end{align}

which is $C \leq \frac{\epsilon}{4} $

Hence, $$ \mathbb{P}(E_{7} \cap E_{8}) \leq \mathbb{P}(E_{9}) $$

Now

$$ \mathbb{P}(E_{7} \cap E_{8}) = 1-\mathbb{P}((E_{7} \cap E_{8})^{c}) \\ =1- \underbrace{\mathbb{P}\left(E_{7}^{c} \cup E_{8}^{c}\right)}_{N} $$

\begin{align} N &\leq \mathbb{P}(E_{7}^{c}) + \mathbb{P}(E_{8}^{c})\\ &\leq 2 - (\mathbb{P}(E_{7}) + \mathbb{P}(E_{8}))\\ \end{align}

Now $$ - \mathbb{P}\left(E_{7}^{c} \cup E_{8}^{c}\right) \geq -2 + (\mathbb{P}(E_{7}) + \mathbb{P}(E_{8})) $$ Adding $1$ both sides, $$ 1 - \mathbb{P}\left(E_{7}^{c} \cup E_{8}^{c}\right) \geq -1 + (\mathbb{P}(E_{7}) + \mathbb{P}(E_{8})) $$

Now $$\mathbb{P}(E_{9}) \geq -1 + (\mathbb{P}(E_{7}) + \mathbb{P}(E_{8}) $$

Now $$ - \mathbb{P}(E_{9}) \leq 1 - \left( (\mathbb{P}(E_{7}) + \mathbb{P}(E_{8}) \right) $$

Adding $1$ both side, $$ 1 - \mathbb{P}(E_{9}) \leq \left(1 - \left( (\mathbb{P}(E_{7})\right) + \left(1 - \mathbb{P}(E_{8})\right) \right) $$

Finally We can write,

\begin{align} \nonumber \mathbb{P}\left(\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n} f\left(x_{i}\right) z_{i} \geq \frac{\epsilon}{4}\right) & \leq \cdot \mathbb{P}\left(\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n}\left(f\left(x_{i}\right)-\mathbb{E}[f]\right) z_{i} \geq \frac{\epsilon}{8}\right)+\mathbb{P}\left(\left|\frac{1}{n} \sum_{i=1}^{n} z_{i}\right| \geq \frac{\epsilon}{8}\right) \\ & \leq 2 \left|\mathcal{F}\right| \cdot \exp \left(-\frac{\epsilon^{2} n d}{9^{4} c L^{2}}\right)+2 \exp \left(-\frac{n \epsilon^{2}}{8^{3}}\right) \end{align}

N.B:-

\begin{align} \mathbb{P}\left(\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n} f\left(x_{i}\right) z_{i} \geq \frac{\epsilon}{4}\right) &\leq \mathbb{P}\left ( \bigcup_{f \in \mathcal{F}} \{ \frac{1}{n}\left| \sum_{i=1}^n (f(x_i) - \mathbb{E}[f])z_i\right| \geq \frac{\epsilon}{8}\} \right ) &\leq \sum_{f \in \mathcal{F}} \mathbb{P}\left (\frac{1}{n}\left| \sum_{i=1}^n (f(x_i) - \mathbb{E}[f])z_i\right| \geq \frac{\epsilon}{8} \right ) \leq 2 |F| \exp\left (-\frac{\epsilon^2nd}{9^4cL^2} \right) \end{align}

Answer 2

By contradiction you can exam if right hand sentence happens not, add them together and the left happens not. You can take $\mathbb{E}[f]=\text{sgn}(z_i)$ to minimize the left hand sentences.