Why Does The "Bootstrap Method" Work?

Question

Consider the "Bootstrap Method" (https://en.wikipedia.org/wiki/Bootstrapping_(statistics)) in Probability and Statistics.

As I understand, the Bootstrap Method is a useful procedure that can be used to estimate the "empirical distribution" of some "statistic" (e.g. mean) for some observed data. In the Bootstrap Method:

• First, we take a random sample (e.g. 70%) of the collected data and calculate the "statistic" from this random sample.
• Next, repeat this above step many times. Each time, you will have a "version" of this "statistic" corresponding to each random resample.
• Finally, rank all these "versions" from smallest to largest - by taking the "version" corresponding to the 5th percentile and the 95th percentile from this ranked list, you can effectively place "Confidence Intervals" on this "statistic".

The Bootstrap Method is said to be particularly advantageous as it allegedly "works" in many otherwise difficult circumstances where a closed-form distribution for the "statistic" of interest might not be readily known or available. Our professor demonstrated that the Bootstrap Method does in fact work, and showed us some examples with randomly simulated data where the closed-form distributions for the "statistic" of interest is known - and it is easy to compare the solutions generated from the Bootstrap Simulations and the analytical answer. But in the back of my mind, I always play "Devils Advocate" and wonder - how do I know that the Bootstrap "just happens" to work in this example, and perhaps in the next example, we might not be as lucky.

I tried asking one of my professors as to why exactly the Bootstrap Method works - but the professor replied that its because of the Law of Large Numbers (https://en.wikipedia.org/wiki/Law_of_large_numbers). While this is probably true, I was hoping to find a more "detailed reason" as to why the Bootstrap Method works. As an example, (my understanding of) the Law of Large Numbers applies in situations where you have access to the entire population and can you can resample this population an infinite number of times - whereas in situations where the Bootstrap is used, you have a (possibly imperfect) sample from the original population, and can only resample this sample of the population. This makes me a bit unsure if extending the use of the Law of Large Numbers to justify the correctness of the Bootstrap is legitimate.

I found what seems to be a very informative University Lecture on this subject (https://www.stat.cmu.edu/~larry/=sml/Boot.pdf) in which proofs are even provided - but I don't think my knowledge of mathematics is currently adequate enough to understand this proof by myself.

I was hoping that perhaps someone here might be able to walk me through a simplified version of this proof - or perhaps provide another simplified version of a similar proof which demonstrates why the Bootstrap Method "works".

Thanks!

Answer 1

The bootstrap works because, by resampling from the sample, you are actually sampling from the empirical distribution of the data.

To take a step back, imagine that you have a situation where the distribution of the population is known. Each of the draws from the population in the random sample can be represented as a random variable, $Y_i$, whose distribution is given by a CDF $F_Y$. We can represent a random sample as a random vector $\mathbf{Y} = (Y_1,Y_2,\ldots, Y_n)$, and so the statistic you are interested in calculating is 1) a function of this random vector and 2) itself a random variable with its own distribution. Denote this statistic $T(\mathbf{Y})$ or $T$. Attempting to derive mathematically the distribution of $T$ is likely not feasible or at least not an easy task in general. It would be much easier if, instead, you could approximate the distribution of $T$ (or the mean or the standard deviation of $T$). To accomplish this, you would draw $M$ random samples of size $n$, $\mathbf{Y}^{(m)} = (Y_1^{(m)}, \ldots, Y_n^{(m)})$, from $F_Y$ and calculate the corresponding statistic $T^{(m)} = T(\mathbf{Y}^{(m)})$ for each random sample. This way, you have $M$ replications of $T$ which you can then use to approximate various properties of the distribution of $T$ (you could compute the mean of the $T^{(m)}$, for example).

Now, in practice, we do not actually know the true distribution of the population we are studying. What's the next best thing? The empirical distribution. Given observed data $(y_1,\ldots,y_n)$, the empirical cumulative density function (ECDF) of the data is given, in terms of indicator variables, as: $$\hat{F}(y) = \frac{1}{n}\sum_{j=1}^n I(y_j \leq y)$$ The ECDF is a step function which jumps to the value $k/n$, where $k$ is the number of the data points less than or equal to $y$. It turns out that, assuming the population has a true distribution $F$, the strong law of large numbers implies that $$\lim_{n \to \infty} \hat{F}(y) = P(Y \leq y)= F(y) \text{ with probability 1,} $$

for all $y \in \mathbb{R}$. This is what I think your professor was alluding to. This fact also makes us comfortable believing that the empirical distribution approximates the true distribution well (given an appropriately large sample size). So, we can use the procedure I described in the second paragraph, except with $F$ replaced by $\hat{F}$. Also, it turns out that taking $n$ random draws from $\hat{F}$ is the exact same thing as drawing a random sample with replacement of size $n$ from the data!