How can the Central Limit Theorem apply to Finite Populations?

Question

In my statistics for beginners course we've just been introduced to the CLT, where it's been stated that a distribution of sample means tends to the normal dist. as your sample size $n$ increases to infinity.

But what if your population is finite (i.e. of size $N$), so that your max sample size can only be of size $N \ll \infty$? Will such a distribution (which must be that of nearly all practical statistical surveys)not follow the CLT?

---

My best attempt at thoughts on this so far go like this: If I were to take a random sample from my population of size N, each sample though containing just a single member of the pop, calculate and plot the 'mean' of each sample (which would just equal the single value) until I've sampled and plotted every member and done so for each only once, I would eventually of course replicate exactly the population distribution.

Suppose then I repeat the experiment, increasing my sample size each repetition, until my sample is of size $N$. I take a single sample, plot its mean, then by definition this is the same as the population mean $\mu$.

So here, as my sample size has increased, my distribution of sample means hasn't tended to the Normal - with an ever thinner distribution with flatter tails and a taller peak - but more like a hyper-idealised version of the Normal - a single value at the population mean.

Clearly then, for finite populations - if I've understood the idea behind the CLT correctly, which is a big if -the CLT does not apply, rather in these practical cases, their sample mean distribution approaches something approximately Normal? Is it the case then that the CLT is more a theoretical concept, that applies to infinitely large populations, from which samples sizes can tend to infinity?

---

Further to this, I've read for the CLT to apply, the random variables of your population have to be I.I.D - if I'm using SRS without replacement for a finite population, does that mean the variables aren't I.I.D anymore, and thus the CLT would also not apply because of this? If the population were infinite though and I used SRSWOR, would the r.v.'s then be I.I.D, thus meaning the CLT would apply?

I appreciate all your insight on this; I'm very new to statistics, so I apologise if a lot of this is pretty basic and if my thoughts were way off. Thanks for any help you can lend, really appreciate it.

Answer 1

As the aggregate of my comments on this questions have gotten quite large, I feel it appropriate to collect them into an answer.

The CLT only applies to independent samples so technically you cannot apply the CLT at all to a SRSWOR. However, in practice, you can treat a SRSWOR as a SRS with replacement provided your sample is no more than some fixed fraction of the population, say 10%. In which case, even though your sample is not independent, many introductory statistics courses say that it is “independent enough” to apply the CLT to. It is important to say that this is all an approximation. There may exist a formal justification for this approximation, but I have not seen it. Intuitively, as we take larger and larger samples with SRSWOR, initially the distribution of sample means becomes more and more normal.

However, as our samples grow quite large, as we increase our sample size the sampling distribution will become less normal as the dependence of the samples begins to kick in. Clearly, as you said, as $n \to N$, then the sample mean will become exactly the population mean. It is important to note that in many (but certainly not all) practical applications, even getting even 10% of your population is nigh-impossible, so this issue is not often encountered in practice. If you're interested in this topic and you know how to program, it might be interesting to perform some numerical experiments and explore the sampling distribution for yourself. To my knowledge, however, there is no theoretical justification that 10% (which as far as I know was kind of picked out of a hat) is some magical limit after which the normal convergence stops working. I imagine the "ceiling" where the CLT approximation starts getting worse would be dependent on your sample data and its distribution.

Provided you use a SRS with replacement, then your random sample will be IID so the CLT applies. It doesn't matter whether your population is finite or infinite. If I use SRS with replacement, then each element of the sample can be picked multiple times. So even if I had a population of $N=2$, I could still take a sample of $n=1000$ by simply picking the first element of my population with probability $0.5$ and the second with probability $0.5$. Thus with SRS with replacement, you can take arbitrarily large IID samples so the CLT holds.

---

As an aside, you refer to a distribution which has probability $1$ of being some constant $C$ as a uniform distribution. It is worth noting that a uniform distribution refers to the much broader class of probability distributions which have an equal probability of being any number between two constants $a$ and $b$. I would refer to the distribution which has probability $1$ of being some constant $C$ as having a constant distribution.

Answer 2

I'd like to add a bit more information.

First, there is a different central limit theorem for a finite population, but it's much uglier, so people don't use it unless they actually need to. Because we rarely sample a meaningful fraction of the population, we ignore it. We also ignore it because ignoring it makes our standard errors too big, rather than too small.

Second, most of the time we're not actually interested in the finite population. For instance, let's say we want to test the hypothesis that an employer discriminates against African-Americans. We sample the entire population and find a tiny, .1% difference in pay (adjusting for other factors). Should we use the finite population variance and say the p-value is 0? After all, we know with 100% probability that African-Americans are getting paid less at the company.

I'd guess not. Even though we know for sure that African-Americans are making a tiny bit less at this company, we're more interested in whether the underlying process involves discrimination. That underlying process is closer to the idealized situation, where we could in theory see what would happen if this company made an infinite number of wage offers to candidates. In that infinite-population setting, a 0.1% difference seems easy to explain as chance (assuming the sample size is small enough).

This might be different if you're e.g. taking a census and want to know the actual differences in income, but I've found these situations to be a lot rarer.