7

The MLE estimator of the variance of a normal distribution is $\hat \sigma^2 = \frac{1}{N} \sum_{i=1}^{N}(X_i - \bar X)^2$, where $\bar X$ is the sample mean and $X_i \sim^{iid} \mathcal{N}(\mu,\sigma^2)$ . I'm curious because I've seen (e.g. http://www.stat.ufl.edu/~winner/sta6208/allreg.pdf , p.20) that the variance of this estimator is equal to $\frac{2\sigma ^4}{N}$, but I find something different:

Since $\frac{1}{\sigma ^2} \sum_{i=1}^{N}(X_i-\bar X)^2 \sim \chi^2_{N-1}$, we have that

\begin{align} var\big(\frac{N\hat \sigma^2}{\sigma^2}\big) = var(\chi^2_{N-1}) = 2(N-1) \\ \implies var(\hat \sigma ^2) = \frac{2(N-1) \sigma^4}{N^2} \end{align}

What am I missing?

Amanjo
  • 127

1 Answers1

3

The variance of the estimator in the course notes is based on maximum likelihood estimation which typically results in biased estimators. The second variance calculation has a "correction" term that makes the estimator unbiased. You have likely seen this phenomenon with the unbiased estimator for the sample mean, i.e., dividing by $n-1$ instead of $n$.

There are lots of different ways to generate estimators and the resulting estimators will have different properties. One property that many people like in their estimators is for them to be unbiased. However, people are sometimes willing to accept a little bias to reduce variance. If you interested in this topic you might want to look up bias-variance tradeoff.

I noticed several years after my original answer there is a small typo in your derivation that makes a difference: $\frac{(N-1) \hat{\sigma}^2}{\sigma^2} \sim \chi_{n-1}^2$. You will still get a different answer to the one in the notes if you start with this instead of $\frac{N \hat{\sigma}^2}{\sigma^2}$. Starting with this you should find that $$ \textrm{var}\; (\hat{\sigma}^2) = \frac{2\sigma^4}{N-1}. $$

shoda
  • 1,478
  • So, if I don't divide by $n-1$, the bias is not 0 and the variance will be $\frac{2(n−1)\sigma^4}{n^2}$, so the bias and the variance are greater than in the case where I divide by $n-1$? – Amanjo Sep 23 '17 at 01:54
  • The estimators themselves are random and have distributions. The bias of an estimator quantifies how far its mean over all samples (usually of a particular size, e.g., $n$) in the population is from the (true/actual but unknown) value of the parameter you are estimating. Similarly, an estimator has its own variance which roughly conveys how far (on average) an estimate based on a particular sample can be from the value of the parameter you are estimating. It might be confusing because you are estimating a variance and both estimators (notes & yours) of the variance have their own variances. – shoda Sep 23 '17 at 02:01
  • I got it. The variance in the no bias case (i.e. dividing by $n-1$) is greater than in the bias case (i.e. dividing by $n$). – Amanjo Sep 23 '17 at 02:09
  • 1
    @shoda, often we may see $2 \sigma^4 / (N-1) $ estimator of $Var(s^2)$. Is it the third case compared to the OP two? If so, which variable does $2 \sigma^4 / N$ correspond to? – garej Jul 31 '21 at 16:14
  • @garej Good observation. I just realized that the derivation in the original question has a small typo. I will update my answer to reflect this. – shoda Aug 02 '21 at 01:07
  • The second variance calculation $\frac{2(N-1) \sigma^4}{N^2}$ is also biased maximum likelihood estimation. – Dan Li Oct 01 '21 at 19:43