Raising the bar

Chapter 7. Sampling and Sampling Distribution

Unit 7.2. Sampling distribution of sample variance

Definition 7.4 (Unbiased and biased estimators). Let $latex T$ be a statistic designed to estimate (to approximate) a population parameter $latex \tau$. $latex T$ is called an estimator of $latex \tau$. To recapitulate, it is a random variable and it depends on the sample data. We say that $latex T$ is an unbiased estimator of $latex \tau$ if $latex ET = \tau$. If $latex ET > \tau$ we say that $latex T$ is biased upward, and if $latex ET < \tau$ it is biased downward.

Comments. (1) If we think of $latex \tau$ as the target, then the equation $latex ET = \tau$ means that $latex T$ hits the target on average (in a large number of trials).

(2) From Eq. (5.13) we know that the sample mean is an unbiased estimator of the population mean. In general, it is believed that for any population parameter there exists a statistic that estimates it. For example, the sample variance $latex s_X^2$ is an unbiased estimator of the population variance $latex \sigma_X^2$:

$latex Es_X^2 = \sigma_X^2. \qquad (\textrm{Eq. 7.1})$

This is the true reason of definition Eq. (3.5).

Controlling variances in sampling is important for at least three reasons:

1. In a production process of a certain product large deviations of its characteristics from technological requirements are undesirable.
2. The statistical procedure for comparing sample means from two populations requires knowledge of the corresponding variances.
3. In survey sampling by reducing the sample variance we reduce the cost of collecting the sample.

Exercise 7.1. Study the proof in section "Mean of the sampling distribution of the sample variance" in the Appendix of Chapter 6 of NCT and work out a version of the proof for yourself with two purposes in mind: (i) you should be able to reproduce the proof with complete explanations and (ii) try to extract general ideas that can be applied in other situations.

Solution. Your solution may be very different from mine. From the proof we see that the variables must be i.i.d.

Step 1. The beginning of the proof reminds me the derivation of the shortcut for variance. Let us write the shortcut formula as $latex E(Y - EY)^2 = EY^2 - (EY)^2$ and let us apply it to a random variable $latex Y$ with values $latex Y_1, \ldots, Y_n$ and uniform distribution $latex p_i = 1/n$. Then from the above formula we get

$latex \frac{1}{n} \sum (Y_i - \bar Y)^2 = \frac{1}{n} \sum Y_i^2 - (\bar Y)^2.$

Choosing $latex Y_i = X_i - \mu_X$ we get $latex \bar Y = \bar X - \mu_X$ and

$latex \frac{1}{n} \sum (X_i - \bar X)^2 = \frac{1}{n} \sum (X_i - \mu_X)^2 - (\bar X - \mu_X)^2. \qquad (\textrm{Eq. 7.2})$

This is an equivalent of the first step in NCT. Introducing $latex \mu_X$ here is necessary because in $latex V(X) = E(X - \mu_X)^2$ we need deviations from the population mean.

This trick of obtaining identities involving summations from general properties of means will prove useful again in Chapter 10. Always try to see general features in particular facts.

Step 2. We intend to apply the population mean to both sides of Eq. (7.2). It's important to distinguish the population mean from the sample mean. Since $latex X_i$ are identically distributed we have

$latex E(X_i - \mu_X)^2 = E(X - \mu_X)^2 = \sigma_X^2. \qquad (\textrm{Eq. 7.3})$

Now equations (7.2), (7.3), (5.13) and the definition of sample variance Eq. (3.5) imply

$latex Es_X^2 = E\frac{1}{n-1} \sum (X_i - \bar X)^2 = \frac{n}{n-1} E \frac{1}{n} \sum (X_i - \bar X)^2$

$latex = \frac{n}{n-1} \left[ \frac{1}{n} \sum E(X_i - \mu_X)^2 - E(\bar X - \mu_X)^2 \right]$

$latex = \frac{n}{n-1} \left[ \sigma_X^2 - \frac{1}{n} \sigma_X^2 \right] = \sigma_X^2. \quad \blacksquare$

In Eq. (7.1) we have found the mean of the sample variance. Now we tackle a much more ambitious task of fully characterizing the distribution of the sample variance.

Definition 7.5 (Chi-square distribution). Let $latex z_1, \ldots, z_n$ be independent standard normal (we know from Joint distribution and independence how to construct them). Then the variable $latex \chi_n^2 = \sum_{i=1}^n z_i^2$ is called a chi-square distribution with $latex n$ degrees of freedom.

Comments. (1) "Degrees of freedom" is just a formal parameter that determines which entry to look up in a statistical table.

(2) As you see, the general definition of $latex \chi_n^2$ has nothing to do with sampling. Those chi-square that arise in applications have degrees of freedom that depend not only on the sample size but also on the statistical procedure.

Figure 7.1. Densities of chi-squared for degrees of freedom 3, 7, 10.

Property 1. $latex \chi_n^2$ is nonnegative, therefore its density $latex p_{\chi_n^2}(t)$ vanishes for $latex t < 0$, see Figure 7.1.

Property 2. $latex E\chi_n^2 = n$.

Proof. $latex E\chi_n^2 = \sum_{i=1}^n Ez_i^2 = n$ by linearity of means and Property 3 of standard normals (see The normal distribution). $latex \blacksquare$

Property 3. $latex V(\chi_n^2) = 2n$.

Proof. By independence and the shortcut for variance

$latex V(\chi_n^2) = \sum_{i=1}^n V(z_i^2) = \sum_{i=1}^n [Ez_i^4 - (Ez_i^2)^2].$

We accept without proof that $latex Ez^4 = 3$ for a standard normal. Hence, $latex V(\chi_n^2) = \sum_{i=1}^n (3 - 1) = 2n$. $latex \blacksquare$

Usually I use the name "Theorem" for complex statements that will not be proved here.

Theorem 7.1 (Distribution of sample variance). For a random sample of $latex n$ independent observations from a normal population with variance $latex \sigma_X^2$ the variable

$latex \chi_{n-1}^2 = \frac{(n-1)s_X^2}{\sigma_X^2}$

is distributed as chi-square with $latex n - 1$ degrees of freedom.

Exercise 7.2. Find $latex V(s_X^2)$.

Solution. By the preceding theorem, homogeneity of variance and Property 3 of chi-square distributions

$latex V(s_X^2) = V\left( \frac{\sigma_X^2}{n-1} \chi_{n-1}^2 \right) = \frac{\sigma_X^4}{(n-1)^2} V(\chi_{n-1}^2) = \frac{2\sigma_X^4}{n-1}. \quad \blacksquare$

As you get used to algebra, don't write every little detail. Try to perform more operations mentally.

Exercise 7.3. Find a confidence interval for sample variance $latex s_X^2$, for an i.i.d. sample from a normal population.

Solution. Since $latex s_X^2$ is positive, in the expression $latex P(a \le s_X^2 \le b)$ it makes sense to consider only nonnegative $latex a$. You might want to review the proof of Corollary 6.2 (see Normal approximation to the binomial), namely, the way we pass from one inequality to another, equivalent, inequality. By Theorem 7.1

$latex P(a \le s_X^2 \le b) = P\left( \frac{(n-1)a}{\sigma_X^2} \le \frac{(n-1)s_X^2}{\sigma_X^2} \le \frac{(n-1)b}{\sigma_X^2} \right)$

$latex = P\left( \frac{(n-1)a}{\sigma_X^2} \le \chi_{n-1}^2 \le \frac{(n-1)b}{\sigma_X^2} \right).$

If we want this probability to be $latex 1 - \alpha$, we can use Table 7 from Appendix Tables in NCT (or Excel and Table 14.7 in this book) to find $latex a, b$ so that both tails have probability $latex \alpha/2$:

$latex P\left( \chi_{n-1}^2 < \frac{(n-1)a}{\sigma_X^2} \right) = P\left( \chi_{n-1}^2 > \frac{(n-1)b}{\sigma_X^2} \right) = \frac{\alpha}{2}. \quad \blacksquare$

Previous: Sampling distribution | Next: Monte Carlo simulations

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.