Raising the bar

Chapter 12. Analysis of variance (ANOVA)

Unit 12.2. One-way ANOVA: distributions and the F-test

In the next theorem the result on distribution of sample variance (Theorem 7.1) is repeated in a form more convenient here.

Theorem 12.1. If $latex X_1, \ldots, X_n$ are independent observations from a normal population with parameters $latex \mu, \sigma$, then $latex \sum_{i=1}^n (X_i - \bar X)^2 / \sigma^2$ is distributed as $latex \chi_{n-1}^2$.

Theorem 12.2 (Chi square addition and subtraction).

(a) Two independent chi square variables $latex \chi_n^2$ and $latex \chi_m^2$ can be added to give a chi square variable $latex \chi_{n+m}^2$.

(b) If we know that a chi square variable $latex \chi_{n+m}^2$ is a sum of two variables, $latex \chi_{n+m}^2 = X_1 + X_2$, one of which is a chi square variable $latex X_1 = \chi_n^2$, and if $latex X_1, X_2$ are independent, then the other is also a chi square variable, $latex X_2 = \chi_m^2$.

Proof. (a) By definition, $latex \chi_n^2$ is a sum of squares of $latex n$ independent standard normals. $latex \chi_m^2$ is also a sum of squares of $latex m$ independent standard normals. We know that the standard normals in the first set are independent of those in the second set. By joining the two sets we obtain a set of $latex n+m$ independent standard normals. Therefore the sum $latex \chi_n^2 + \chi_m^2$ is distributed as $latex \chi_{n+m}^2$.

(b) For the proof of this part we refer to Bingham, N. H., Fry, John M., Regression: Linear Models in Statistics. Series: Springer Undergraduate Mathematics Series. 2010.

Both for addition and subtraction induction can be used to apply these properties to sums of more than two variables.

Exercise 12.2.

$latex \quad$(1) Find the distribution of $latex \sum_{i=1}^{n_g} (x_{gi} - \bar x_g)^2$.

$latex \quad$(2) Characterize the distribution of $latex SSW$. State the conclusions in terms of $latex MSW = SSW/(n-G)$.

Solution. (1) By Theorem 12.1

$latex \sum_{i=1}^{n_g} (x_{gi} - \bar x_g)^2 / \sigma^2 = \chi_{n_g - 1}^2.$

(2) Remembering that the samples are independent, by Theorem 12.2 from the above equation we have

$latex SSW = \sigma^2 \sum_{g=1}^G \chi_{n_g - 1}^2 = \sigma^2 \chi_{n - G}^2. \qquad (\textrm{Eq. 12.1})$

This equation rewrites in a familiar format

$latex \frac{(n - G) MSW}{\sigma^2} = \frac{SSW}{\sigma^2} = \chi_{n - G}^2. \qquad (\textrm{Eq. 12.2})$

These results hold regardless of whether or not $latex H_0$ is true.

Exercise 12.3.

$latex \quad$(1) What is the distribution of $latex SST$ under the null hypothesis?

$latex \quad$(2) Find the distribution of $latex SSB$ under the null. State the conclusion in terms of $latex MSG = SSB/(G - 1)$.

$latex \quad$(3) Find the distribution of $latex MSG/MSW$ under the null.

Solution. (1) Under the null all observations come from the same population with the same mean $latex \mu$ and variance $latex \sigma^2$, and we have one large sample of size $latex n$. Application of Theorem 12.1 gives

$latex SST/\sigma^2 = \sum_{g=1}^G \sum_{i=1}^{n_g} (x_{gi} - \bar x)^2 / \sigma^2 = \chi_{n - 1}^2. \qquad (\textrm{Eq. 12.3})$

(2) Under the null hypothesis equations (Eq. 12.2) and (Eq. 12.3) imply

$latex SSB/\sigma^2 = SST/\sigma^2 - SSW/\sigma^2 = \chi_{n - 1}^2 - \chi_{n - G}^2 = \chi_{G - 1}^2.$

According to Theorem 12.2, the last step needs justification in the form of independence of $latex SSB$ and $latex SSW$, which we cannot do here. We take independence for granted. It follows that

$latex (G - 1) MSG / \sigma^2 = \chi_{G - 1}^2.$

(3) From the last equation and (Eq. 12.2) it's easy to see that

$latex MSG/MSW = \frac{\chi_{G - 1}^2 / (G - 1)}{\chi_{n - G}^2 / (n - G)} = F_{G - 1, n - G}.$

Idea. If variability between groups is higher than variability within groups, then group data come from populations with different means.

Decision rule. We reject $latex H_0$ at the $latex \alpha$ significance level if $latex MSG/MSW > F_{G - 1, n - G, \alpha}$ where the critical value $latex F_{G - 1, n - G, \alpha}$ is defined by $latex P(F_{G - 1, n - G} > F_{G - 1, n - G, \alpha}) = \alpha$.

Table 12.2. Format of one-way ANOVA table.

Exercise 12.4. Indicate the relations between different quantities in Table 12.2 and degrees of freedom for the statistics in the last two columns. Write out the null and alternative hypotheses and the decision rule based on the selected level of significance.

Previous: One-way ANOVA: setup and the SST decomposition | Next: Two-way ANOVA: setup and population model

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.