Raising the bar

Chapter 12. Analysis of variance (ANOVA)

Unit 12.1. One-way ANOVA: setup and the $latex SST$ decomposition

ANOVA comes with several complex formal relationships. Its reliance on the linear model adds to its complexity. The outcome of the theoretical study is summarized in the form of a table. Interpreting that table is just another exercise in hypothesis testing. Studying ANOVA for the sake of the outcome interpretation does not justify its inclusion in a statistics course. My suggestion is to either skip it or study with the theory. There are two basic forms of ANOVA: one-way and two-way. The two-way ANOVA, in my view, is too complex for an introductory course. For both forms the algebraic derivations are given in full and only one probabilistic argument (the subtraction property of chi square variables) is not proved.

Motivating example (from NCT). Suppose that it is our objective to compare the fuel consumption for three different makes of automobile, A-cars, B-cars and C-cars. We take 20 drivers and randomly assign 7 of them to A-cars, 7 to B-cars and 6 to C-cars. The 20 resulting fuel consumption figures are naturally divided in three groups of observations. There will be differences in mean fuel consumption between the groups. We would like to know if those differences arise by chance or if there is some kind of statistical evidence that the three groups of observations come from three different populations. This question can be addressed in many different ways. The answer presented here is based on the following idea due to Ronald Fisher: if variability between samples is large relative to variability within samples, then the samples are likely to come from different populations.

Comments. (1) Unlike the methods designed for matched pairs and independent samples, this method allows us to compare sample means from more than two samples.

(2) Differences in fuel consumption may arise from differences in driving habits of the drivers (aggressive drivers accelerate more and brake more, which leads to more gas burning). The random assignment of available drivers to different car makes allows us to eliminate the dependence of fuel consumption on drivers.

(3) No method is perfect, and each one has its limitations. Here the limitation is that the population variance $latex \sigma^2$ is the same for all samples.

(4) On the other hand, there is no need to have equal numbers of trials for each type of car.

Formal setup. The total number of observations is denoted $latex n$ and the observations are divided in $latex G$ groups. The samples are independent and the population variance $latex \sigma^2$ is the same for all samples. The number of observations may change across groups: it is $latex n_1$ in the first one, $latex \ldots$, $latex n_g$ in group numbered $latex g$, $latex \ldots$, $latex n_G$ in the last group.

Table 12.1. Visualization of groups of observations.

Table12.1. Visualization of groups of observations.

In each group $latex g$ the observations are listed as $latex x_{g1}, x_{g2}, \ldots, x_{gn_g}$; the column lengths $latex n_1, \ldots, n_G$ may differ across groups.

Objective: test $latex H_0: \mu_1 = \ldots = \mu_G$ against $latex H_a: \mu_i \ne \mu_j$ for at least one pair $latex i \ne j$.

Definition 12.1 (Sums of squares for one-way ANOVA).

Measure of variability within group: $latex SS_g = \sum_{i=1}^{n_g} (x_{gi} - \bar x_g)^2$.

Total within-groups variability: $latex SSW = \sum_{g=1}^G SS_g = \sum_{g=1}^G \sum_{i=1}^{n_g} (x_{gi} - \bar x_g)^2$.

Measure of variability between groups: $latex SSB = \sum_{g=1}^G n_g (\bar x_g - \bar x)^2$.

Total sum of squares: $latex SST = \sum_{g=1}^G \sum_{i=1}^{n_g} (x_{gi} - \bar x)^2$.

Exercise 12.1.

$latex \quad$(1) Write down the expression for the mean $latex \bar x_g$ in $latex g$th group and for the total number of observations.

$latex \quad$(2) How is the overall mean $latex \bar x$ related to group means?

$latex \quad$(3) Prove that $latex SST = SSW + SSB$.

Solution. (1) From the notation introduced it is clear that

$latex \bar x_g = \frac{1}{n_g} \sum_{i=1}^{n_g} x_{gi}, \quad n = \sum_{g=1}^G n_g.$

(2) This is in fact the grouped data formula from Unit 5.2:

$latex \bar x = \frac{1}{n} \sum_{g=1}^G \sum_{i=1}^{n_g} x_{gi} = \frac{1}{n} \sum_{g=1}^G n_g \frac{1}{n_g} \sum_{i=1}^{n_g} x_{gi} = \frac{1}{n} \sum_{g=1}^G n_g \bar x_g.$

(3) In the expression of $latex SST$ we add and subtract $latex \bar x_g$ and then square the parentheses:

$latex SST = \sum_{g=1}^G \sum_{i=1}^{n_g} (x_{gi} - \bar x)^2$

$latex = \sum_{g=1}^G \sum_{i=1}^{n_g} (x_{gi} - \bar x_g)^2 + 2 \sum_{g=1}^G \sum_{i=1}^{n_g} (x_{gi} - \bar x_g)(\bar x_g - \bar x) + \sum_{g=1}^G \sum_{i=1}^{n_g} (\bar x_g - \bar x)^2.$

Here in the second and third sums $latex (\bar x_g - \bar x)$ does not depend on $latex i$ and can be pulled out of the inner sum, so

$latex SST = SSW + 2 \sum_{g=1}^G (\bar x_g - \bar x) \sum_{i=1}^{n_g} (x_{gi} - \bar x_g) + \sum_{g=1}^G (\bar x_g - \bar x)^2 \sum_{i=1}^{n_g} 1.$

The inner sum $latex \sum_{i=1}^{n_g} (x_{gi} - \bar x_g) = n_g \bar x_g - n_g \bar x_g = 0$, and $latex \sum_{i=1}^{n_g} 1 = n_g$, so

$latex SST = SSW + \sum_{g=1}^G n_g (\bar x_g - \bar x)^2 = SSW + SSB.$

Previous: Questions for repetition (Chapter 11) | Next: One-way ANOVA: distributions and the F-test

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.