Raising the bar

Chapter 10. Simple regression

Unit 10.1. Algebra of sample means

This chapter will be densely populated. It's because solutions to some exercises from NCT are completely spelled out. This chapter covers the first chapter of a standard econometrics text.

There will be many small and large facts provided with small and large derivations. In order not to be lost, write down the facts on post-it notes. Move the notes around, try to prove the facts and establish relationships between them on your own.

Motivating example. Suppose we want to know the relationship between the weight and height of a person. We have to think what depends on what: weight on height or height on weight. Logically, it is weight that depends on height. In practice, we obtain a series of measurements $latex (w_1, h_1), \ldots, (w_n, h_n)$ on $latex n$ individuals. It's important to realize that the data do not contain any functional dependence and there may not be one. Even if there is one, $latex w = f(h)$, we may never know it. As a first approximation, we simplify the problem and seek a linear dependence $latex w_i = \beta_1 + \beta_2 h_i$, where $latex \beta_1, \beta_2$ are some unknown numbers. As statistical data are usually irregular, an exact dependence of this type does not exist. Therefore the dependence we are looking for should contain errors: $latex w_i = \beta_1 + \beta_2 h_i + u_i$.

Thus, there are $latex n$ observations. The observations come in pairs (two-dimensional vectors). The first component of a vector is thought to be a linear function of the second, plus the error term. This dependence may not exist in reality. It is something that we are imagining. The coefficients and the error terms are unobservable. The coefficients are the parameters that we need to estimate.

Algebra of sample means. For a discrete random variable $latex X$ with values $latex X_1, \ldots, X_n$ and uniform distribution denote $latex E_u X = X_1 \frac{1}{n} + \ldots + X_n \frac{1}{n}$. This is just the sample mean: $latex E_u X = \bar X$. Therefore all properties we have established for the mean value operator translate to corresponding properties of sample means. We need:

$latex \quad$(a) linearity of means $latex \overline{(aX + bY)} = a \bar X + b \bar Y$,

$latex \quad$(b) shortcut for covariance $latex \overline{(X - \bar X)(Y - \bar Y)} = \overline{XY} - \bar X \bar Y$ and

$latex \quad$(c) shortcut for variance $latex \overline{(X - \bar X)^2} = \overline{X^2} - (\bar X)^2$

(pay attention to the order of operations).

To see usefulness of these abbreviations, compare the last equation to its full form (and try to prove the full form):

$latex \frac{1}{n} \sum_{i=1}^n (X_i - \bar X)^2 = \frac{1}{n} \sum_{i=1}^n X_i^2 - \left( \frac{1}{n} \sum_{i=1}^n X_i \right)^2.$

The tricky part of the notation just introduced is that $latex X_1, \ldots, X_n$ can be treated not as real values of a random variable $latex X$ but as random variables sampled from the same population (this is why they are written using capital X's). To emphasize the distinction between the population mean and sample mean, I use the notation $latex E_u$ for the latter. It gives rise to

$latex \textrm{cov}_u(X, Y) = \overline{(X - \bar X)(Y - \bar Y)}, \quad V_u(X) = \overline{(X - \bar X)^2}.$

Previous: Questions for repetition (Chapter 9) | Next: Linear model setup

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.