Raising the bar

Chapter 10. Simple regression

Unit 10.3.

Section 10.3.4. Variances of OLS estimators

In addition to the basic assumptions of Unit 10.2 suppose that

$latex E u_1^2 = \ldots = E u_n^2 = \sigma_u^2 \quad (\textit{homoscedasticity}) \qquad (\textrm{Eq. 10.10})$

$latex E u_i u_j = 0 \textrm{ for all } i \ne j \quad (\textit{absence of autocorrelation}) \qquad (\textrm{Eq. 10.11})$

Comments. (1) Homoscedasticity means that variances of the error terms are the same across observations. It holds, in particular, when the errors are identically distributed. In economics usually variances of variables are proportionate to their levels (variation in income grows with income) so the errors are heteroscedastic. Check that the conditions imposed so far imply $latex V(u_1) = \ldots = V(u_n) = \sigma_u^2$.

(2) Absence of autocorrelation holds, in particular, for independent errors (provided that their means are null). Check that condition (10.11) can be equivalently written as $latex \textrm{cov}(u_i, u_j) = 0$ for $latex i \ne j$. Time series data are usually autocorrelated.

Exercise 10.2. Prove that under the above assumptions

$latex \sigma_{b_2}^2 = \frac{\sigma_u^2}{\sum_{i=1}^n (x_i - \bar x)^2}, \quad \sigma_{b_1}^2 = \sigma_u^2 \left( \frac{1}{n} + \frac{\bar x^2}{\sum_{i=1}^n (x_i - \bar x)^2} \right). \qquad (\textrm{Eq. 10.12})$

Proof. Step 1. We need the following generalization of the "mean $latex +$ deviation-from-the-mean" representation. Suppose that a random variable $latex Z$ is a linear combination of the errors with constant coefficients:

$latex Z = \sum a_i u_i. \qquad (\textrm{Eq. 10.13})$

Then $latex E Z = \sum a_i E u_i = 0$ and

$latex E Z^2 = V \left( \sum a_i u_i \right) \quad \textrm{(the mean of } Z \textrm{ is zero)}$

$latex = \sum V(a_i u_i) \quad \textrm{(by additivity of variance)}$

$latex = \sum a_i^2 V(u_i) \quad \textrm{(by homogeneity of variance)}$

$latex = \sigma_u^2 \sum a_i^2 \quad (\textrm{using Eq. 10.10}).$

Step 2. To make use of Step 1, we obtain a representation of type Eq. (10.13) for $latex b_2$. Rewrite the working formula Eq. (10.8) as

$latex b_2 = \beta_2 + \frac{\sum_{i=1}^n (x_i - \bar x)(u_i - \bar u)}{\sum_{i=1}^n (x_i - \bar x)^2}.$

Split the numerator into two pieces:

$latex b_2 = \beta_2 + \frac{\sum_{i=1}^n (x_i - \bar x) u_i}{\sum_{i=1}^n (x_i - \bar x)^2} + \frac{\sum_{i=1}^n (x_i - \bar x) \bar u}{\sum_{i=1}^n (x_i - \bar x)^2}.$

Here $latex \bar u$ does not depend on the index of summation $latex i$ so

$latex \sum_{i=1}^n (x_i - \bar x) \bar u = \bar u \sum_{i=1}^n (x_i - \bar x) = \bar u (n \bar x - n \bar x) = 0.$

The result is

$latex b_2 = \beta_2 + \sum_{i=1}^n a_i u_i, \qquad (\textrm{Eq. 10.14})$

where

$latex a_i = \frac{x_i - \bar x}{\sum_{i=1}^n (x_i - \bar x)^2} = \frac{x_i - \bar x}{n V_u(x)}.$

Note: here we applied a simple trick. In the expression $latex (\sum_i a_i)/(\sum_i b_i)$ the sum in the bottom is just a constant, so it can be introduced under the summation sign in the top to give $latex \sum_i (a_i / \sum_i b_i)$. Remember that the sums do not depend on the summation index $latex i$.

Step 3. In Eq. (10.14) the coefficients $latex a_i$ are deterministic. Therefore by unbiasedness and Step 1

$latex \sigma_{b_2}^2 = E(b_2 - E b_2)^2 = E(b_2 - \beta_2)^2$

$latex = E \left( \sum_{i=1}^n a_i u_i \right)^2 = \sigma_u^2 \sum_{i=1}^n a_i^2. \qquad (\textrm{Eq. 10.15})$

The sum of squares involved here can be found using properties of means:

$latex \sum_{i=1}^n a_i^2 = n \overline{a^2} = \frac{n}{n^2 V_u^2(x)} \overline{(x - \bar x)^2}$

$latex = \frac{V_u(x)}{n V_u^2(x)} = \frac{1}{\sum (x_i - \bar x)^2}. \qquad (\textrm{Eq. 10.16})$

The last two equations prove the first equation in Eq. (10.12).

Step 4. We need to derive a representation similar to Eq. (10.14) for $latex b_1$. Using equations (10.9) and (10.14),

$latex b_1 = \beta_1 + \beta_2 \bar x + \bar u - \left( \beta_2 + \sum a_i u_i \right) \bar x$

$latex = \beta_1 + \frac{1}{n} \sum u_i - \sum a_i \bar x u_i$

$latex = \beta_1 + \sum \left( \frac{1}{n} - a_i \bar x \right) u_i.$

With the notation $latex c_i = \frac{1}{n} - a_i \bar x$ this becomes $latex b_1 = \beta_1 + \sum c_i u_i$. In the way similar to Eq. (10.15) we get

$latex \sigma_{b_1}^2 = E(b_1 - E b_1)^2 = E(b_1 - \beta_1)^2$

$latex = E \left( \sum_{i=1}^n c_i u_i \right)^2 = \sigma_u^2 \sum_{i=1}^n c_i^2.$

It remains to calculate the sum of squares (see Eq. (10.16)):

$latex \overline{c^2} = \overline{\left( \frac{1}{n} - a \bar x \right)^2} = \overline{\frac{1}{n^2} - 2 a \bar x \frac{1}{n} + a^2 \bar x^2}$

$latex = \frac{1}{n^2} - 2 \bar a \bar x \frac{1}{n} + \overline{a^2} \, \bar x^2$

$latex = \frac{1}{n^2} + \frac{\bar x^2}{n \sum (x_i - \bar x)^2}$

(using $latex \bar a = 0$). The second of the formulas (10.12) follows from the last two equations.

---

Previous: Unbiasedness of OLS estimators | Next: Statistics for testing hypotheses

---

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.