Raising the bar

Chapter 10. Simple regression

Unit 10.3. Ordinary Least Squares (OLS): definition and derivation

Section 10.3.3. Unbiasedness of OLS estimators

Exercise 10.1.

1. Main condition for existence of OLS estimators. If variance $latex V_u(x) \ne 0$ then the OLS estimators exist. The condition $latex V_u(x) \ne 0$ means that there is some variation in the regressor or, equivalently, the observed points are not on a vertical straight line.

2. Main condition for unbiasedness of OLS estimators. If, further, $latex E u_i = 0$ for all $latex i$ and the regressor is deterministic, then the OLS estimators are unbiased.

Proof. (i) The statement about existence is obvious because $latex V_u(x)$ is in the denominator of $latex b_2$.

(ii) Step 1. Obtaining the working formula for the slope estimator (the "mean $latex +$ deviation-from-the-mean" representation). Using notation (10.3) let us write the system of equations (10.1) in a vector form

$latex y = \beta_1 + \beta_2 x + u.$

Using linearity of covariances we rearrange the slope estimator

$latex b_2 = \frac{\textrm{cov}_u(x, \beta_1 + \beta_2 x + u)}{V_u(x)} = \frac{\textrm{cov}_u(x, \beta_1) + \beta_2 \textrm{cov}_u(x, x) + \textrm{cov}_u(x, u)}{V_u(x)}.$

Covariance of a constant with anything is zero. Covariance of a variable with itself is its variance. Therefore from the last equation we obtain a working formula for the slope estimator:

$latex b_2 = \beta_2 + \frac{\textrm{cov}_u(x, u)}{V_u(x)}. \qquad (\textrm{Eq. 10.8})$

Step 2. Proving unbiasedness of the slope estimator. Since, by assumption, $latex x$ is deterministic and the error has mean zero, we get unbiasedness of the slope estimator

$latex E b_2 = \beta_2 + \frac{\textrm{cov}_u(x, E u)}{V_u(x)} = \beta_2.$

Usually the last transition, where the expected value operator is introduced under the covariance sign, presents problems for students. This is the place where it is better to expand summation:

$latex E \frac{\textrm{cov}(x, u)}{V_u(x)} = E \frac{\frac{1}{n} \sum_{i=1}^n x_i u_i - \bigg( \frac{1}{n} \sum_{i=1}^n x_i \bigg) \bigg( \frac{1}{n} \sum_{i=1}^n u_i \bigg)}{V_u(x)}$

$latex = \frac{1}{V_u(x)} \left[ \frac{1}{n} \sum_{i=1}^n x_i E u_i - \left( \frac{1}{n} \sum_{i=1}^n x_i \right) \left( \frac{1}{n} \sum_{i=1}^n E u_i \right) \right] = 0.$

Everything that depends on $latex x$ is constant and can be pulled out of the expected value.

Study this proof carefully and remember about the distinction between sample means and population means.

Step 3. Proving unbiasedness of the intercept estimator. The vector form of the linear model implies

$latex \bar y = \beta_1 + \beta_2 \bar x + \bar u. \qquad (\textrm{Eq. 10.9})$

Many students think that $latex \bar u = 0$. This is not true (the left side is a random variable which may or may not be zero). Once again, you have to think about the difference between the sample mean and population mean.

We use the last two equations to prove unbiasedness of the intercept estimator:

$latex E b_1 = E(\beta_1 + \beta_2 \bar x + \bar u - b_2 \bar x) = \beta_1 + \beta_2 \bar x + E \bar u - (E b_2) \bar x = \beta_1$

(using $latex E \bar u = 0$ and $latex E b_2 = \beta_2$).

Previous: OLS: definition and derivation | Next: Variances of OLS estimators

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.