Raising the bar

Proving unbiasedness of OLS estimators - the do's and don'ts

Groundwork

Here we derived the OLS estimators. To distinguish between sample and population means, the variance and covariance in the slope estimator will be provided with the subscript u (for "uniform", see the rationale here).

(1) $latex \hat{b}=\frac{Cov_u(x,y)}{Var_u(x)}$,

(2) $latex \hat{a}=\bar{y}-\hat{b}\bar{x}$.

These equations are used in conjunction with the model

(3) $latex y_i=a+bx_i+e_i$

where we remember that

(4) $latex Ee_i=0$ for all $latex i$.

Since (2) depends on (1), we have to start with unbiasedness of the slope estimator.

Using the right representation is critical

We have to show that $latex E\hat{b}=b$.

Step 1. Don't apply the expectation directly to (1). Do separate in (1) what is supposed to be $latex E\hat{b}$. To reveal the role of errors in (1), plug (3) in (1) and use linearity of covariance with respect to each argument when the other argument is fixed:

$latex \hat{b}=\frac{Cov_u(x,a+bx+e)}{Var_u(x)}=\frac{Cov_u(x,a)+bCov_u(x,x)+Cov_u(x,e)}{Var_u(x)}$.

Here $latex Cov_u(x,a)=0$ (a constant is uncorrelated with any variable), $latex Cov_u(x,x)=Var_u(x)$ (covariance of $latex x$ with itself is its variance), so

(5) $latex \hat{b}=\frac{bVar_u(x)+Cov_u(x,e)}{Var_u(x)}=b+\frac{Cov_u(x,e)}{Var_u(x)}$.

Equation (5) is the mean-plus-deviation-from-the-mean decomposition. Many students think that $latex Cov_u(x,e)=0$ because of (4). No! The covariance here does not involve the population mean.

Step 2. It pays to make one more step to develop (5). Write out the numerator in (5) using summation:

$latex \hat{b}=b+\frac{1}{n}\sum(x_i-\bar{x})(e_i-\bar{e})/Var_u(x).$

Don't write out $latex Var_u(x)$! Presence of two summations confuses many students.

Multiplying parentheses and using the fact that $latex \sum(x_i-\bar{x})=n\bar{x}-n\bar{x}=0$ we have

$latex \hat{b}=b+\frac{1}{n}[\sum(x_i-\bar{x})e_i-\bar{e}\sum(x_i-\bar{x})]/Var_u(x)$

$latex =b+\frac{1}{n}\sum\frac{(x_i-\bar{x})}{Var_u(x)}e_i.$

To simplify calculations, denote $latex a_i=(x_i-\bar{x})/Var_u(x).$ Then the slope estimator becomes

(6) $latex \hat{b}=b+\frac{1}{n}\sum a_ie_i.$

This is the critical representation.

Unbiasedness of the slope estimator

Convenience condition. The regressor $latex x$ is deterministic. I call it a convenience condition because it's just a matter of mathematical expedience, and later on we'll study ways to bypass it.

From (6), linearity of means and remembering that the deterministic coefficients $latex a_i$ behave like constants,

(7) $latex E\hat{b}=E[b+\frac{1}{n}\sum a_ie_i]=b+\frac{1}{n}\sum a_iEe_i=b$

by (4). This proves unbiasedness.

You don't know the difference between the population and sample means until you see them working in the same formula.

Unbiasedness of the intercept estimator

As above we plug (3) in (2): $latex \hat{a}=\overline{a+bx+e}-\hat{b}\bar{x}=a+b\bar{x}+\bar{e}-\hat{b}\bar{x}$. Applying expectation:

$latex E\hat{a}=a+b\bar{x}+E\bar{e}-E\hat{b}\bar{x}=a+b\bar{x}-b\bar{x}=a.$

Conclusion

Since in (1)  there is division by $latex Var_u(x)$, the condition $latex Var_u(x)\ne 0$ is the main condition for existence of OLS estimators. From the above proof we see that (4) is the main condition for unbiasedness.