Raising the bar

Simple regression: a useful comparison of what we have before and after estimation

Before anything

Initially, we have only observations

(1) $latex (x_1,y_1),...,(x_n,y_n)$.

Before estimation

Then we assume dependence between  y's and x's of the form

(2)  $latex y_i=\beta_0+\beta_1x_i+u_i,\ i=1,...,n.$

Here $latex \beta_0,\beta_1$ are unknown parameters to be estimated and $latex u_i$ are random errors which satisfy the basic assumption

(3) $latex Eu_i=0.$

It is convenient to call $latex \beta_0+\beta_1x_i$ a linear part of model (2).

After estimation

The OLS estimators of $latex \beta_1,\beta_0$ are, respectively,

$latex \hat{\beta}_1=\frac{Cov(x,y)}{Var(x)},\ \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}.$

Using these estimators, we define the fitted value $latex \hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$ which mimics the linear part. To mimic the errors, we define residuals $latex e_i=y_i-\hat{y}_i$. These definitions give a sample analog of (2):

(2') $latex y_i=\hat{\beta}_0+\hat{\beta}_1x_i+e_i.$

The residuals also possess the property

(3') $latex \bar{e}=0$

which is a sample analog of (3).

Comparison

Tricky question. Put $latex e=\big(e_1,...,e_n)$. Ask your students to show that if $latex x_i$ are deterministic, then under condition (3) along with (3') one has

(3'') $latex Ee=0.$

This will reveal if they know the difference between sample means and population means.