Raising the bar

What is an OLS estimator?

Most sources give too much theory and a long procedure of OLS (Ordinary Least Squares) derivation. In fact, there is a simplified derivation which can be used in other situations. For example, in case of an Instrumental Variables estimator it is the ONLY derivation. However, simplicity comes at a cost. You need to know the properties of variances and covariances.

Tip. By all means avoid using summation signs because: 1) not everybody is good at them and 2) they clog the picture. For this reason alone it is worth using variances and covariances. For example, if $latex X_1,...,X_n$ are observations, let us call by variance the quantity $latex Var(X)=\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})^2$ where $latex \bar{X}=\frac{1}{n}\sum_{i=1}^n X_i$ is the sample mean. If, in addition, we have observations $latex Y_1,...,Y_n$ on another variable, then we can introduce covariance by $latex Cov(X,Y)=\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})(Y_i-\bar{Y}).$ One can notice that

(1) $latex Var(X)=Cov(X,X).$

Try to rewrite this using summation signs, and you will see the gain from using shorter notation. The next property is less easy to establish. It is called linearity of covariance: if $latex Z$ is a third observed variable and $latex a,b$ are any numbers, then

(2) $latex Cov(X,aY+bZ)=aCov(X,Y)+bCov(X,Z).$

Now consider simple regression

(3) $latex Y=\beta_0+\beta_1 X+u$

where $latex u$ is the error term and $latex \beta_0,\beta_1$ are parameters to be estimated. The well-known formula for the OLS estimator of $latex \beta_1$ is in our notation

(4) $latex \hat{\beta}_1=\frac{Cov(X,Y)}{Var(X)}.$

The question is: What is an easy way to obtain this?

Answer: Notice that there is $latex Cov(X,Y)$ in the numerator and try to expand it using (3) and (2):

$latex Cov(X,Y)=Cov(X,\beta_0+\beta_1 X+u)=Cov(X,\beta_0)+\beta_1Cov(X,X)+Cov(X,u).$

The first term on the right is zero (this is shown directly), for the second term we apply (1). In case of the third term, we formally put $latex Cov(X,u)=0$ (remember, this is a simplified derivation and we are cheating). The result is $latex Cov(X,Y)=\beta_1Var(X).$ It remains to solve this for $latex \beta_1$, putting a hat on it, to obtain (4). The whole derivation takes just one paragraph!