What is an OLS estimator - simplified derivation

Feb 16

What is an OLS estimator - simplified derivation

category: AP Stats and Business Stats, EC2020, EC2020 Elements of econometrics, Econometrics, ST104A, ST104B, Statistics 1, Statistics 2 4 Comments

What is an OLS estimator?

Most sources give too much theory and a long procedure of OLS (Ordinary Least Squares) derivation. In fact, there is a simplified derivation which can be used in other situations. For example, in case of an Instrumental Variables estimator it is the ONLY derivation. However, simplicity comes at a cost. You need to know the properties of variances and covariances.

Tip. By all means avoid using summation signs because: 1) not everybody is good at them and 2) they clog the picture. For this reason alone it is worth using variances and covariances. For example, if $X_1,...,X_n$ are observations, let us call by variance the quantity $Var(X)=\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})^2$ where $\bar{X}=\frac{1}{n}\sum_{i=1}^n X_i$ is the sample mean. If, in addition, we have observations $Y_1,...,Y_n$ on another variable, then we can introduce covariance by $Cov(X,Y)=\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})(Y_i-\bar{Y}).$ One can notice that

(1) $Var(X)=Cov(X,X).$

Try to rewrite this using summation signs, and you will see the gain from using shorter notation. The next property is less easy to establish. It is called linearity of covariance: if $Z$ is a third observed variable and $a,b$ are any numbers, then

(2) $Cov(X,aY+bZ)=aCov(X,Y)+bCov(X,Z).$

Now consider simple regression

(3) $Y=\beta_0+\beta_1 X+u$

where $u$ is the error term and $\beta_0,\beta_1$ are parameters to be estimated. The well-known formula for the OLS estimator of $\beta_1$ is in our notation

(4) $\hat{\beta}_1=\frac{Cov(X,Y)}{Var(X)}.$

The question is: What is an easy way to obtain this?

Answer: Notice that there is $Cov(X,Y)$ in the numerator and try to expand it using (3) and (2):

$Cov(X,Y)=Cov(X,\beta_0+\beta_1 X+u)=Cov(X,\beta_0)+\beta_1Cov(X,X)+Cov(X,u).$

The first term on the right is zero (this is shown directly), for the second term we apply (1). In case of the third term, we formally put $Cov(X,u)=0$ (remember, this is a simplified derivation and we are cheating). The result is $Cov(X,Y)=\beta_1Var(X).$ It remains to solve this for $\beta_1$ , putting a hat on it, to obtain (4). The whole derivation takes just one paragraph!