13
Feb 16

What is an OLS estimator - simplified derivation

category: AP Stats and Business Stats, EC2020, EC2020 Elements of econometrics, Econometrics, ST104A, ST104B, Statistics 1, Statistics 2 4 Comments

What is an OLS estimator?

Most sources give too much theory and a long procedure of OLS (Ordinary Least Squares) derivation. In fact, there is a simplified derivation which can be used in other situations. For example, in case of an Instrumental Variables estimator it is the ONLY derivation. However, simplicity comes at a cost. You need to know the properties of variances and covariances.

Tip. By all means avoid using summation signs because: 1) not everybody is good at them and 2) they clog the picture. For this reason alone it is worth using variances and covariances. For example, if X_1,...,X_n are observations, let us call by variance the quantity Var(X)=\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})^2 where \bar{X}=\frac{1}{n}\sum_{i=1}^n X_i is the sample mean. If, in addition, we have observations Y_1,...,Y_n on another variable, then we can introduce covariance by Cov(X,Y)=\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})(Y_i-\bar{Y}). One can notice that

(1) Var(X)=Cov(X,X).

Try to rewrite this using summation signs, and you will see the gain from using shorter notation. The next property is less easy to establish. It is called linearity of covariance: if Z is a third observed variable and a,b are any numbers, then

(2) Cov(X,aY+bZ)=aCov(X,Y)+bCov(X,Z).

Now consider simple regression

(3) Y=\beta_0+\beta_1 X+u

where u is the error term and \beta_0,\beta_1 are parameters to be estimated. The well-known formula for the OLS estimator of \beta_1 is in our notation

(4) \hat{\beta}_1=\frac{Cov(X,Y)}{Var(X)}.

The question is: What is an easy way to obtain this?

Answer: Notice that there is Cov(X,Y) in the numerator and try to expand it using (3) and (2):

Cov(X,Y)=Cov(X,\beta_0+\beta_1 X+u)=Cov(X,\beta_0)+\beta_1Cov(X,X)+Cov(X,u).

The first term on the right is zero (this is shown directly), for the second term we apply (1). In case of the third term, we formally put Cov(X,u)=0 (remember, this is a simplified derivation and we are cheating). The result is Cov(X,Y)=\beta_1Var(X). It remains to solve this for \beta_1, putting a hat on it, to obtain (4). The whole derivation takes just one paragraph!

Post Views: 46

Leave a Reply

You must be logged in to post a comment.