## Application: Ordinary Least Squares estimator

### Generalized Pythagoras theorem

**Exercise 1**. Let be a projector and denote Then

**Proof**. By the scalar product properties

is symmetric and idempotent, so

This proves the statement.

### Ordinary Least Squares (OLS) estimator derivation

**Problem statement**. A vector (the *dependent vector*) and vectors (*independent vectors* or *regressors*) are given. The **OLS estimator** is defined as that vector which minimizes the **total sum of squares**

Denoting we see that and that finding the OLS estimator means approximating with vectors from the image should be linearly independent, otherwise the solution will not be unique.

**Assumption**. are linearly independent. This, in particular, implies that

**Exercise 2**. Show that the OLS estimator is

(2)

**Proof**. By Exercise 1 we can use Since belongs to the image of doesn't change it: Denoting also we have

(by Exercise 1)

This shows that is a lower bound for This lower bound is achieved when the second term is made zero. From

we see that the second term is zero if satisfies (2).

Usually the above derivation is applied to the dependent vector of the form where is a random vector with mean zero. But it holds without this assumption. See also simplified derivation of the OLS estimator.

## Leave a Reply

You must be logged in to post a comment.