Application: Ordinary Least Squares estimator
Generalized Pythagoras theorem
Exercise 1. Let be a projector and denote
Then
Proof. By the scalar product properties
is symmetric and idempotent, so
This proves the statement.
Ordinary Least Squares (OLS) estimator derivation
Problem statement. A vector (the dependent vector) and vectors
(independent vectors or regressors) are given. The OLS estimator is defined as that vector
which minimizes the total sum of squares
Denoting we see that
and that finding the OLS estimator means approximating
with vectors from the image
should be linearly independent, otherwise the solution will not be unique.
Assumption. are linearly independent. This, in particular, implies that
Exercise 2. Show that the OLS estimator is
(2)
Proof. By Exercise 1 we can use Since
belongs to the image of
doesn't change it:
Denoting also
we have
(by Exercise 1)
This shows that is a lower bound for
This lower bound is achieved when the second term is made zero. From
we see that the second term is zero if satisfies (2).
Usually the above derivation is applied to the dependent vector of the form where
is a random vector with mean zero. But it holds without this assumption. See also simplified derivation of the OLS estimator.
Leave a Reply
You must be logged in to post a comment.