Application: Ordinary Least Squares estimator
Generalized Pythagoras theorem
Exercise 1. Let be a projector and denote Then
Proof. By the scalar product properties
is symmetric and idempotent, so
This proves the statement.
Ordinary Least Squares (OLS) estimator derivation
Problem statement. A vector (the dependent vector) and vectors (independent vectors or regressors) are given. The OLS estimator is defined as that vector which minimizes the total sum of squares
Denoting we see that and that finding the OLS estimator means approximating with vectors from the image should be linearly independent, otherwise the solution will not be unique.
Assumption. are linearly independent. This, in particular, implies that
Exercise 2. Show that the OLS estimator is
Proof. By Exercise 1 we can use Since belongs to the image of doesn't change it: Denoting also we have
(by Exercise 1)
This shows that is a lower bound for This lower bound is achieved when the second term is made zero. From
we see that the second term is zero if satisfies (2).
Usually the above derivation is applied to the dependent vector of the form where is a random vector with mean zero. But it holds without this assumption. See also simplified derivation of the OLS estimator.