Raising the bar

Application: Ordinary Least Squares estimator

Generalized Pythagoras theorem

Exercise 1. Let $latex P$ be a projector and denote $latex Q=I-P.$ Then $latex \Vert x\Vert^2=\Vert Px\Vert^2+\Vert Qx\Vert^2.$

Proof. By the scalar product properties

$latex \Vert x\Vert^2=\Vert Px+Qx\Vert^2=\Vert Px\Vert^2+2(Px)\cdot (Qx)+\Vert Qx\Vert^2.$

$latex P$ is symmetric and idempotent, so

$latex (Px)\cdot (Qx)=(Px)\cdot[(I-P)x]=x\cdot[(P-P^2)x]=0.$

This proves the statement.

Ordinary Least Squares (OLS) estimator derivation

Problem statement. A vector $latex y\in R^n$ (the dependent vector) and vectors $latex x^{(1)},...,x^{(k)}\in R^n$ (independent vectors or regressors) are given. The OLS estimator is defined as that vector $latex \beta \in R^k$ which minimizes the total sum of squares $latex TSS=\sum_{i=1}^n(y_i-x^{(1)}\beta_1-...-x^{(k)}\beta_k)^2.$

Denoting $latex X=(x^{(1)},...,x^{(k)}),$ we see that $latex TSS=\Vert y-X\beta\Vert^2$ and that finding the OLS estimator means approximating $latex y$ with vectors from the image $latex \text{Img}X.$ $latex x^{(1)},...,x^{(k)}$ should be linearly independent, otherwise the solution will not be unique.

Assumption. $latex x^{(1)},...,x^{(k)}$ are linearly independent. This, in particular, implies that $latex k\leq n.$

Exercise 2. Show that the OLS estimator is

(2) $latex \hat{\beta}=(X^TX)^{-1}X^Ty.$

Proof. By Exercise 1 we can use $latex P=X(X^TX)^{-1}X^T.$ Since $latex X\beta$ belongs to the image of $latex P,$ $latex P$ doesn't change it: $latex X\beta=PX\beta.$ Denoting also $latex Q=I-P$ we have

$latex \Vert y-X\beta\Vert^2=\Vert y-Py+Py-X\beta\Vert^2$

$latex =\Vert Qy+P(y-X\beta)\Vert^2$ (by Exercise 1)

$latex =\Vert Qy\Vert^2+\Vert P(y-X\beta)\Vert^2.$

This shows that $latex \Vert Qy\Vert^2$ is a lower bound for $latex \Vert y-X\beta\Vert^2.$ This lower bound is achieved when the second term is made zero. From

$latex P(y-X\beta)=Py-X\beta =X(X^TX)^{-1}X^Ty-X\beta=X[(X^TX)^{-1}X^Ty-\beta]$

we see that the second term is zero if $latex \beta$ satisfies (2).

Usually the above derivation is applied to the dependent vector of the form $latex y=X\beta+e$ where $latex e$ is a random vector with mean zero. But it holds without this assumption. See also simplified derivation of the OLS estimator.

← Previous: Constructing a projector onto a given subspace | Next: Eigenvalues and eigenvectors of a projector →

Get the PDF book “Matrix Algebra” on Lulu