Raising the bar

Constructing a projector onto a given subspace

Let $latex L$ be a subspace of $latex R^n.$ Let $latex k=\dim L\ (\leq n)$ and fix some basis $latex x^{(1)},...,x^{(k)}$ in $latex L.$ Define the matrix $latex X=(x^{(1)},...,x^{(k)})$ of size $latex n\times k$ (the vectors are written as column vectors).

Exercise 1. a) With the above notation, the matrix $latex (X^TX)^{-1}$ exists. b) The matrix $latex P=X(X^TX)^{-1}X^T$ exists. c) $latex P$ is a projector.

Proof. a) The determinant of $latex A=X^TX$ is not zero by linear independence of the basis vectors, so its inverse $latex A^{-1}$ exists. We also know that $latex A$ and its inverse are symmetric:

(1) $latex A^T=A,$ $latex (A^{-1})^T=A^{-1}.$

b) To see that $latex P$ exists just count the dimensions.

c) Let's prove that $latex P$ is a projector. (1) allows us to make the proof compact. $latex P$ is idempotent:

$latex P^2=(XA^{-1}X^T)(XA^{-1}X^T)=XA^{-1}(X^TX)A^{-1}X^T$

$latex =X(A^{-1}A)A^{-1}X^T=XA^{-1}X^T=P.$

$latex P$ is symmetric:

$latex P^T=[XA^{-1}X^T]^T=(X^T)^T(A^{-1})^TX^T=XA^{-1}X^T=P.$

Exercise 2. $latex P$ projects onto $latex L$:  $latex \text{Img}(P)=L.$

Proof. First we show that $latex \text{Img}(P)\subseteq L.$ Put

(2) $latex y=A^{-1}X^Tx,$

for any $latex x\in R^n.$ Then

(3) $latex Px=XA^{-1}X^Tx=Xy=\sum x^{(j)}y_j\in L.$

This shows that $latex \text{Img}(P)\subseteq L.$

Let's prove the opposite inclusion. Any element of $latex L$ is of form

(4) $latex x=Xz$

with some $latex z.$ Plugging (4) in (3) we have $latex Px=XA^{-1}X^TXz=Xz$ which shows that $latex \text{Img}(P)\supseteq L.$

Exercise 3. Let $latex L_1$ be a subspace of $latex R^n$ and let $latex L_2=(L_1)^\perp$. Then $latex R^n=L_1\oplus L_2$.

Proof. Since any element of $latex L_1$ is orthogonal to any element of $latex L_2$, we have only to show that any $latex x\in R^n$ can be represented as $latex x=l_1+l_2$ with $latex l_j\in L_j$. Let $latex P$ be the projector from Exercise 1, where $latex L=L_1$. Put $latex l_1=Px$, $latex l_2=(I-P)x$. $latex l_1\in L_1=\text{Img}(P)$ is obvious. For any $latex y\in R^n$, $latex (Py)\cdot l_2=y\cdot (P(I-P)x)=0$, so $latex l_2$ is orthogonal to $latex L_1$. By definition of $latex L_2$, we have $latex l_2\in L_2$.

← Previous: Geometry and algebra of projectors | Next: Application: Ordinary Least Squares estimator →

Get the PDF book “Matrix Algebra” on Lulu