Raising the bar Constructing a projector onto a given subspace: use a basis

Nov 18

Constructing a projector onto a given subspace

Constructing a projector onto a given subspace

Let $L$ be a subspace of $R^n.$ Let $k=\dim L\ (\leq n)$ and fix some basis $x^{(1)},...,x^{(k)}$ in $L.$ Define the matrix $X=(x^{(1)},...,x^{(k)})$ of size $n\times k$ (the vectors are written as column vectors).

Exercise 1. a) With the above notation, the matrix $(X^TX)^{-1}$ exists. b) The matrix $P=X(X^TX)^{-1}X^T$ exists. c) $P$ is a projector.

Proof. a) The determinant of $A=X^TX$ is not zero by linear independence of the basis vectors, so its inverse $A^{-1}$ exists. We also know that $A$ and its inverse are symmetric:

(1) $A^T=A,$ $(A^{-1})^T=A^{-1}.$

b) To see that $P$ exists just count the dimensions.

c) Let's prove that $P$ is a projector. (1) allows us to make the proof compact. $P$ is idempotent:

$P^2=(XA^{-1}X^T)(XA^{-1}X^T)=XA^{-1}(X^TX)A^{-1}X^T$

$=X(A^{-1}A)A^{-1}X^T=XA^{-1}X^T=P.$

$P$ is symmetric:

$P^T=[XA^{-1}X^T]^T=(X^T)^T(A^{-1})^TX^T=XA^{-1}X^T=P.$

Exercise 2. $P$ projects onto $L$ : $\text{Img}(P)=L.$

Proof. First we show that $\text{Img}(P)\subseteq L.$ Put

(2) $y=A^{-1}X^Tx,$

for any $x\in R^n.$ Then

(3) $Px=XA^{-1}X^Tx=Xy=\sum x^{(j)}y_j\in L.$

This shows that $\text{Img}(P)\subseteq L.$

Let's prove the opposite inclusion. Any element of $L$ is of form

(4) $x=Xz$

with some $z.$ Plugging (4) in (3) we have $Px=XA^{-1}X^TXz=Xz$ which shows that $\text{Img}(P)\supseteq L.$

Exercise 3. Let $L_1$ be a subspace of $R^n$ and let $L_2=(L_1)^\perp$ . Then $R^n=L_1\oplus L_2$ .

Proof. Since any element of $L_1$ is orthogonal to any element of $L_2$ , we have only to show that any $x\in R^n$ can be represented as $x=l_1+l_2$ with $l_j\in L_j$ . Let $P$ be the projector from Exercise 1, where $L=L_1$ . Put $l_1=Px$ , $l_2=(I-P)x$ . $l_1\in L_1=\text{Img}(P)$ is obvious. For any $y\in R^n$ , $(Py)\cdot l_2=y\cdot (P(I-P)x)=0$ , so $l_2$ is orthogonal to $L_1$ . By definition of $L_2$ , we have $l_2\in L_2$ .