Geometry of linear equations: orthogonal complement and equation solvability

Jul 18

Geometry of linear equations: orthogonal complement and equation solvability

Geometry of linear equations: orthogonal complement and equation solvability

From orthogonality of vectors to orthogonality of subspaces

Definition 1. Let $L$ be a linear subspace of $R^{n}.$ Its orthogonal complement $L^{\perp }$ is defined as the set of all vectors orthogonal to all elements of $L:$ $L^{\perp }=\{ x\in R^n:x\cdot y=0$ $\text{ for all}\ y\in L\}.$

Exercise 1. Let $L=\{ (x_1,0):x_1\in R\}$ be the $x$ axis on the plane. Find $L^{\perp}.$

Solution. The condition $x\cdot y=x_1y_1=0$ for all $x=(x_1,x_2)\in L$ implies $y_1=0.$ The component $y_2$ is free. Thus, $L^{\perp}=\{ (0,y_2):y_2\in R\}$ is the $y$ axis.

Similarly, the orthogonal complement $(L^{\perp})^{\perp}$ of the $y$ axis is the $x$ axis. This fact is generalized as follows:

Theorem (on second orthocomplement) For any linear subspace $L,$ one has $(L^{\perp})^{\perp}=L$ .

This statement is geometrically simple but the proof is rather complex and will be omitted (see Akhiezer & Glazman, Theory of Linear Operators in Hilbert Space, Dover Publications, 1993, Chapter 1, sections 6 and 7). It's good to keep in mind what makes it possible. For any set $A\subset R^n,$ its orthogonal complement $A^{\perp}$ can be defined in the same way: $A^{\perp}=\{ x\in R^n:x\cdot y=0$ $\text{ for all }y\in A\}.$ As in Exercise 1, you can show that if $A=\{(1,0)\}$ contains just the unit vector of the $x$ axis, then $A^{\perp}$ is the $y$ axis and therefore $(A^{\perp})^{\perp}$ is the $x$ axis. Thus, in this example we have a strict inclusion $A\subset (A^{\perp})^{\perp}.$ The equality is achieved only for linear subspaces.

Exercise 2. In $R^3,$ consider the $x$ axis $L=\{(x_1,0,0):x_1\in R\}.$ Can you tell what $L^{\perp}$ is?

Exercise 3. $\{ 0\}^{\perp}=R^n$ and $(R^{n})^{\perp}=\{ 0\} .$

Link between image of a matrix and null space of its transpose

Exercise 4. The rule for a transpose of a product $(AB)^T=B^TA^T$ implies that for any $x\in R^k,$ $y\in R^n$

(1) $(Ax)\cdot y=(Ax)^Ty=x^TA^Ty=x\cdot (A^Ty).$

Exercise 5 (second characterization of matrix image) $\text{Img}(A)=N(A^T)^{\perp}$ (and, consequently, $\text{Img}(A^T)=N(A)^{\perp}$ ).

Proof. Let us prove that

(2) $\text{Img}(A)^{\perp}=N(A^T)$ .

To this end, we first prove the inclusion $\text{Img}(A)^{\perp}\subset N(A^T).$ Let $y\in \text{Img}(A)^{\perp}$ . For an arbitrary $x,$ we have $Ax\in \text{Img}(A).$ Hence by (1) $0=(Ax)\cdot y=x\cdot(A^Ty).$ Since $x$ is arbitrary, we can put $x=A^Ty$ and obtain $\|A^Ty\|^2=0.$ This implies $A^Ty=0$ and $y\in N(A^T).$

Conversely, to prove $N(A^T)\subset\text{Img}(A)^{\perp},$ suppose $y\in N(A^T).$ Then $A^Ty=0$ and for an arbitrary $x$ we have $0=x\cdot(A^Ty)=(Ax)\cdot y.$ Since $Ax$ runs over $\text{Img}(A),$ this means that $y\in\text{Img}(A)^{\perp}.$

Passing to orthogonal complements in (2), by the theorem on second orthocomplement we get what we need.

Definition 2. Let $L_{1},\ L_{2}$ be two subspaces. We say that $L=L_{1}\oplus L_{2}$ is their orthogonal sum if 1) every element $l\in L$ can be decomposed as $l=l_{1}+l_{2}$ with $l_{i}\in L_{i},$ $i=1,2,$ and 2) every element of $L_{1}$ is orthogonal to every element of $L_{2}.$ Orthogonality of $L_{1}$ to $L_{2}$ implies $L_{1}\cap L_{2}=\{0\}$ which, in turn, guarantees uniqueness of the representation $l=l_{1}+l_{2}.$

Exercise 6. Assume that $A$ is of size $n\times k$ . Exercise 5 and Exercise 3 imply $R^n=\text{Img}(A)\oplus N(A^T)$ and $R^k=\text{Img}(A^T)\oplus N(A)$ .

The importance of Exercise 5 is explained by the fact that the null space of a matrix is easier to describe analytically than the image. For the next summary, you might want to review the conclusion on the role of the null space.

Summary. 1) In order to see if $Ax=y$ has solutions, check if $y$ is orthogonal to $N(A^T).$ In particular, if $A^T$ is one-to-one, then $N(A^T)=\{0\}$ and $Ax=y$ has solutions for all $y$ (see Exercise 3 above).