Raising the bar

Linear dependence of vectors: definition and principal result

This is a topic most students have trouble with. It's because there is a lot of logic involved, so skimming it is not a good idea. We start with the most common definition.

Definition 1. Let $latex x^{(1)},...,x^{(k)}\in R^n$ be some vectors. They are called linearly dependent if there exist numbers $latex a_1,...,a_k,$ not all of which are zero, such that

(1) $latex a_1x^{(1)}+...+a_kx^{(k)}=0.$

The sheer length of this definition scares some people and all they remember is equation (1). Stating just (1) instead of the whole definition is like skinning an animal.

Seizing the bull by the horns

The first matter of business is to shorten the definition and relate it to what we already know.

It's better to join the set of numbers $latex a_1,...,a_k$ into a vector $latex a=(a_1,...,a_k).$ Then the requirement that "not all of $latex a_1,...,a_k$ are zero" is equivalent to a single equation $latex a\neq 0.$ Further, let us write the vectors $latex x^{(1)},...,x^{(k)}$ as columns and put them into a matrix $latex X=\left( x^{(1)},...,x^{(k)}\right) .$ Using multiplication of partitioned matrices we see that (1) is equivalent to

(2) $latex Xa=0$

and therefore Definition 1 is equivalent to the following.

Definition 2. Vectors $latex x^{(1)},...,x^{(k)}\in R^n$ are called linearly dependent if the homogeneous equation (2) has nonzero solutions $latex a\in R^k,$ that is, the null space of $latex X$ is not $latex \{0\}.$

Negating Definition 2 gives the definition of linear independence. Negating Definition 2 is easier than negating Definition 1.

Definition 3. Vectors $latex x^{(1)},...,x^{(k)}\in R^{n}$ are called linearly independent if $latex N(X)=\{0\}$ (for any nonzero $latex a\in R^k$ we have $latex Xa\neq 0$ or, alternatively, $latex Xa=0$ only for $latex a=0$).

Exercise 1. For any matrix $latex X,$ one has

(3) $latex N(X)=N(X^TX).$

Proof. 1) Proving that $latex N(X)\subset N(X^TX).$ If $latex a\in N(X),$ then $latex Xa=0$ and $latex X^TXa=0,$ so $latex a\in N(X^TX).$ 2) Proving that $latex N(X^TX)\subset N(X).$ If $latex a\in N(X^TX),$ then $latex X^TXa=0$ and $latex 0=(X^TXa)\cdot a=(Xa)\cdot(Xa)=\|Xa\|^2,$ so $latex a\in N(X).$

Exercise 2. $latex X^TX$ is a square symmetric matrix, for any $latex X.$

Proof. If $latex X$ is of size $latex n\times k,$ then $latex X^TX$ is $latex k\times k.$ It is symmetric: $latex (X^TX)^T=X^T(X^T)^T=X^TX.$ By the way, some students write $latex (X^TX)^{-1}=X^{-1}(X^T)^{-1}.$ You cannot do this if $latex X$ is not square.

Criterion of linear independence. Vectors $latex x^{(1)},...,x^{(k)}\in R^n$ are linearly independent if and only if $latex \det X^TX\neq 0.$

Proof. We are going to use (3). By Exercise 2, $latex A=X^TX$ is a square matrix and for a square matrix we have the equivalence $latex N(A)=\{0\}\Longleftrightarrow \det A\neq 0.$ Application of this result proves the statement.

Direct application of Definition 1 can be problematic. To prove that some vectors are linearly dependent, you have to produce $latex a_1,...,a_k$ such that Definition 1 is satisfied. This usually involves some guesswork, see exercises below. The criterion above doesn't require guessing and can be realized on the computer. The linear independence requirement is common in multiple regression analysis but not all econometricians know this criterion.

Putting some flesh on the bones

These are simple facts you need to know in addition to the above criterion.

Exercise 3. 1) Why do we exclude the case when all $latex a_1,...,a_k$ are zero?

2) What happens if among $latex x^{(1)},...,x^{(k)}$ there are zero vectors?

3) Show that in case of two non-zero vectors Definition 1 is equivalent to just proportionality of one vector to another.

Solution. 1) If all $latex a_1,...,a_k$ are zero, (1) is trivially satisfied, no matter what the vectors are.

2) If one of the vectors $latex x^{(1)},...,x^{(k)}$ is zero, the coefficient of that vector can be set to one and all others to zero, so such vectors will be linearly dependent by Definition 1.

3) Consider two non-zero vectors $latex x^{(1)},x^{(2)}.$ If they are linearly dependent, then

(4) $latex a_1x^{(1)}+a_2x^{(2)}=0$

where at least one of $latex a_1,a_2$ is not zero. Suppose $latex a_1\neq 0.$ Then $latex a_2\neq 0$ because otherwise $latex x^{(1)}$ would be zero. Hence

(5) $latex x^{(1)}=-a_2/a_1x^{(2)}=cx^{(2)}$

where the proportionality coefficient $latex c$ is not zero. Conversely, (5) implies (4) (you have to produce the coefficients).

Exercise 4. Prove that if to the system of unit vectors we add any vector $latex x\in R^n$, the resulting system will be linearly dependent.

← Previous: Solvability of an equation with a square matrix | Next: Basis and dimension →

Get the PDF book “Matrix Algebra” on Lulu