Raising the bar

Chapter 13. Bridge to econometrics

Unit 13.4. Linear dependence of vectors

The whole section 3.4 from the book is vague because the true explanation of the nature of multicollinearity requires matrix algebra. Below I go half-way: my explanation relies on the notion of linear dependence of vectors, which all econometricians are supposed to know, while deeper facts from matrix algebra are avoided (but can be found here).

With abstract notions, the thinking process works like this: a part of the argument can be written and seen, whereas the other part happens in the head. Only you can control what happens in your head. This section is a litmus paper to check if you can handle abstract notions.

Background. No matter what are vectors $latex X_1, \ldots, X_k$, we can always find numbers $latex c_1, \ldots, c_k$ such that

$latex c_1 X_1 + \ldots + c_k X_k = 0. \qquad (\textrm{Eq. 13.7})$

For instance, we can put

$latex c_1 = \ldots = c_k = 0. \qquad (\textrm{Eq. 13.8})$

Logically, there are two possibilities: (1) Eq. (13.7) is true only for the trivial set of constants (Eq. 13.8), and (2) there are other sets of constants such that Eq. (13.7) is true. The first case gives us linear independence and the second one gives linear dependence.

Definition 13.1 (Linearly dependent vectors). Vectors $latex X_1, \ldots, X_k$ are called linearly dependent if there exist numbers $latex c_1, \ldots, c_k$, not all of which are zero, such that the linear combination $latex c_1 X_1 + \ldots + c_k X_k$ is zero (more precisely, a null vector).

Comments. (1) The condition that not all of $latex c_1, \ldots, c_k$ are zero is equivalent to one inequality $latex c_1^2 + \ldots + c_k^2 \neq 0$.

(2) The vector whose coordinates are all zero is called a null vector. For example, in the two-dimensional case the null vector is $latex \bigl( \begin{smallmatrix} 0 \\ 0 \end{smallmatrix} \bigr)$. The null vector is denoted by the same $latex 0$, to avoid heavy notation. Two vectors are called equal if their respective coordinates are the same. For example, in two dimensions the equality $latex \bigl( \begin{smallmatrix} X_1 \\ X_2 \end{smallmatrix} \bigr) = \bigl( \begin{smallmatrix} 0 \\ 0 \end{smallmatrix} \bigr)$ means that $latex X_1 = 0$, $latex X_2 = 0$.

(3) Taking into account the above two comments, Definition 13.1 can be rewritten like this: Vectors $latex X_1, \ldots, X_k$ are called linearly dependent if there exist numbers $latex c_1, \ldots, c_k$ such that (a) $latex c_1^2 + \ldots + c_k^2 \neq 0$ and (b) Eq. (13.7) is true. If you forget condition (a), then you don't have a definition, because Eq. (13.7) is satisfied with trivial $latex c$'s.

(4) The order of words is important. If you say "vectors $latex \ldots$ are linearly dependent if there exist numbers $latex \ldots$", then the numbers depend on the vectors. If you start with "If there exist numbers $latex \ldots$" you get a completely different definition.

Definition 13.2 (Linearly dependent vectors, alternative form). Vectors $latex X_1, \ldots, X_k$ are called linearly dependent if one of them can be expressed as a linear combination of the others.

Exercise 13.1. Definitions 13.1 and 13.2 are equivalent.

Proof. Let $latex X_1, \ldots, X_k$ be linearly dependent in the sense of the first definition. We know that at least one of the numbers $latex c_1, \ldots, c_k$ is different from zero. Suppose it is $latex c_1$. Then the equation $latex c_1 X_1 + \ldots + c_k X_k = 0$ can be solved for $latex X_1$ giving

$latex X_1 = -\frac{c_2}{c_1} X_2 - \ldots - \frac{c_k}{c_1} X_k.$

We showed that one of them is a linear combination of the others.

Conversely, suppose that one of them, say, $latex X_1$, is a linear combination of the others: $latex X_1 = a_2 X_2 + \ldots + a_k X_k$. Then $latex X_1 - a_2 X_2 - \ldots - a_k X_k = 0$, and we can put $latex c_1 = 1$, $latex c_2 = -a_2$, $latex \ldots$, $latex c_k = -a_k$ to satisfy the first definition.

Definition 13.3 (Linearly independent vectors). If Eq. (13.7) holds only for trivial constants $latex c_i$, then $latex X_1, \ldots, X_k$ are called linearly independent.

An immediate consequence is that if vectors are linearly independent, then none of them can be expressed as a linear combination of the others.

Exercise 13.2.

$latex \quad$(a) Prove that on the plane the vectors

$latex X = \left( \begin{array}{c} x_1 \\ x_2 \end{array} \right), \quad e_1 = \left( \begin{array}{c} 1 \\ 0 \end{array} \right), \quad e_2 = \left( \begin{array}{c} 0 \\ 1 \end{array} \right)$

are linearly dependent.

$latex \quad$(b) Prove that $latex e_1, e_2$ are linearly independent.

Proof. (a) Using vector operations we have

$latex X = \left( \begin{array}{c} x_1 \\ x_2 \end{array} \right) = \left( \begin{array}{c} x_1 \\ 0 \end{array} \right) + \left( \begin{array}{c} 0 \\ x_2 \end{array} \right) = x_1 \left( \begin{array}{c} 1 \\ 0 \end{array} \right) + x_2 \left( \begin{array}{c} 0 \\ 1 \end{array} \right) = x_1 e_1 + x_2 e_2.$

The second definition of linear dependence is satisfied.

(b) Suppose we have a condition of type (Eq. 13.7): $latex c_1 e_1 + c_2 e_2 = 0$. Using vector operations we write this condition as

$latex \left( \begin{array}{c} c_1 \\ c_2 \end{array} \right) = 0.$

This, obviously, implies $latex c_1 = c_2 = 0$, so that $latex e_1, e_2$ are indeed independent.

This exercise illustrates a common situation: a set of vectors may be linearly independent but as you add to that set new vectors, the resulting set may become linearly dependent. It also leads to the definition of the dimension of a space.

Definition 13.4 (Space dimension). Suppose in a given space $latex L$ we find such vectors $latex e_1, \ldots, e_n$ that (1) $latex e_1, \ldots, e_n$ are linearly independent and (2) if we add any vector $latex X$ to that system, the system $latex e_1, \ldots, e_n, X$ is linearly dependent. Then $latex L$ is of dimension $latex n$.

Previous: Model with two regressors | Next: Criterion of linear dependence and the dummy variable trap

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.