Raising the bar

Chapter 14. Excel statistical functions

Unit 14.2. Excel functions for critical values

In the previous unit each function takes the argument $latex x$ (or $latex z$, $latex \lambda$, $latex \ldots$) and returns a probability (or a density value). In this unit the direction is reversed: we specify the probability of the right tail and ask Excel for the cut-off value that delivers it. The reference values that NCT puts into Tables 7$latex -$9 of its appendix are produced this way.

Cutoff points of the chi-square distribution function. For given probability of the right tail $latex \alpha$ and degrees of freedom $latex \nu$ the cut-off value $latex \chi_{\nu, \alpha}^2$ is a solution of the equation (see Figure 14.2)

$latex P(\chi_\nu^2 > \chi_{\nu, \alpha}^2) = \alpha.$

Figure 14.2: Cut-off point for chi-square distribution.

Table 14.7. Cutoff points of the chi-square distribution function.

In Mathematica, enter (on one line, without spaces)

v=9;a=0.995;InverseCDF[ChiSquareDistribution[v],1-a]

Cutoff points for the Student's $latex t$ distribution. For given probability of the right tail $latex \alpha$ and degrees of freedom $latex \nu$ the cut-off value $latex t_{\nu, \alpha}$ is a solution of the equation (see Figure 14.3)

$latex P(t_\nu > t_{\nu, \alpha}) = \alpha.$

Figure 14.3: Cut-off point for Student's t distribution.

Table 14.8. Cutoff points for the Student's $latex t$ distribution.

In Mathematica, enter (on one line, without spaces)

v=9;a=0.1;InverseCDF[StudentTDistribution[v],1-a]

Cutoff points for the $latex F$ distribution. For given probability of the right tail $latex \alpha$, degrees of freedom $latex v_1$ (numerator) and $latex v_2$ (denominator) the cut-off value $latex F_{v_1, v_2, \alpha}$ is a solution of the equation (see Figure 14.4)

$latex P(F_{v_1, v_2} > F_{v_1, v_2, \alpha}) = \alpha.$

Figure 14.4: Cut-off point for F distribution.

Table 14.9. Cutoff points for the $latex F$ distribution.

In Mathematica, enter (on one line, without spaces)

v1=9;v2=10;a=0.05;InverseCDF[FRatioDistribution[v1,v2],1-a]

Previous: Excel functions for distributions | Next: (end of book)

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 14. Excel statistical functions

Unit 14.1. Excel functions for distributions

Here I show how to replace Tables 1 through 9 from the appendix in NCT (that's about 30 pages) with one spreadsheet in Excel or one notebook file in Mathematica. The numbering of tables corresponds to that in NCT. In all cases in the first row of the Excel spreadsheet put names of variables, to be able to use the formulas repetitively. In the second row put numerical values of the variables and the formulas shown, with references to cells that contain numerical values. The values I put are for illustration only. In reality, the cell with the formula will show the function value. In case of Mathematica, type the formula shown, with the values of the arguments, all on one line, and then press Shift+Enter to see the result. Instead of a desktop version of Mathematica, you can use the online version wolframalpha.com for free.

Excel doesn't have functions corresponding to Tables 10$latex -$14. The Excel functions below are for Excel 2010 and 2013. Some functions supply the density value or the distribution function value, depending on the choice of the logical argument TRUE or FALSE. See Excel Help for more details.

Cumulative distribution function of the standard normal distribution. For a given real $latex z$, the value of the distribution function of the standard normal is (see Figure 14.1)

$latex F(z) = \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^z \exp(-t^2 / 2) dt.$

Table 14.1. Distribution function of the standard normal distribution.

In Mathematica, enter (on one line, without spaces)

z=0.5;CDF[NormalDistribution[0,1],z]

Figure 14.1: Cut-off point for standard normal distribution.

Probability function of the binomial distribution. For given number of successes $latex x$, number of trials $latex n$ and probability $latex p$ the probability is

$latex P(\textrm{Binomial} = x) = C_x^n p^x (1 - p)^{n - x}.$

Table 14.2. Probability function of the binomial distribution.

In Mathematica, enter (on one line, without spaces)

x=2;n=4;p=0.1;PDF[BinomialDistribution[n,p],x]

Cumulative binomial probabilities. For a given cut-off value $latex x$, number of trials $latex n$ and probability $latex p$ the cumulative probability is

$latex P(\textrm{Binomial} \leq x) = \sum_{t=0}^x C_t^n p^t (1 - p)^{n - t}.$

Table 14.3. Cumulative binomial probabilities.

In Mathematica, enter (on one line, without spaces)

x=2;n=4;p=0.1;CDF[BinomialDistribution[n,p],x]

Values of the exponential function $latex e^{-\lambda}$.

Table 14.4. Values of the exponential function $latex e^{-\lambda}$.

In Mathematica, enter (on one line, without spaces)

l=0.5;Exp[-l]

Individual Poisson probabilities. For given number of successes $latex x$ and arrival rate $latex \lambda$ the probability is

$latex P(\textrm{Poisson} = x) = \frac{e^{-\lambda} \lambda^x}{x!}.$

Table 14.5. Individual Poisson probabilities.

In Mathematica, enter (on one line, without spaces)

x=3;l=0.1;PDF[PoissonDistribution[l],x]

Cumulative Poisson probabilities. For given cut-off $latex x$ and arrival rate $latex \lambda$ the cumulative probability is

$latex P(\textrm{Poisson} \leq x) = \sum_{t=0}^x \frac{e^{-\lambda} \lambda^t}{t!}.$

Table 14.6. Cumulative Poisson probabilities.

In Mathematica, enter (on one line, without spaces)

x=3;l=0.1;CDF[PoissonDistribution[l],x]

Previous: Questions for repetition (Chapter 13) | Next: Excel functions for critical values

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 13. Bridge to econometrics

Unit 13.6. Questions for repetition

1. Using Example 13.2 (UoL exam 2012, zone B) as an illustration, explain why an LSE exam problem may be formally simple (only properties of variance and covariance plus simple regression are needed) and yet research-level in difficulty.

2. State the omitted-variable scenario: the true model is $latex y_i = \beta_0 + \beta_1 x_i + \beta_2 z_i + u_i$, but $latex z_i$ is unobserved and you fit $latex y_i = \beta_0 + \beta_1 x_i + v_i$. Show that the new error is $latex v_i = \beta_2 z_i + u_i$ and explain why $latex E v_i$ may be non-zero so that the slope estimator for the wrong model is biased.

3. Aggregation in economics means summation. Starting from the individual consumption function $latex c_{i, t} = \beta_0^i + \beta_1^i y_{i, t} + u_{i, t}$, derive the aggregate equation $latex C_t = \beta_0 + \beta_1 Y_t + u_t$ and identify $latex \beta_0$ and $latex u_t$. Explain why $latex \beta_1$ has to be "forced into" the equation.

4. Distinguish model misspecification from functional misspecification and give an example of each (use Table 13.1).

5. State the elimination method for solving the 2-by-2 linear system (Eq. 13.2). Define the determinant of a 2-by-2 matrix and derive the solution formulas $latex x_1 = \det B_1 / \det A$, $latex x_2 = \det B_2 / \det A$ (Eq. 13.6).

6. In the two-regressor model $latex y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + u$, apply linearity of covariance with respect to the first argument (and the formal condition $latex \textrm{cov}(u, x_j) = 0$) to derive the system of two equations in $latex \beta_1, \beta_2$, and write out the resulting expression for $latex \hat \beta_1$.

7. State both definitions of linearly dependent vectors and prove that they are equivalent. Why is the requirement "not all $latex c_i$ are zero" essential in the first definition?

8. Define linearly independent vectors. Prove that on the plane the vectors $latex X = \bigl( \begin{smallmatrix} x_1 \\ x_2 \end{smallmatrix} \bigr)$, $latex e_1 = \bigl( \begin{smallmatrix} 1 \\ 0 \end{smallmatrix} \bigr)$, $latex e_2 = \bigl( \begin{smallmatrix} 0 \\ 1 \end{smallmatrix} \bigr)$ are linearly dependent, but $latex e_1, e_2$ alone are linearly independent. Define the dimension of a space.

9. State Theorem 13.1 (Criterion of linear dependence) and explain why it is computationally useful. Write down the matrix formula for the OLS estimator $latex \hat \beta = (X^T X)^{-1} X^T y$ and the variance formula $latex \textrm{Var}(\hat \beta) = \sigma^2 (X^T X)^{-1}$, and use them to explain the consequences of (a) exact multicollinearity and (b) approximate multicollinearity for the OLS estimator.

10. Define the dummy variable trap. Illustrate it with the four-seasons example, and state the two ways to avoid it. What is the interpretation of the coefficients in each case?

Previous: Criterion of linear dependence and the dummy variable trap | Next: Excel functions for distributions

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 13. Bridge to econometrics

Unit 13.5. Criterion of linear dependence and the dummy variable trap

We conclude this chapter with two applications of the notion of linear dependence introduced in the previous unit. The first one is an algebraic way to check linear dependence. The second one explains the so-called dummy variables trap.

Criterion of linear dependence.

As explained in Box 3.1 in the book, the equations $latex y_i = \beta_1 + \beta_2 X_{2i} + \ldots + \beta_k X_{ki} + u_i$, $latex i = 1, \ldots, n$, can be written in the form

$latex y_i = \beta_1 X_{1i} + \beta_2 X_{2i} + \ldots + \beta_k X_{ki} + u_i, \quad i = 1, \ldots, n, \qquad (\textrm{Eq. 13.9})$

by introducing the trivial regressor $latex X_{1i} = 1$. Let us put the coordinates of the vectors $latex X_1, \ldots, X_k$ into one big matrix

$latex X = \left( \begin{array}{ccc} X_{11} & \ldots & X_{k1} \\ \ldots & \ldots & \ldots \\ X_{1n} & \ldots & X_{kn} \end{array} \right)$

(the first column contains the coordinates of $latex X_1$, $latex \ldots$, the $latex k$th column contains those of $latex X_k$). This matrix has as many rows as there are observations, and the number of columns equals the number of regressors, including the trivial regressor.

Definition 13.5 (Transposed matrix). The transposed matrix is obtained from $latex X$ by putting rows into columns and columns into rows:

$latex X^T = \left( \begin{array}{ccc} X_{11} & \ldots & X_{1n} \\ \ldots & \ldots & \ldots \\ X_{k1} & \ldots & X_{kn} \end{array} \right).$

You can think of transposition and determinant of a matrix as convenient formal ways to express facts of interest to us. We don't need to study their properties because software packages, including Excel, handle all matrix operations.

Theorem 13.1 (Criterion of linear dependence). Let $latex n > k$. Then $latex X_1, \ldots, X_k$ are linearly dependent if and only if $latex \det(X^T X) = 0$.

Comments. (1) The significance of this result is that, for a given set of vectors, searching for constants $latex c_1, \ldots, c_k$ that satisfy the linear dependence definition is usually difficult, while Theorem 13.1 does not appeal to any such constants. All you need is to know the matrix $latex X$, and the computer does for you the rest.

(2) Besides, it provides an exact condition for linear dependence. Therefore, by rejecting that condition we obtain the condition for linear independence: $latex X_1, \ldots, X_k$ are linearly independent if and only if $latex \det(X^T X) \neq 0$.

(3) The condition $latex \det(X^T X) \neq 0$ also happens to be a precise condition for existence of the inverse matrix $latex (X^T X)^{-1}$.

(4) In case of simple regression, we have to require $latex V(X) \neq 0$ to avoid division by zero in the slope estimator formula $latex \hat \beta_1 = \textrm{cov}(x, y) / V(x)$. The matrix formula of the OLS estimator for the multiple regression (Eq. 13.9) is

$latex \hat \beta = (X^T X)^{-1} X^T y. \qquad (\textrm{Eq. 13.10})$

You don't need to learn matrix algebra to understand that there is division by $latex X^T X$. Existence of the matrix $latex (X^T X)^{-1}$ is ensured by the condition $latex \det(X^T X) \neq 0$, which is an analog of the condition $latex V(X) \neq 0$ for simple regression. Summarizing, the following facts are equivalent (provided that $latex n > k$):

$latex \quad$(1) the regressors are linearly independent,

$latex \quad$(2) $latex \det(X^T X) \neq 0$,

$latex \quad$(3) the OLS estimator exists.

Definition 13.6 (Multicollinearity).

$latex \quad$(a) Precise (or exact) multicollinearity is when the regressors are linearly dependent or, equivalently, $latex \det(X^T X) = 0$. In this case the OLS estimator does not exist.

$latex \quad$(b) Multicollinearity is defined as the situation when the regressors are approximately linearly dependent. In this case $latex \det(X^T X)$ is not zero but is close to zero.

Perhaps, the variance-covariance formula for the OLS estimator (Eq. 13.10),

$latex \textrm{Var}(\hat \beta) = \sigma^2 (X^T X)^{-1}, \qquad (\textrm{Eq. 13.11})$

will help explain the problem with multicollinearity. Since $latex \det(X^T X)$ is not zero, the inverse matrix $latex (X^T X)^{-1}$ exists. However, that determinant is close to zero, and therefore the inverse $latex (X^T X)^{-1}$ has large elements. That's why the book says that some coefficient estimates have large variances.

The dummy variable trap.

Example 13.5. Suppose we want to see how power consumption varies with the season of the year. Let us denote $latex d_{win}$ the dummy variable for winter: it is 1 during winter and 0 during the rest of the year. Similarly, $latex d_{spr}, d_{sum}, d_{aut}$ are dummies for spring, summer and autumn, respectively. If our observations are seasonal and winter is considered the first season, the regressors look like this:

$latex d_{win} = \left( \begin{array}{c} 1 \\ 0 \\ 0 \\ 0 \\ \ldots \end{array} \right), \quad d_{spr} = \left( \begin{array}{c} 0 \\ 1 \\ 0 \\ 0 \\ \ldots \end{array} \right), \quad d_{sum} = \left( \begin{array}{c} 0 \\ 0 \\ 1 \\ 0 \\ \ldots \end{array} \right), \quad d_{aut} = \left( \begin{array}{c} 0 \\ 0 \\ 0 \\ 1 \\ \ldots \end{array} \right).$

Suppose we want to run the regression

$latex powercons = \beta_0 + \beta_1 d_{win} + \beta_2 d_{spr} + \beta_3 d_{sum} + \beta_4 d_{aut} + u.$

In addition to the four dummies, there is the trivial regressor

$latex X = \left( \begin{array}{c} 1 \\ 1 \\ 1 \\ 1 \\ \ldots \end{array} \right).$

It is easy to see that $latex X = d_{win} + d_{spr} + d_{sum} + d_{aut}$, that is, the regressors are linearly dependent, and our regression cannot be run.

In this example there are four categories (four seasons) and a full set of dummies, one for each category. In such a situation, the regressors, including the trivial one, are linearly dependent, so that perfect multicollinearity is present.

Definition 13.7 (Dummy variable trap). A dummy variable trap is the case of perfect multicollinearity resulting from inclusion in the model of the intercept and full set of dummies.

To avoid this trap, we can do one of two things: either (1) throw out the trivial regressor or (2) remove one of the dummies. The interpretation of the coefficients depends on what regressor you omit. In the first case the coefficients represent ups and downs from some abstract zero level. In the second case, suppose you remove $latex d_{win}$. Then the trivial regressor is responsible for winter, which is the so-called base category, and the coefficients of other seasonal dummies represent deviations from the winter level.

Previous: Linear dependence of vectors | Next: Questions for repetition (Chapter 13)

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 13. Bridge to econometrics

Unit 13.4. Linear dependence of vectors

The whole section 3.4 from the book is vague because the true explanation of the nature of multicollinearity requires matrix algebra. Below I go half-way: my explanation relies on the notion of linear dependence of vectors, which all econometricians are supposed to know, while deeper facts from matrix algebra are avoided (but can be found here).

With abstract notions, the thinking process works like this: a part of the argument can be written and seen, whereas the other part happens in the head. Only you can control what happens in your head. This section is a litmus paper to check if you can handle abstract notions.

Background. No matter what are vectors $latex X_1, \ldots, X_k$, we can always find numbers $latex c_1, \ldots, c_k$ such that

$latex c_1 X_1 + \ldots + c_k X_k = 0. \qquad (\textrm{Eq. 13.7})$

For instance, we can put

$latex c_1 = \ldots = c_k = 0. \qquad (\textrm{Eq. 13.8})$

Logically, there are two possibilities: (1) Eq. (13.7) is true only for the trivial set of constants (Eq. 13.8), and (2) there are other sets of constants such that Eq. (13.7) is true. The first case gives us linear independence and the second one gives linear dependence.

Definition 13.1 (Linearly dependent vectors). Vectors $latex X_1, \ldots, X_k$ are called linearly dependent if there exist numbers $latex c_1, \ldots, c_k$, not all of which are zero, such that the linear combination $latex c_1 X_1 + \ldots + c_k X_k$ is zero (more precisely, a null vector).

Comments. (1) The condition that not all of $latex c_1, \ldots, c_k$ are zero is equivalent to one inequality $latex c_1^2 + \ldots + c_k^2 \neq 0$.

(2) The vector whose coordinates are all zero is called a null vector. For example, in the two-dimensional case the null vector is $latex \bigl( \begin{smallmatrix} 0 \\ 0 \end{smallmatrix} \bigr)$. The null vector is denoted by the same $latex 0$, to avoid heavy notation. Two vectors are called equal if their respective coordinates are the same. For example, in two dimensions the equality $latex \bigl( \begin{smallmatrix} X_1 \\ X_2 \end{smallmatrix} \bigr) = \bigl( \begin{smallmatrix} 0 \\ 0 \end{smallmatrix} \bigr)$ means that $latex X_1 = 0$, $latex X_2 = 0$.

(3) Taking into account the above two comments, Definition 13.1 can be rewritten like this: Vectors $latex X_1, \ldots, X_k$ are called linearly dependent if there exist numbers $latex c_1, \ldots, c_k$ such that (a) $latex c_1^2 + \ldots + c_k^2 \neq 0$ and (b) Eq. (13.7) is true. If you forget condition (a), then you don't have a definition, because Eq. (13.7) is satisfied with trivial $latex c$'s.

(4) The order of words is important. If you say "vectors $latex \ldots$ are linearly dependent if there exist numbers $latex \ldots$", then the numbers depend on the vectors. If you start with "If there exist numbers $latex \ldots$" you get a completely different definition.

Definition 13.2 (Linearly dependent vectors, alternative form). Vectors $latex X_1, \ldots, X_k$ are called linearly dependent if one of them can be expressed as a linear combination of the others.

Exercise 13.1. Definitions 13.1 and 13.2 are equivalent.

Proof. Let $latex X_1, \ldots, X_k$ be linearly dependent in the sense of the first definition. We know that at least one of the numbers $latex c_1, \ldots, c_k$ is different from zero. Suppose it is $latex c_1$. Then the equation $latex c_1 X_1 + \ldots + c_k X_k = 0$ can be solved for $latex X_1$ giving

$latex X_1 = -\frac{c_2}{c_1} X_2 - \ldots - \frac{c_k}{c_1} X_k.$

We showed that one of them is a linear combination of the others.

Conversely, suppose that one of them, say, $latex X_1$, is a linear combination of the others: $latex X_1 = a_2 X_2 + \ldots + a_k X_k$. Then $latex X_1 - a_2 X_2 - \ldots - a_k X_k = 0$, and we can put $latex c_1 = 1$, $latex c_2 = -a_2$, $latex \ldots$, $latex c_k = -a_k$ to satisfy the first definition.

Definition 13.3 (Linearly independent vectors). If Eq. (13.7) holds only for trivial constants $latex c_i$, then $latex X_1, \ldots, X_k$ are called linearly independent.

An immediate consequence is that if vectors are linearly independent, then none of them can be expressed as a linear combination of the others.

Exercise 13.2.

$latex \quad$(a) Prove that on the plane the vectors

$latex X = \left( \begin{array}{c} x_1 \\ x_2 \end{array} \right), \quad e_1 = \left( \begin{array}{c} 1 \\ 0 \end{array} \right), \quad e_2 = \left( \begin{array}{c} 0 \\ 1 \end{array} \right)$

are linearly dependent.

$latex \quad$(b) Prove that $latex e_1, e_2$ are linearly independent.

Proof. (a) Using vector operations we have

$latex X = \left( \begin{array}{c} x_1 \\ x_2 \end{array} \right) = \left( \begin{array}{c} x_1 \\ 0 \end{array} \right) + \left( \begin{array}{c} 0 \\ x_2 \end{array} \right) = x_1 \left( \begin{array}{c} 1 \\ 0 \end{array} \right) + x_2 \left( \begin{array}{c} 0 \\ 1 \end{array} \right) = x_1 e_1 + x_2 e_2.$

The second definition of linear dependence is satisfied.

(b) Suppose we have a condition of type (Eq. 13.7): $latex c_1 e_1 + c_2 e_2 = 0$. Using vector operations we write this condition as

$latex \left( \begin{array}{c} c_1 \\ c_2 \end{array} \right) = 0.$

This, obviously, implies $latex c_1 = c_2 = 0$, so that $latex e_1, e_2$ are indeed independent.

This exercise illustrates a common situation: a set of vectors may be linearly independent but as you add to that set new vectors, the resulting set may become linearly dependent. It also leads to the definition of the dimension of a space.

Definition 13.4 (Space dimension). Suppose in a given space $latex L$ we find such vectors $latex e_1, \ldots, e_n$ that (1) $latex e_1, \ldots, e_n$ are linearly independent and (2) if we add any vector $latex X$ to that system, the system $latex e_1, \ldots, e_n, X$ is linearly dependent. Then $latex L$ is of dimension $latex n$.

Previous: Model with two regressors | Next: Criterion of linear dependence and the dummy variable trap

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 13. Bridge to econometrics

Unit 13.3. Model with two regressors

The purpose is to derive equation (3.11) from the book. This will be done in two steps.

Step 1. A linear system with two unknowns. We start with formulas for solutions of a system of linear equations with two unknowns. Consider the system

$latex \left\{ \begin{array}{l} a_{11} x_1 + a_{12} x_2 = b_1, \\ a_{21} x_1 + a_{22} x_2 = b_2. \end{array} \right. \qquad (\textrm{Eq. 13.2})$

Here $latex x_1, x_2$ are unknown and everything else is assumed known. We use the elimination method: the given equations are rearranged in such a way that it is easy to eliminate one of the unknowns and solve the resulting equation for the other. Multiplying the first equation by $latex a_{22}$ and the second by $latex a_{12}$ we get

$latex \begin{array}{rcl} a_{11} a_{22} x_1 + a_{12} a_{22} x_2 & = & b_1 a_{22}, \\ a_{21} a_{12} x_1 + a_{22} a_{12} x_2 & = & b_2 a_{12}. \end{array}$

The coefficients of $latex x_2$ here are the same. If we subtract the second equation from the first, $latex x_2$ will disappear:

$latex (a_{11} a_{22} - a_{21} a_{12}) x_1 = b_1 a_{22} - b_2 a_{12}. \qquad (\textrm{Eq. 13.3})$

We want to rewrite this equation using a couple of definitions from matrix algebra.

Definition (Matrix and determinant). By definition, a matrix is a table of numbers:

$latex A = \left( \begin{array}{cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array} \right).$

Its determinant is defined as

$latex \det A = a_{11} a_{22} - a_{21} a_{12}.$

One can see that the expression on the right-hand side of Eq. (13.3) is the determinant of the matrix

$latex B_1 = \left( \begin{array}{cc} b_1 & a_{12} \\ b_2 & a_{22} \end{array} \right).$

With the notation introduced Eq. (13.3) rewrites as

$latex x_1 \det A = \det B_1. \qquad (\textrm{Eq. 13.4})$

Denote

$latex B_2 = \left( \begin{array}{cc} a_{11} & b_1 \\ a_{21} & b_2 \end{array} \right).$

You can check that similar arguments lead to

$latex x_2 \det A = \det B_2. \qquad (\textrm{Eq. 13.5})$

As we see, if $latex \det A \neq 0$, then equations (Eq. 13.4) and (Eq. 13.5) have solutions

$latex x_1 = \frac{\det B_1}{\det A}, \quad x_2 = \frac{\det B_2}{\det A}. \qquad (\textrm{Eq. 13.6})$

Step 2. Application to a regression with two regressors. Consider a regression (written in vector form) with two regressors:

$latex y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + u.$

Using this model and linearity of covariance with respect to the first argument we have

$latex \begin{array}{rcl} \textrm{cov}(y, x_1) & = & \textrm{cov}(\beta_0 + \beta_1 x_1 + \beta_2 x_2 + u, x_1) \\ & = & \textrm{cov}(\beta_0, x_1) + \beta_1 \textrm{cov}(x_1, x_1) + \beta_2 \textrm{cov}(x_2, x_1) + \textrm{cov}(u, x_1). \end{array}$

Here $latex \textrm{cov}(\beta_0, x_1) = 0$ (covariance with a constant is zero), $latex \textrm{cov}(x_1, x_1) = V(x_1)$. We formally put $latex \textrm{cov}(u, x_1) = 0$ (this is not a rigorous derivation). The result is

$latex \textrm{cov}(y, x_1) = \beta_1 V(x_1) + \beta_2 \textrm{cov}(x_2, x_1).$

Similarly,

$latex \textrm{cov}(y, x_2) = \beta_1 \textrm{cov}(x_1, x_2) + \beta_2 V(x_2).$

We see that the coefficients $latex \beta_1, \beta_2$ satisfy a system of type (Eq. 13.2) with

$latex A = \left( \begin{array}{cc} V(x_1) & \textrm{cov}(x_2, x_1) \\ \textrm{cov}(x_1, x_2) & V(x_2) \end{array} \right), \quad b_1 = \textrm{cov}(y, x_1), \quad b_2 = \textrm{cov}(y, x_2).$

Therefore application of the first equation in (Eq. 13.6) gives

$latex \hat \beta_1 = \frac{\textrm{cov}(y, x_1) V(x_2) - \textrm{cov}(y, x_2) \textrm{cov}(x_2, x_1)}{V(x_1) V(x_2) - [\textrm{cov}(x_1, x_2)]^2}.$

We formally replaced $latex \beta_1$ by $latex \hat \beta_1$. This is equation (3.11) from the book.

Previous: How to read Dougherty's book | Next: Linear dependence of vectors

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 13. Bridge to econometrics

Unit 13.2. How to read Dougherty's book

1. For each property use a name, like additivity, homogeneity etc. Divide all large proofs into smaller parts that you can understand. Initially it will take a lot of time but will pay itself later. Remember that understanding the first 6 or 7 chapters with proofs is critically important, because in the subsequent chapters most proofs are skipped and you will have to use analogy with what you know from the first several chapters.

2. Equation (R.54) and many others: do yourself a favor, use the notation from Unit 10.1 wherever summation is involved, to avoid long, indigestible formulas.

3. Equation (R.55): use a two-step definition of a normal variable (first define the standard normal and then a general normal as a linear transformation of the standard normal), as it is done in Unit 6.6.

4. Prepare to conquer Dougherty's book in advance by reading my book. Reading the review chapter (that's almost 100 pages) can be avoided if you have mastered my book when taking second year Stats. You should become friends with algebra by doing two-page proofs.

5. The relationship between $latex t$ statistic and $latex F$ statistic on p. 147 has a simpler proof, see Comment (2) in Unit 9.6. It is a general probabilistic fact not related to any model.

6. Question 14 from Unit 5.18 is related to the logit model from Chapter 10, see equation (10.6) in Dougherty.

Translate all verbal statements to math.

Example 13.4. Reasons to include the disturbance term (Section 1.1 of the book).

Reason 1. It often happens that there are variables that you would like to include in the regression equation but cannot because you are unable to measure them.

Translation. The true model is $latex y_i = \beta_0 + \beta_1 x_i + \beta_2 z_i + u_i$. Omitting $latex z_i$ we get the wrong model $latex y_i = \beta_0 + \beta_1 x_i + v_i$. Looking at it from the point of view of the true model we see that the error becomes worse, $latex v_i = \beta_2 z_i + u_i$. In particular, $latex E v_i = \beta_2 E z_i$ may not be zero, in which case the slope estimator for the wrong model is biased.

Reason 2. In many cases, the relationship is an attempt to summarize in aggregate a number of microeconomic relationships. For example, the aggregate consumption function is an attempt to summarize a set of individual expenditure decisions.

Translation. Aggregation in economics means summation. An individual consumption function models dependence of individual consumption $latex c_{i, t}$ on individual income $latex y_{i, t}$ through

$latex c_{i, t} = \beta_0^i + \beta_1^i y_{i, t} + u_{i, t}. \qquad (\textrm{Eq. 13.1})$

Here the subscript $latex i$ stands for "individual" and $latex t$ stands for "time". ☞ The coefficients $latex \beta_0^i, \beta_1^i$ depend on the individual. In practice, we don't have data on individual consumption and incomes. We have their aggregates $latex C_t = \sum_i c_{i, t}$ and $latex Y_t = \sum_i y_{i, t}$. The equation for aggregate figures obtains from Eq. (13.1) by summing:

$latex C_t = \sum_i c_{i, t} = \sum_i \beta_0^i + \sum_i \beta_1^i y_{i, t} + \sum_i u_{i, t}.$

The intercept $latex \beta_0 = \sum_i \beta_0^i$ for the aggregate equation easily obtains from individual intercepts but the slope $latex \beta_1$ does not. We have to force it into the equation:

$latex \begin{array}{rcl} C_t & = & \beta_0 + \beta_1 \sum_i y_{i, t} + \sum_i (\beta_1^i - \beta_1) y_{i, t} + \sum_i u_{i, t} \\ & = & \beta_0 + \beta_1 Y_t + u_t. \end{array}$

Here the error $latex u_t = \sum_i (\beta_1^i - \beta_1) y_{i, t} + \sum_i u_{i, t}$ includes the errors from microeconomic relationships plus the additional disturbance that arises from aggregation.

Reason 3. The model may be misspecified in terms of its structure.

Reason 4. The functional relationship between $latex Y$ and $latex X$ may be misspecified mathematically.

Translation. Reasons 3 and 4 are better understood if put side by side, which we do in Table 13.1.

Table 13.1. Two types of misspecification.

Reason 5 (measurement error model). If the measurement of one or more of the variables in the relationship is subject to error, the observed values will not appear to conform to an exact relationship, and the discrepancy contributes to the disturbance term.

Translation. Suppose that the true model is $latex y_i = \beta_0 + \beta_1 x_i + u_i$ and that $latex x_i$ is measured with error. That is, $latex x_i$ is not observed and instead one has $latex x_i^*$ such that $latex x_i = x_i^* + \varepsilon_i$ where $latex \varepsilon_i$ is the measurement error. The true dependence becomes $latex y_i = \beta_0 + \beta_1 x_i^* + (\beta_1 \varepsilon_i + u_i)$. One can regress $latex y_i$ on the observed $latex x_i^*$ but this increases the error.

This discussion shows that one has to read between the lines to understand Dougherty.

Previous: Problems with LSE econometrics | Next: Model with two regressors

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 13. Bridge to econometrics

Unit 13.1. Problems with LSE econometrics

In the exposition below we refer to Dougherty, C. Introduction to Econometrics. Oxford: Oxford University Press, 4th ed. 2011.

Problem with coverage. Dougherty's book is packed with information (it is 530 pages without appendices). There are many small and large models, math is mixed with economic considerations, and literally any piece of that information may appear on the exam.

Example 13.1. In the UoL exam 2012, zone B there is a question: briefly explain what the error correction model is.

The answer from Dougherty is one paragraph of text on pp. 405$latex -$406. By the time the students study that paragraph, they are fed up with different models. They will not remember that paragraph because it is a small detail in a rich landscape. Professionals encounter many such details in their work, easily understand them and don't try to remember. I would reduce the number of small details and concentrate on a deep study of a small number of most important theories, as it is done in my Chapter 10.

Problem with level.

Example 13.2 (UoL exam 2012, zone B). The model is $latex y_t = \beta x_t + u_t$, $latex E u_t = 0$, $latex E u_t^2 = \sigma^2$, $latex t = 1, \ldots, n$. An econometrician suggests an estimator for $latex \beta$ given by

$latex \tilde \beta = \frac{1}{n} \sum_{t=1}^n y_t / x_t.$

Prove that $latex \tilde \beta$ is an unbiased estimator for $latex \beta$ and that its variance is

$latex V(\tilde \beta) = \frac{\sigma^2}{n^2} \left( \sum_{t=1}^n 1/x_t^2 \right)$

under certain assumptions. What extra assumptions have you used?

Formally, to answer this question, all you need is properties of variances and covariances and a notion of simple regression. Informally, this is a truly research problem. You need to understand the proofs in Boxes 2.2 and 2.3 of the book (this is two pages of dense math; among other things, there is double summation which many students don't understand). Moreover, you have to extract the ideas and apply them to the exam problem. Your understanding of the ideas should be deep enough as to be able to impose appropriate conditions on the errors. If you think it's hard to come up with more difficult exam problems, you are wrong. Look at the next example.

Example 13.3 (UoL exam 2013, zone A). Consider the model $latex y_t = \alpha x_t + u_t$, $latex t = 1, \ldots, T$, where $latex E u_t = 0$, $latex E u_t^2 = \sigma^2 x_t^2$, $latex E u_s u_t = 0$ if $latex s \neq t$, for all $latex s$ and $latex t$. $latex x_t$ is an observed non-random variable. The density function of $latex u_t$ is

$latex f(u_t) = \left( 2 \pi \sigma^2 x_t^2 \right)^{-1/2} \exp \left[ -\frac{1}{2} \left( \frac{u_t}{\sigma x_t} \right)^2 \right].$

$latex \quad$(1) Derive the maximum likelihood (ML) estimators of $latex \alpha$ and $latex \sigma^2$. Show that $latex \hat \alpha$, the ML estimator of $latex \alpha$, is unbiased. (8 marks)

$latex \quad$(2) Compare the ML estimator of $latex \alpha$ with the weighted least squares estimator of $latex \alpha$. (7 marks)

The model in this example is the same as in Example 13.2. However, in Example 13.2 the estimator is given, while in this one it has to be derived. This already represents a major hazard for some students. But they also have to repeat the derivations required for Example 13.2 (unbiasedness and variance) and compare the resulting properties of the OLS estimator with those of the ML estimator (which is to be derived too)!

I love research problems but giving them on the exam is too much. If the LSE wants to use its exams as a screening device for future Nobel prize winners, that's fine. But then the LSE Stats courses should provide an adequate background. It is not adequate, because in affiliate centers the first year Stats is based on AP Stats textbooks, and thus is a waste of time, and the second year Stats does not prepare the students to handle two-page proofs.

Previous: Questions for repetition (Chapter 12) | Next: How to read Dougherty's book

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 12. Analysis of variance (ANOVA)

Unit 12.4. Two-way ANOVA: decompositions and F-tests

Leading idea. From Eq. (12.6) it follows that $latex \bar x_{g \cdot}$ estimates $latex \mu + \mu_g$ (because $latex E \bar x_{g \cdot} = \mu + \mu_g$), $latex \bar x_{\cdot b}$ estimates $latex \mu + \mu_b$ (because $latex E \bar x_{\cdot b} = \mu + \mu_b$), and $latex \bar x$ estimates $latex \mu$ (because $latex E \bar x = \mu$). Therefore in the representation

$latex x_{gb} - \bar x = (\bar x_{g \cdot} - \bar x) + (\bar x_{\cdot b} - \bar x) + (x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x) \qquad (\textrm{Eq. 12.9})$

the term $latex (\bar x_{g \cdot} - \bar x)$ estimates $latex \mu_g$, the term $latex (\bar x_{\cdot b} - \bar x)$ estimates $latex \mu_b$ and $latex (x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)$ is a sample analog of $latex \varepsilon_{gb}$.

Definition 12.2 (Sums of squares for two-way ANOVA).

Total sum of squares: $latex SST = \sum_{g, b} (x_{gb} - \bar x)^2$.

Group sum of squares: $latex SSG = \sum_{g, b} (\bar x_{g \cdot} - \bar x)^2 = B \sum_g (\bar x_{g \cdot} - \bar x)^2$.

Block sum of squares: $latex SSB = \sum_{g, b} (\bar x_{\cdot b} - \bar x)^2 = G \sum_b (\bar x_{\cdot b} - \bar x)^2$.

Sum of squared errors: $latex SSE = \sum_{g, b} (x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)^2$.

Exercise 12.7. Prove that

$latex SST = SSG + SSB + SSE. \qquad (\textrm{Eq. 12.10})$

Solution. Squaring identity (Eq. 12.9) gives

$latex SST = \sum_{g, b} (x_{gb} - \bar x)^2$

$latex = \sum_{g, b} (\bar x_{g \cdot} - \bar x)^2 + \sum_{g, b} (\bar x_{\cdot b} - \bar x)^2 + \sum_{g, b} (x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)^2 + S_1 + S_2 + S_3.$

Here $latex S_1, S_2, S_3$ denote cross-products:

$latex S_1 = 2 \sum_{g, b} (\bar x_{g \cdot} - \bar x)(\bar x_{\cdot b} - \bar x) = 2 \sum_g (\bar x_{g \cdot} - \bar x) \sum_b (\bar x_{\cdot b} - \bar x)$

$latex = 2 (G \bar x - G \bar x)(B \bar x - B \bar x) = 0,$

$latex S_2 = 2 \sum_{g, b} (\bar x_{g \cdot} - \bar x)(x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)$

$latex = 2 \sum_g (\bar x_{g \cdot} - \bar x) \sum_b (x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)$

$latex = 2 \sum_g (\bar x_{g \cdot} - \bar x)(B \bar x_{g \cdot} - B \bar x_{g \cdot} - B \bar x + B \bar x) = 0,$

$latex S_3 = 2 \sum_{g, b} (\bar x_{\cdot b} - \bar x)(x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)$

$latex = 2 \sum_b (\bar x_{\cdot b} - \bar x) \sum_g (x_{gb} - \bar x_{g \cdot} - \bar x_{\cdot b} + \bar x)$

$latex = 2 \sum_b (\bar x_{\cdot b} - \bar x)(G \bar x_{\cdot b} - G \bar x - G \bar x_{\cdot b} + G \bar x) = 0.$

It remains to apply the definitions of $latex SSG, SSB, SSE$ to finish the proof.

Three types of null hypotheses.

1. All observations come from the same population: $latex H_0: \mu_g = \mu_b = 0$ for all $latex g, b$.

2. There are no differences between groups: $latex H_0^{\textrm{groups}}: \mu_g = 0$ for all $latex g$.

3. There are no differences between blocks: $latex H_0^{\textrm{blocks}}: \mu_b = 0$ for all $latex b$.

Exercise 12.8.

$latex \quad$(1) Find the distribution of $latex SST$ under the null $latex H_0$.

$latex \quad$(2) Find the distribution of $latex SSG$ under $latex H_0^{\textrm{groups}}$.

$latex \quad$(3) Find the distribution of $latex SSB$ under $latex H_0^{\textrm{blocks}}$.

$latex \quad$(4) Find the distribution of $latex SSE$ under the null $latex H_0$.

Solution. (1) From Eq. (12.4) we see that under the null hypothesis $latex H_0$ the variables $latex x_{gb}$ are i.i.d. normal with mean $latex \mu$ and variance $latex \sigma^2$. By Theorem 12.1

$latex SST / \sigma^2 = \chi_{GB - 1}^2. \qquad (\textrm{Eq. 12.11})$

(2) If $latex H_0^{\textrm{groups}}$ holds, then from Eq. (12.6) we see that $latex \bar x_{g \cdot}$ have mean $latex \mu$, and by Eq. (12.7) their variances are $latex \sigma^2 / B$. Since they are i.i.d. normal, Theorem 12.1 allows us to conclude that $latex \sum_g (\bar x_{g \cdot} - \bar x)^2 / (\sigma^2 / B) = \chi_{G - 1}^2$. This equation can be equivalently written as

$latex SSG / \sigma^2 = \chi_{G - 1}^2. \qquad (\textrm{Eq. 12.12})$

(3) If $latex H_0^{\textrm{blocks}}$ holds, then from Eq. (12.6) we see that $latex \bar x_{\cdot b}$ have mean $latex \mu$, and by Eq. (12.8) their variances are $latex \sigma^2 / G$. As above, since $latex \bar x_{\cdot b}$ are i.i.d. normal, by Theorem 12.1 $latex \sum_b (\bar x_{\cdot b} - \bar x)^2 / (\sigma^2 / G) = \chi_{B - 1}^2$. This equation can be equivalently written as

$latex SSB / \sigma^2 = \chi_{B - 1}^2. \qquad (\textrm{Eq. 12.13})$

(4) If $latex H_0$ holds, then $latex H_0^{\textrm{groups}}$ and $latex H_0^{\textrm{blocks}}$ also hold and equations (Eq. 12.11)$latex -$(Eq. 12.13) are true. Assuming that $latex SSG, SSB$ and $latex SSE$ are independent (we lack the tools to prove this), by the subtraction property of chi square distributions we see that

$latex \frac{SSE}{\sigma^2} = \frac{SST - SSG - SSB}{\sigma^2} = \chi_{GB - 1}^2 - \chi_{G - 1}^2 - \chi_{B - 1}^2 = \chi_{(G - 1)(B - 1)}^2.$

Now it is a pure formality to rewrite the relationships obtained using the familiar format. Let $latex MSG = SSG / (G - 1)$, $latex MSB = SSB / (B - 1)$, $latex MSE = SSE / [(G - 1)(B - 1)]$. Then

$latex \frac{(G - 1) MSG}{\sigma^2} = \chi_{G - 1}^2, \quad \frac{(B - 1) MSB}{\sigma^2} = \chi_{B - 1}^2,$

$latex \frac{(G - 1)(B - 1) MSE}{\sigma^2} = \chi_{(G - 1)(B - 1)}^2.$

We summarize the results in Table 12.4.

Table 12.4. Two-way ANOVA.

Decision rule. (a) Reject the hypothesis that there is no difference between groups if at the $latex \alpha$ significance level $latex MSG/MSE > F_{G - 1, (B - 1)(G - 1), \alpha}$ where the critical value $latex F_{G - 1, (B - 1)(G - 1), \alpha}$ is defined by $latex P(F_{G - 1, (B - 1)(G - 1)} > F_{G - 1, (B - 1)(G - 1), \alpha}) = \alpha$.

(b) Reject the hypothesis that there is no difference between blocks if at the $latex \alpha$ significance level $latex MSB/MSE > F_{B - 1, (B - 1)(G - 1), \alpha}$ where the critical value $latex F_{B - 1, (B - 1)(G - 1), \alpha}$ is defined by $latex P(F_{B - 1, (B - 1)(G - 1)} > F_{B - 1, (B - 1)(G - 1), \alpha}) = \alpha$.

Previous: Two-way ANOVA: setup and population model | Next: Questions for repetition (Chapter 12)

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 12. Analysis of variance (ANOVA)

Unit 12.3. Two-way ANOVA: setup and population model

Difference between one-way and two-way ANOVA.

One-way ANOVA: three car makes are tested for differences in fuel consumption. The differences may be influenced by driving habits of drivers selected to test the cars. To remove this difference, out of a pool of, say, 15 drivers three teams of 5 drivers are randomly selected to test the three car types. Thus, the variability in fuel consumption is caused only by one factor $latex -$ car make.

Two-way ANOVA attempts to reveal dependence of variability on two factors: car make and driver habits. The second variable is called a blocking factor. Unlike the one-way ANOVA example above, all drivers drive all cars. If there are many drivers, they can be joined into groups (called blocks), say, by age.

Table 12.3. Visualization of data for two-way ANOVA. Observation $latex x_{gb}$ is recorded for group $latex g$ (e.g., car make) in block $latex b$ (e.g., driver group). Every group is observed in every block.

Table 12.3. Visualization of data for two-way ANOVA.

Exercise 12.5.

$latex \quad$(1) Write out the formulas for the sample means by group (summation over $latex b$) and by block (summation over $latex g$).

$latex \quad$(2) What are the expressions for the total number of observations, for the overall sample mean, and how is the overall sample mean related to the group and block means?

Solution. (1) We use a dot to indicate a variable that has been averaged out. For example, in the notation of the group sample mean the dot is put in place of $latex b$ to indicate that the block variable has been used for averaging:

$latex \bar x_{g \cdot} = \frac{1}{B} \sum_{b=1}^B x_{gb}, \quad g = 1, \ldots, G.$

Similarly, the block sample mean is defined by

$latex \bar x_{\cdot b} = \frac{1}{G} \sum_{g=1}^G x_{gb}, \quad b = 1, \ldots, B.$

(2) The total number of observations is $latex n = GB$. The overall sample mean

$latex \bar x = \frac{1}{n} \sum_{g, b} x_{gb}$

is related to the block and group means by

$latex \bar x = \frac{1}{B} \sum_{b=1}^B \bar x_{\cdot b} = \frac{1}{G} \sum_{g=1}^G \bar x_{g \cdot}.$

Population model. Assume that the observations follow a linear model

$latex x_{gb} = \mu + \mu_g + \mu_b + \varepsilon_{gb} \qquad (\textrm{Eq. 12.4})$

where the errors $latex \varepsilon_{gb}$ are normal, independent, have mean zero and the same variance $latex \sigma^2$. $latex \mu$ is an overall population mean. The parameters $latex \mu_g, \mu_b$ give group- and block-specific corrections and satisfy

$latex \sum_b \mu_b = \sum_g \mu_g = 0. \qquad (\textrm{Eq. 12.5})$

This condition tells us that $latex \mu_b$ and $latex \mu_g$ may have alternating signs. Depending on their signs, $latex \mu + \mu_g + \mu_b$ may be higher or lower than $latex \mu$.

Exercise 12.6. Derive the population models for $latex \bar x_{g \cdot}, \bar x_{\cdot b}, \bar x$.

Solution. Just average equations (Eq. 12.4) and use equations (Eq. 12.5) to get

$latex \bar x_{g \cdot} = \mu + \mu_g + \bar \varepsilon_{g \cdot}, \quad \bar x_{\cdot b} = \mu + \mu_b + \bar \varepsilon_{\cdot b}, \quad \bar x = \mu + \bar \varepsilon. \qquad (\textrm{Eq. 12.6})$

Since the errors are i.i.d., we have by homogeneity and additivity of variance

$latex V(\bar x_{g \cdot}) = V(\bar \varepsilon_{g \cdot}) = \frac{1}{B^2} V\left( \sum_{b=1}^B \varepsilon_{gb} \right) = \frac{1}{B^2} \sum_{b=1}^B V(\varepsilon_{gb}) = \frac{\sigma^2}{B}, \qquad (\textrm{Eq. 12.7})$

$latex V(\bar x_{\cdot b}) = V(\bar \varepsilon_{\cdot b}) = \frac{1}{G^2} V\left( \sum_{g=1}^G \varepsilon_{gb} \right) = \frac{1}{G^2} \sum_{g=1}^G V(\varepsilon_{gb}) = \frac{\sigma^2}{G}. \qquad (\textrm{Eq. 12.8})$

We shall use also the fact that $latex \bar \varepsilon_{g \cdot}, g = 1, \ldots, G$, are independent and $latex \bar \varepsilon_{\cdot b}, b = 1, \ldots, B$, are also independent.

Previous: One-way ANOVA: distributions and the F-test | Next: Two-way ANOVA: decompositions and F-tests

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.