Raising the bar

Chapter 10. Simple regression

Unit 10.4. Orthogonality and its consequences

Definition 10.4 (Fitted value and residuals). After having estimated the coefficients, for prediction purposes we put $latex \hat y = b_1 + b_2 x$ (can you prove that $latex E \hat y = \beta_1 + \beta_2 x$?). This real-valued linear function is called a fitted value. It is defined everywhere, both inside and outside the sample. It is customary to use the same notation for the vector $latex \hat y = (b_1 + b_2 x_1, \ldots, b_1 + b_2 x_n)$ where the fitted value is calculated only at the sample points. Hopefully you will see from the context if the fitted value is a real-valued function or a vector. The residual vector is defined as $latex e = y - \hat y$ and residuals are its components $latex e_i = y_i - \hat y_i$.

This definition immediately implies

$latex y = \hat y + e = b_1 + b_2 x + e \qquad (\textrm{Eq. 10.17})$

which is the sample analog of the linear model.

Definition 10.5 (Orthogonality of vectors). Vectors $latex (x_1, \ldots, x_n), (y_1, \ldots, y_n)$ are called orthogonal if $latex x_1 y_1 + \ldots + x_n y_n = 0$.

Exercise 10.3.

(1) Check that the vectors $latex (0, 1)$ and $latex (1, 0)$ are orthogonal.

(2) The notion of uncorrelated random variables has been borrowed from orthogonality. Let $latex (X, Y)$ be arbitrary random variables such that $latex V(X) \ne 0$. Put $latex Z = Y - \frac{\textrm{cov}(X, Y)}{V(X)} X$. Prove that then $latex X, Z$ are uncorrelated. How can you guess a result like this?

Exercise 10.4. First orthogonality relation. Prove that $latex \bar e = 0$.

Comments. (1) This equation is a sample analog of the population equation $latex E u = 0$.

(2) Keep in mind that for a model without the intercept $latex y_i = \beta x_i + u_i$ the sum of residuals may not be zero.

(3) This relation also means that the residual is orthogonal to the vector of unities $latex (1, \ldots, 1)$: $latex \sum e_i = \sum e_i \cdot 1 = n \bar e = 0$.

Proof. By the first normal equation (10.4) $latex \bar e = \overline{y - b_1 - b_2 x} = \bar y - b_1 - b_2 \bar x = 0$.

Corollary 10.1. The fitted value has the same sample mean as $latex y$: $latex \overline{\hat y} = \bar y$.

Proof. The definition of the residual vector implies (see Eq. (10.17)) $latex \bar y = \overline{\hat y + e} = \overline{\hat y} + \bar e = \overline{\hat y}$.

Exercise 10.5. Second orthogonality relation. The fitted value and residual are orthogonal: $latex \sum \hat y_i e_i = 0$.

Proof. From the second normal equation (10.4) we have orthogonality of the residual and $latex (x_1, \ldots, x_n)$:

$latex \sum x_i e_i = n \overline{xe} = n \overline{(xy - b_1 x - b_2 x^2)}$

$latex = n(\overline{xy} - b_1 \bar x - b_2 \overline{x^2}) = 0.$

This equation and the first orthogonality relation give

$latex \sum \hat y_i e_i = \sum (b_1 + b_2 x_i) e_i = b_1 \sum e_i + b_2 \sum x_i e_i = 0.$

Theorem 10.3 (Pythagorean theorem). If the vectors $latex x, y$ are orthogonal, $latex \overline{xy} = 0$, then $latex \overline{(x + y)^2} = \overline{x^2} + \overline{y^2}$.

Proof. $latex \overline{(x + y)^2} = \overline{x^2 + 2xy + y^2} = \overline{x^2} + 2 \overline{xy} + \overline{y^2} = \overline{x^2} + \overline{y^2}$.

Idea. If in the linear model the coefficient $latex \beta_2$ and errors are zero, then all $latex y_i$ are the same and equal to $latex \bar y$ and there is no variation in the dependent variable. Now suppose $latex y_i$ follows the linear model with nontrivial $latex \beta_2$ and errors. The idea is to still measure its deviation from the sample mean $latex \bar y$ but decompose it into two parts: one is due to the linear part $latex \beta_1 + \beta_2 x$ and the other is due to the error. Subtracting the sample mean from both sides of Eq. (10.17) we realize this idea as

$latex y - \bar y = (\hat y - \bar y) + e. \qquad (\textrm{Eq. 10.18})$

Definition 10.6 (Definitions accompanying Eq. (10.18)).

Total Sum of Squares $latex =$ measure of variation of $latex y$ around its mean: $latex TSS = \sum_{i=1}^n (y_i - \bar y)^2$,

Explained Sum of Squares $latex =$ measure of variation of $latex \hat y$ around its mean: $latex ESS = \sum_{i=1}^n (\hat y_i - \bar y)^2$,

Residual Sum of Squares $latex =$ measure of magnitude of the error: $latex RSS = \sum_{i=1}^n e_i^2$.

Exercise 10.6. $latex TSS = ESS + RSS$.

Proof. In the representation (10.18) the two terms on the right are orthogonal by the 1st and 2nd orthogonality relations:

$latex \overline{(\hat y - \bar y) e} = \overline{\hat y e} - \overline{\bar y e} = \overline{\hat y e} - (\bar y)(\bar e) = 0.$

By the Pythagorean theorem then

$latex TSS = n \overline{(y - \bar y)^2} = n \overline{(\hat y - \bar y)^2} + n \overline{e^2} = ESS + RSS.$

Previous: Statistics for testing hypotheses | Next: Goodness of fit

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.