Raising the bar

Chapter 10. Simple regression

Unit 10.3. Ordinary Least Squares (OLS): definition and derivation

Section 10.3.1.

Definition 10.1. OLS estimators (Gauss). Follow this definition on Figure 10.1.

$latex \quad$1. Take an arbitrary straight line $latex y = b_1 + b_2 x$,

$latex \quad$2. find the residuals $latex e_i = y_i - (b_1 + b_2 x_i)$ from approximation of the observed values $latex y_i$ by the corresponding values $latex b_1 + b_2 x_i$ along the straight line,

$latex \quad$3. form the sum of squared residuals

$latex \quad$$latex \quad$$latex RSS = f(b_1, b_2) = \sum_{i=1}^n (y_i - b_1 - b_2 x_i)^2 = \sum_{i=1}^n e_i^2$

$latex \quad$4. and accept as OLS estimators of $latex \beta_1$ and $latex \beta_2$ those values $latex b_1, b_2$ which minimize RSS.

Figure 10.1: The geometry of OLS estimation.

Section 10.3.2. Derivation of OLS estimators.

In any long derivation there are different ways to obtain the same result. Try to use derivations that (a) you understand and (b) are simple and transparent enough for you to remember. The derivations contained here satisfy an alternative set of quality conditions: (c) they are the shortest possible and (d) they allow for easy generalization.

Recall the chain rule. The squared residual $latex e_i^2$ is a composite function of a quadratic function $latex g(h) = h^2$ and linear function $latex h(b_1, b_2) = y_i - b_1 - b_2 x_i$. Using the chain rule we obtain

$latex \frac{\partial e_i^2}{\partial b_1} = 2 h(b_1, b_2) \frac{\partial h}{\partial b_1} = 2(y_i - b_1 - b_2 x_i)(-1),$

$latex \frac{\partial e_i^2}{\partial b_2} = 2 h(b_1, b_2) \frac{\partial h}{\partial b_2} = 2(y_i - b_1 - b_2 x_i)(- x_i).$

Therefore the first order conditions for the minimization of $latex f(b_1, b_2)$ are

$latex \frac{\partial f(b_1, b_2)}{\partial b_1} = -2 \sum_{i=1}^n (y_i - b_1 - b_2 x_i) = 0,$

$latex \frac{\partial f(b_1, b_2)}{\partial b_2} = -2 \sum_{i=1}^n (y_i - b_1 - b_2 x_i) x_i = 0.$

Getting rid of $latex -2$ and dividing both equations by $latex n$ we obtain

$latex \frac{1}{n} \sum_{i=1}^n (y_i - b_1 - b_2 x_i) = 0, \quad \frac{1}{n} \sum_{i=1}^n (y_i - b_1 - b_2 x_i) x_i = 0. \qquad (\textrm{Eq. 10.2})$

Introducing the vector notation ($latex T$ stands for transposition)

$latex y = (y_1, \ldots, y_n)^T, \quad x = (x_1, \ldots, x_n)^T, \quad u = (u_1, \ldots, u_n)^T \qquad (\textrm{Eq. 10.3})$

we can write Eq. (10.2) as $latex \overline{y - b_1 - b_2 x} = 0$, $latex \overline{(y - b_1 - b_2 x) x} = 0$. Using linearity of sample means rearrange this:

$latex \bar y - b_1 - b_2 \bar x = 0, \quad \overline{yx} - b_1 \bar x - b_2 \overline{x^2} = 0. \qquad (\textrm{Eq. 10.4})$

Definition 10.2. (Normal equations). What we have obtained in Eq. (10.4) is called normal equations.

From the first of them we see that one can find

$latex b_1 = \bar y - b_2 \bar x \qquad (\textrm{Eq. 10.5})$

if one knows $latex b_2$. To find $latex b_2$, plug Eq. (10.5) into the second normal equation:

$latex \overline{yx} - (\bar x)(\bar y) + b_2 \bar x^2 - b_2 \overline{x^2} = 0.$

Note the importance of the order of operations: the bar stands for averaging; in $latex \bar x^2$ the vector $latex x$ is averaged and then the result is squared, while in $latex \overline{x^2}$ the components of that vector are squared and then the result is averaged.

This last equation and equations (b), (c) from Unit 10.1 imply

$latex b_2 = \frac{\overline{yx} - (\bar x)(\bar y)}{\overline{x^2} - \bar x^2} = \frac{\textrm{cov}_u(x, y)}{V_u(x)}. \qquad (\textrm{Eq. 10.6})$

This expression is the main tool for studying various theoretical properties of $latex b_2$.

On the other hand, the main formula used in applications is obtained by writing out the averages:

$latex b_2 = \frac{\frac{1}{n} \sum_{i=1}^n x_i y_i - \left( \frac{1}{n} \sum_{i=1}^n x_i \right) \left( \frac{1}{n} \sum_{i=1}^n y_i \right)}{\frac{1}{n} \sum_{i=1}^n x_i^2 - \left( \frac{1}{n} \sum_{i=1}^n x_i \right)^2} = \frac{n \sum_{i=1}^n x_i y_i - \left( \sum_{i=1}^n x_i \right) \left( \sum_{i=1}^n y_i \right)}{n \sum_{i=1}^n x_i^2 - \left( \sum_{i=1}^n x_i \right)^2}. \qquad (\textrm{Eq. 10.7})$

It is easy to see from equations (10.5) and (10.7) that knowledge of sums $latex \sum x_i y_i$, $latex \sum x_i$, $latex \sum y_i$ and $latex \sum x_i^2$ is sufficient to find $latex b_1, b_2$.

Equations (10.5) and (10.6) are the desired formulas of OLS estimators. Formulas with summation signs are unwieldy and hard to remember. In the Econometrics context it is especially important to use shorter notation.

Previous: Linear model setup | Next: Unbiasedness of OLS estimators

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.