Raising the bar

Chapter 10. Simple regression

Unit 10.3

Section 10.3.5. Statistics for testing hypotheses

Definition 10.3 (Standard errors of OLS estimators). Standard errors of OLS estimators are obtained from standard deviations by replacing $latex \sigma_u$ with its estimator $latex s_u$:

$latex s_{b_2}^2 = \frac{s_u^2}{\sum_{i=1}^n (x_i - \bar x)^2}, \quad s_{b_1}^2 = s_u^2 \left( \frac{1}{n} + \frac{\bar x^2}{\sum_{i=1}^n (x_i - \bar x)^2} \right).$

These definitions require knowledge of $latex s_u^2$. Intuitively, the squared residual $latex e_i^2$ should estimate $latex \sigma_u^2$. In fact, the right estimator is given by the next theorem.

Theorem 10.1. Define $latex s_u^2 = \sum_i e_i^2 / (n - 2)$. Then $latex E s_u^2 = \sigma_u^2$.

A more general result is proved here.

Theorem 10.2. Let the regressor $latex x$ be deterministic with $latex V_u(x)$ different from zero. Assume that the errors $latex u_1, \ldots, u_n$ are i.i.d. normal with mean zero and that the true coefficients are $latex \beta_1^0, \beta_2^0$. Then the variables

$latex t_{n-2} = \frac{b_1 - \beta_1^0}{s_{b_1}}, \quad t_{n-2} = \frac{b_2 - \beta_2^0}{s_{b_2}}$

are distributed as $latex t$ distribution with $latex n - 2$ degrees of freedom.

Comment. The ratios that define the $latex t$ distributions are exactly the same as before. However, the degrees of freedom are different. We provide here an explanation that can serve as guidance in more complex situations. Initially we have $latex n$ observations, that is, we are in an $latex n$-dimensional space. Estimation of each parameter takes out one dimension. Since in simple regression we have two parameters to estimate, the resulting degrees of freedom is $latex n - 2$.

Previous: Variances of OLS estimators | Next: Orthogonality and its consequences

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.