Raising the bar

Chapter 11. Maximum likelihood method

Unit 11.3. Application to the linear model

Exercise 11.3. Suppose observations are generated according to the linear model

$latex y_i = \beta_1 + \beta_2 x_i + u_i, \quad i = 1, \ldots, n. \qquad (\textrm{Eq. 11.1})$

Here

1. the variable $latex x$ is deterministic,

2. $latex u_1, \ldots, u_n$ are i.i.d. normal with mean $latex 0$ and variance $latex \sigma^2$.

The variance is one more parameter that needs to be estimated. Find ML estimators of $latex \beta_1, \beta_2, \sigma^2$.

Solution. Step 1. $latex y_i$ are normal. To make this clearer, we write the errors as $latex u_i = \sigma z_i$ where $latex z_i$ are independent standard normal. Then Eq. (11.1) becomes $latex y_i = \beta_1 + \beta_2 x_i + \sigma z_i$ which is a linear transformation of $latex z_i$. By Exercise 6.2(b) and Eq. (6.18) the density of observation $latex (x_i, y_i)$ is

$latex f(x_i, y_i | \beta_1, \beta_2, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}} \, e^{-\frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^2}}. \qquad (\textrm{Eq. 11.2})$

Step 2. The likelihood function is a product of these densities (because of independence)

$latex L(\beta_1, \beta_2, \sigma^2 | y, x) = \prod_{i=1}^n \frac{1}{\sqrt{2 \pi \sigma^2}} \, e^{-\frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^2}}.$

Pulling the powers of $latex (2 \pi \sigma^2)^{-1/2}$ out and combining the exponents,

$latex L(\beta_1, \beta_2, \sigma^2 | y, x) = (2 \pi \sigma^2)^{-n/2} \, e^{-\sum_{i=1}^n \frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^2}}.$

Step 3. The log-likelihood is

$latex \lambda(\beta_1, \beta_2, \sigma^2 | y, x) = -\frac{n}{2} \log(2 \pi \sigma^2) - \sum_{i=1}^n \frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^2}.$

Splitting $latex \log(2 \pi \sigma^2) = \log(2 \pi) + \log \sigma^2$ gives

$latex \lambda(\beta_1, \beta_2, \sigma^2 | y, x) = -\frac{n}{2} \log(2 \pi) - \frac{n}{2} \log \sigma^2 - \sum_{i=1}^n \frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^2}.$

The first-order conditions are

$latex \frac{\partial \lambda}{\partial \beta_1} = \frac{\partial \lambda}{\partial \beta_2} = \frac{\partial \lambda}{\partial \sigma^2} = 0$

(technically, it is easier to differentiate with respect to $latex \sigma^2$ than to $latex \sigma$). We obtain a system of three equations for determining the parameters:

$latex \frac{\partial \lambda}{\partial \beta_1} = \sum_{i=1}^n \frac{y_i - \beta_1 - \beta_2 x_i}{\sigma^2}$

$latex = \frac{n}{\sigma^2}(\bar y - \beta_1 - \beta_2 \bar x) = 0,$

$latex \frac{\partial \lambda}{\partial \beta_2} = \sum_{i=1}^n \frac{(y_i - \beta_1 - \beta_2 x_i) x_i}{\sigma^2}$

$latex = \frac{n}{\sigma^2} \left( \overline{yx} - \beta_1 \bar x - \beta_2 \overline{x^2} \right) = 0,$

$latex \frac{\partial \lambda}{\partial \sigma^2} = -\frac{n}{2 \sigma^2} + \sum_{i=1}^n \frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^4}$

$latex = \frac{1}{\sigma^2} \left( -\frac{n}{2} + \sum_{i=1}^n \frac{(y_i - \beta_1 - \beta_2 x_i)^2}{2 \sigma^2} \right) = 0.$

Solving this system is left as an exercise. The estimators of $latex \beta_1, \beta_2$ should be the same as in Chapter 10.

Previous: Exponential and Bernoulli distributions | Next: Questions for repetition (Chapter 11)

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.