Raising the bar

Chapter 11. Maximum likelihood method

Unit 11.2. Exponential and Bernoulli distributions

Exercise 11.1. Suppose the observations come from an exponential distribution with parameter $latex \mu$. Derive the ML estimator of this parameter.

Solution. Step 1. $latex f(x_i | \mu) = \mu e^{-\mu x_i}$.

Step 2. Assuming independent observations, the joint density is a product of these densities $latex \mu^n e^{-\mu x_1} \ldots e^{-\mu x_n}$.

Step 3. The log-likelihood function is

$latex \lambda = n \log \mu - \mu(x_1 + \ldots + x_n).$

The first order condition is

$latex \frac{\partial \lambda}{\partial \mu} = \frac{n}{\mu} - (x_1 + \ldots + x_n) = 0$

and its solution is $latex \mu = \frac{1}{\bar x}$. To make sure that this is a maximum, we need to check the second order condition. Since $latex \frac{\partial^2 \lambda}{\partial \mu^2} = -\frac{n}{\mu^2}$ is negative, we have really found the maximum.

Conclusion: $latex \hat \mu_{ML} = \frac{1}{\bar x}$ is the ML estimator for $latex \mu$.

Exercise 11.2. Suppose that the parent population is Bernoulli. Derive a ML estimator of the parameter $latex p$.

Solution. Step 1. First check that the information in Table 5.7 can be written in the form $latex f(x) = p^x (1 - p)^{1 - x}, \, x = 0, 1$. This function is a discrete analog of a density. Its value at one observation is $latex f(x_i | p) = p^{x_i} (1 - p)^{1 - x_i}$.

Step 2. If the observations are independent, the joint density is

$latex f(x_1, \ldots, x_n | p) = p^{x_1 + \ldots + x_n} (1 - p)^{n - (x_1 + \ldots + x_n)}.$

Step 3. The log-likelihood function is

$latex \lambda = (x_1 + \ldots + x_n) \log p + [n - (x_1 + \ldots + x_n)] \log(1 - p).$

The first order condition is

$latex \frac{\partial \lambda}{\partial p} = \frac{x_1 + \ldots + x_n}{p} - [n - (x_1 + \ldots + x_n)] \frac{1}{1 - p} = 0.$

This is rearranged to

$latex (x_1 + \ldots + x_n) \left( \frac{1}{p} + \frac{1}{1 - p} \right) = \frac{n}{1 - p}.$

Solving for $latex p$ we get $latex p = \frac{x_1 + \ldots + x_n}{n}$. Since we are talking about an estimator, the result should be written as $latex \hat p_{ML} = \bar x$.

Previous: Main idea and algorithm | Next: Application to the linear model

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.