Raising the bar

Chapter 11. Maximum likelihood method

Unit 11.1. Main idea and algorithm

Figure 11.1. Sample and three different densities.

The main idea of the maximum likelihood (ML) method is illustrated in Figure 11.1. We start with the sample depicted with points on the horizontal axis. Then we think which of the densities shown on the figure is more likely to have generated that sample. Of course, it's the one on the left, filled with grey.

This density takes higher values at observed points than the other two. Note also that, if we take densities from the same parametric family, the location of the density is determined by its parameters. This explains the main idea: choose the parameters so as to maximize the density at the observed points.

Algorithm.

Step 1. A statistical model usually contains a random term. To describe that term, choose a density from some parametric family. Denote it $latex f(x | \theta)$ where $latex \theta$ is a parameter or a set of parameters. $latex f(x_i | \theta)$ is the value of the density at the $latex i$th observation.

Step 2. Assume that observations are independent. Then the joint density is a product of own densities: $latex f(x_1, \ldots, x_n | \theta) = f(x_1 | \theta) \ldots f(x_n | \theta)$. Since the observations are fixed, the joint density is a function of just parameters.

Definition 11.1 (Likelihood function and maximum likelihood estimators). The joint density as a function of just parameters is called a likelihood function and denoted $latex L(\theta | x_1, \ldots, x_n) = f(x_1, \ldots, x_n | \theta)$, to reflect the fact that the parameters are the main argument. The parameters that maximize the likelihood function, if they exist, are the maximum likelihood estimators, and thus the name Maximum Likelihood (ML) method.

Step 3. Since $latex \log x$ is a monotone function, the likelihood $latex L(\theta | x_1, \ldots, x_n)$ and the log-likelihood function $latex \lambda(\theta) = \log L(\theta | x_1, \ldots, x_n)$ are maximized at the same time. The likelihood is often a multiplicative function, in which case maximizing the log-likelihood is technically easier.

Comments. (1) Most of the time the likelihood function is difficult to maximize analytically. Then maximization is done on the computer. A numerical algorithm can give the solution only approximately. Moreover, the likelihood function may not have maximums at all or may have many maximums; in the latter case the computer gives only one solution.

(2) One should distinguish models and methods. The OLS method has been applied to the linear model in Chapter 10. The ML method in case of linear models often yields the same estimators as the least squares method. Most practical applications are dealt with using the least squares method. All exercises for maximum likelihood are theoretical, as it is seen from the algorithm.

Previous: Questions for repetition (Chapter 10) | Next: Exponential and Bernoulli distributions

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.