Raising the bar

Chapter 8. Estimation: One or Two Populations

Unit 8.1. Unbiasedness and efficiency

Exercise 8.1. If there exists one unbiased estimator of a population parameter, then there exist many unbiased estimators of that parameter.

Proof. Let $latex ET = \tau$. Take any random variable $latex Y$ with mean zero and put $latex X = T + Y$. Then $latex EX = ET + EY = \tau. \quad \blacksquare$

This exercise makes clear that even though unbiasedness is a desirable property, it is not enough to choose from many competing estimators of the same parameter. In statistics we want not only to be able to hit the target on average but also to reduce the spread of estimates around their mean.

Definition 8.1 (Efficiency). If $latex T_1$ and $latex T_2$ are two unbiased estimators of the same parameter $latex \tau$ and $latex V(T_1) < V(T_2)$, then we say that $latex T_1$ is more efficient than $latex T_2$. Thus efficiency is a relative property.

Exercise 8.2. If $latex T_1$ and $latex T_2$ are two unbiased estimators of the same parameter $latex \tau$ and $latex V(T_1) \ne V(T_2)$, then, with exception of two cases, there exists an unbiased estimator of the same parameter which is more efficient than either $latex T_1$ or $latex T_2$.

Proof. Define $latex T(a) = a T_1 + (1 - a) T_2$ where $latex a$ is real. This is a special type of a linear combination which is convenient to parameterize the points of the segment $latex [T_1, T_2]$. With $latex a = 0$ or $latex a = 1$ we get the left and right endpoints of the segment: $latex T(1) = T_1, T(0) = T_2$. Values $latex a \in (0,1)$ give all other points of the segment. Values $latex a \notin [0,1]$ give points of the straight line drawn through $latex T_1, T_2$.

$latex T(a)$ is unbiased: $latex ET(a) = a E T_1 + (1 - a) E T_2 = \tau$. Its variance is

$latex V(T) = a^2 V(T_1) + 2a(1 - a) \textrm{cov}(T_1, T_2) + (1 - a)^2 V(T_2).$

This is a parabola with branches looking upward because the coefficient of $latex a^2$

$latex V(T_1) - 2 \textrm{cov}(T_1, T_2) + V(T_2) = V(T_1 - T_2)$

is positive, excluding the trivial case $latex T_1 = T_2$. The first-order condition for its minimization is

$latex 2a V(T_1) + (2 - 4a) \textrm{cov}(T_1, T_2) - 2(1 - a) V(T_2) = 0.$

Solving this equation for $latex a$, we get

$latex a = \frac{V(T_2) - \textrm{cov}(T_1, T_2)}{V(T_1) - 2 \textrm{cov}(T_1, T_2) + V(T_2)}.$

The minimum of $latex V(T)$ (the bottom of the parabola) is at point $latex a = 0$ if $latex V(T_2) = \textrm{cov}(T_1, T_2)$ and at point $latex a = 1$ if $latex V(T_1) = \textrm{cov}(T_1, T_2)$. In these two cases it is not possible to reduce variance by forming linear combinations. In all other cases $latex T(a)$ is more efficient than either $latex T_1$ or $latex T_2$. $latex \blacksquare$

When $latex T$ is an unbiased estimator of $latex \tau$, variance of $latex T$ is an adequate measure of dispersion of values of $latex T$. If, on the other hand, $latex T$ is biased, it makes sense to consider dispersion around the true parameter $latex E(T - \tau)^2$ instead of $latex E(T - ET)^2$.

Definition 8.2 (Mean squared error). The quantity $latex MSE = E(T - \tau)^2$ is called a mean squared error. In practice, since the true parameter is usually unknown, it is useless. It is useful in Monte Carlo simulations, when the true parameter is determined by the researcher.

Exercise 8.3. What is the relationship between $latex MSE$ and variance?

Solution.

$latex MSE = E(T - \tau)^2 = E(T - ET + ET - \tau)^2$ (open parentheses and use linearity)

$latex = V(T) + 2 E(T - ET)(ET - \tau) + (ET - \tau)^2 = V(T) + \textrm{Bias}^2.$

It is useful to accompany this equation with a couple of definitions. Thinking about the case of a biased estimator, represent $latex T - \tau$ as $latex T - \tau = (T - ET) + (ET - \tau)$. Some people call $latex \textrm{Bias} = ET - \tau$ a systematic error (it is a constant) and $latex T - ET$ is called a random error (it is a random variable with zero mean). The relationship between $latex MSE$ and variance says that $latex MSE$ is a sum of the random error variance and squared systematic error. $latex \blacksquare$

Previous: Questions for repetition (Chapter 7) | Next: Consistency

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.