Raising the bar

Chapter 8. Estimation: One or Two Populations

Unit 8.2. Consistency

This is one of the topics most students don't understand, partly because too little information about it is given in NCT and partly because the notion is complex. I give more information but am not sure that it will simplify the topic. Consistency is an asymptotic property and therefore we have to work with a sequence of estimators.

Figure 8.1. Illustration of consistency.

Definition 8.3 (Consistency). Let $latex {T_n}$ be a sequence of estimators of the same parameter $latex \tau$. We say that $latex T_n$ is a consistent estimator of $latex \tau$ if the probability of deviations of $latex T_n$ from $latex \tau$ decreases as $latex n$ increases. More precisely, this condition is formulated as follows: for any $latex \varepsilon > 0$, $latex P(|T_n - \tau| \ge \varepsilon) \to 0$ as $latex n \to \infty$ (see Figure 1 here for an illustration). In this case we also say that the sequence $latex T_n$ converges to $latex \tau$ in probability.

Comments. (1) Let $latex p_n$ denote the density of $latex T_n$. Then

$latex P(|T_n - \tau| \ge \varepsilon) = \int_{-\infty}^{\tau - \varepsilon} p_n(t) dt + \int_{\tau + \varepsilon}^{+\infty} p_n(t) dt = \textrm{sum of two tails}.$

In Figure 8.1 the sum of two tails (the area below the density outside the interval $latex (\tau - \varepsilon, \tau + \varepsilon)$) must tend to zero as $latex n \to \infty$, and this should be true for any $latex \varepsilon > 0$. The density of $latex T_n$ collapses to a spike at $latex \tau$.

(2) The notion of consistency reflects a change in the point of view. Most of the time in our arguments the sample size was fixed. In the central limit theorem and consistency notion we let the sample size grow to infinity. The related facts are called large sample properties or asymptotic properties. As the sample size grows, instead of one sample mean we have a sequence of sample means corresponding to increasing sample sizes.

Theorem 8.1 (Chebyshov inequality). For any continuous random variable $latex X$ and for any $latex \varepsilon > 0$ one has $latex P(|X| \ge \varepsilon) \le \frac{1}{\varepsilon^2} EX^2.$

Proof. In the area $latex |t| \ge \varepsilon$ the inequality $latex 1 \le \frac{t^2}{\varepsilon^2}$ is true. Therefore

$latex P(|X| \ge \varepsilon) = \int_{-\infty}^{-\varepsilon} p_X(t) dt + \int_{\varepsilon}^{+\infty} p_X(t) dt$ (using the above inequality)

$latex \le \frac{1}{\varepsilon^2} \left( \int_{-\infty}^{-\varepsilon} t^2 p_X(t) dt + \int_{\varepsilon}^{+\infty} t^2 p_X(t) dt \right)$ (increasing the domain of integration)

$latex \le \frac{1}{\varepsilon^2} \int_{\mathbb{R}} t^2 p_X(t) dt = \frac{1}{\varepsilon^2} EX^2. \quad \blacksquare$

Consistency of the sample mean. If $latex X_1, X_2, \ldots$ are i.i.d., then $latex \bar X$ is a consistent estimator of $latex \mu_X$.

Proof. By the Chebyshov inequality for any $latex \varepsilon > 0$

$latex P(|\bar X - \mu_X| \ge \varepsilon) \le \frac{1}{\varepsilon^2} E(\bar X - \mu_X)^2 = \frac{1}{\varepsilon^2} E(\bar X - E\bar X)^2 = \frac{\sigma_X^2}{n \varepsilon^2} \to 0, \quad n \to \infty. \quad \blacksquare$

Relationship between consistency and unbiasedness.

Remark 1. Consistency is an asymptotic property and unbiasedness is a fixed-sample property, so in general one has nothing to do with the other. If we want to compare them, in both cases we have to assume an infinite sequence of estimators.
Remark 2. Suppose we have a sequence of estimators $latex {T_n}$ of $latex \tau$. We can ask what is the relationship between (a) consistency of $latex T_n$ and (b) unbiasedness of $latex T_n$ for every $latex n$.
Remark 3. (a) does not imply (b).

Example 8.1. Put $latex T_n = \bar X + 1/n$. The sample mean consistently estimates the population mean and $latex 1/n \to 0$, therefore $latex T_n$ consistently estimates the population mean but it is biased for all $latex n$.

Remark 4. (b) does not imply (a).

Example 8.2. Take a variable $latex Y$ independent of $latex \bar X$ for all $latex n$ and such that $latex EY = 0,\ V(Y) = c > 0$. Then $latex T_n = \bar X + Y$ is an unbiased estimator of the population mean but it is not consistent because by independence $latex V(T_n) = V(\bar X) + V(Y) = \sigma_X^2/n + c \ge c$.

Remark 5. An unbiased estimator sequence may have a large variance. When choosing between such a sequence and a consistent sequence we face a tradeoff: we can allow for a small bias if variance reduces significantly.

Previous: Unbiasedness and efficiency | Next: Confidence interval for the population mean: known variance

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.