Raising the bar

Chapter 9. Hypothesis Testing

Unit 9.2. Tradeoff between Type I and Type II errors

We always try to minimize the probability of error and maximize the probability of being right.

Definition 9.4 (Significance level and power of a test). The probability $latex \alpha = P(\textrm{Type I error})$ is called a significance level (we want it to be low). The probability $latex \beta = P(\textrm{Type II error})$ doesn't have a special name but $latex 1 - \beta = P(\textrm{Accepting } H_a \textrm{ when it is true})$ is called a power of a test (we want it to be high).

Thus, in hypothesis testing we want the significance level to be low and the power to be high. Unfortunately, these are contradicting requirements.

Exercise 9.1. Suppose applicants to a university are given a test which is graded on a 100 point scale. The admission committee has to choose the passing grade. In this context discuss the tradeoff between Type I and Type II errors.

Solution. We categorize applicants as "weak" and "strong". These characteristics are not directly observable and the committee has to use the grade as a proxy. It is imperfect: strong applicants may get lower grades than weak ones, due to a number of reasons. In statistics all statements are probabilistic. Committee's task is to make the decision using conditional probabilities

$latex p_w = P(\textrm{applicant is weak | grade}), \quad p_s = P(\textrm{applicant is strong | grade}).$

The first of these probabilities decreases with grade and the second increases, see Figures 9.1 and 9.2.

Figure 9.1. Probability that applicant is weak.

Figure 9.2. Probability that applicant is strong.

A higher cut-off level $latex g_2$ is better to sort out weak students. A lower passing grade $latex g_1$ is preferable if the objective is to admit all potentially good students. Since these are contradictory objectives, the role of hypotheses is not symmetric: $latex H_a$ is given preference. If the emphasis is on admission of strong students, the choice should be

$latex H_0: \textrm{applicant is weak}, \quad H_a: \textrm{applicant is strong}.$

Universities that maximize tuition do the opposite: they don't want to lose anybody who is able to study and pay.

Important: the distribution used for decision making is determined by the null hypothesis.

Cautious terminology. When we reject the null, we indicate the significance level to emphasize that our decision may be wrong with probability $latex \alpha$. Instead of saying "we accept the null" we say "fail to reject the null" bearing in mind that $latex H_0$ can still be wrong.

Previous: Concepts of hypothesis testing | Next: Confidence intervals and hypothesis testing

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.