Raising the bar

Chapter 9. Hypothesis Testing

Unit 9.4. $latex p$-values

Figure 9.4. Changing level of significance.

Preamble. Consider the situation in Figure 9.4.

Let us think about the dynamics of the confidence interval when $latex \alpha$ changes. When $latex \alpha$ decreases, the confidence interval becomes wider. The sample mean, being random, can take values both inside and outside the given confidence interval. It is possible to have the situation depicted in Figure 9.4: $latex w_0 + ME_1 < \bar w < w_0 + ME_2$ where $latex ME_i$ corresponds to $latex \alpha_i, \, i = 1, 2$ and $latex \alpha_2 < \alpha_1$. We can reject the null at the significance level $latex \alpha_1$ and would be able to do that at a slightly lower significance level. Similarly, we fail to reject the null at the level of significance $latex \alpha_2$ and the decision would be the same at a slightly higher significance level. As it is desirable to reduce the probability of Type I error, this leeway is not good. We would like to know the precise value of the significance level, beyond which we would always fail to reject the null hypothesis.

Definition 9.5 ($latex p$-value). The $latex p$-value is the smallest significance level at which the null hypothesis can still be rejected.

In case of long verbal definitions divide and repeat them in short logical groups:

The $latex p$-value

$latex \quad$ is the smallest significance level

$latex \quad$ $latex \quad$ at which the null hypothesis

$latex \quad$ $latex \quad$ $latex \quad$ can still be rejected.

Comments. (1) The critical value $latex z_{\alpha/2}$ is a function of $latex \alpha$ and is defined from $latex P(z > z_{\alpha/2}) = \alpha/2$. It doesn't depend on the realized value $latex \bar w$. The $latex p$-value, on the other hand, is defined (for a two-tailed test) by $latex P(z > z_{\bar w}) = p/2$, where $latex z_{\bar w} = (\bar w - w_0)/(\sigma_w/\sqrt n)$ is the standardized realized sample mean, so $latex p$ depends on the data.

(2) It must be clear from the definition that

(a) the null is rejected at any $latex \alpha \ge p$ and

(b) for any $latex \alpha < p$ the decision is "Fail to reject the null".

Previous: Confidence intervals and hypothesis testing | Next: Power of a test

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.