Raising the bar

Chapter 6. Continuous Random Variables and Probability Distributions

Unit 6.6. The normal distribution

Parametric distributions. Following the general recommendation to think in terms of sets rather than elements, let us think about families of similar distributions. One element of a family of similar distributions can be thought of as a value of a function. The argument of the function is called a parameter. This parameter can be a number or a vector. All families we have seen so far are either one-parametric or two-parametric.

Let us look at some examples.

1. The family of Bernoulli distributions is one-parametric. A Bernoulli distribution is uniquely identified by the probability $latex p \in (0,1)$.
2. The binomial variable has two parameters: $latex n$ and $latex p$. Its value $latex x$ which we write in the expression $latex P({S_n} = x)$ is not a parameter.
3. The Poisson distribution has one parameter: $latex EX = V(X) = \lambda$.
4. The uniform distribution has two parameters, which are the endpoints of the segment $latex [a,b]$.
5. Remembering about parameters is especially important when you work with more than one distribution. For example, the result from Poisson approximation to the binomial on Poisson approximation to the binomial distribution is clearer if you write it indicating the parameters after the pipe sign $latex |$: $latex P(\textrm{Binomial} = x|n,p) \approx P(\textrm{Poisson} = x|\lambda = np).$

We define normal distributions in two steps: first we introduce the standard normal distribution (as a special case of a normal distribution) and then a general normal distribution (as a linear transformation of the standard normal distribution).

Standard normal distribution

Standard normal distribution. A standard normal distribution, denoted $latex z$, has a density function

$latex {p_z}(t) = \frac{1}{{\sqrt {2\pi } }}{e^{ - \frac{{{t^2}}}{2}}},\,\,-\infty < t < \infty. \qquad (\textrm{Eq. 6.18})$

It is universal in the sense that many other distributions converge to it. The exact meaning of this will be explained in Theorem 6.1 (see Normal approximation to the binomial).

Odd and even functions. We say that a function $latex f(x)$ of a real argument is odd if $latex f(-x) = -f(x).$ Such a function is called symmetric about zero or even if $latex f(-x) = f(x).$ We shall need a simple statement: If a function $latex f(x)$ is symmetric, then $latex xf(x)$ is odd: $latex (-x)f(-x) = -xf(x).$

Property 1. The density of a standard normal is bell-shaped and symmetric about zero.

Proof. From the properties of the exponential function we see that $latex {p_z}$ is smooth and $latex \lim _{t \to \pm \infty }{p_z}(t) = 0$. It is even: $latex {p_z}(-t) = \frac{1}{{\sqrt {2\pi } }}{e^{ - \frac{{{{( - t)}^2}}}{2}}} = {p_z}(t)$ for any real $latex t$.

Property 2. $latex Ez = 0$.

Proof. The symmetry of the density implies that $latex t{p_z}(t)$ is an odd function and by additivity of integrals

$latex Ez = \int_{\mathbb{R}}^{} {t{p_z}(t)dt} = \int_{ - \infty }^0 {t{p_z}(t)dt} + \int_0^{ + \infty } {t{p_z}(t)dt}.$

This mean is zero because the area above the x-axis $latex A = \int_0^{ + \infty } {t{p_z}(t)dt}$ cancels out with $latex -A = \int_{ - \infty }^0 {t{p_z}(t)dt}$, see Figure 6.3.

Figure 6.3. Density of standard normal.

Property 3. $latex E{z^2} = 1$.

We take this for granted (the proof is formidable). This property implies $latex V(z) = E{z^2} - {(Ez)^2} = 1$ and $latex \sigma (z) = 1$.

Property 4. Even though the standard normal can take arbitrarily large values, the probability of $latex z$ taking large values quickly declines. More precisely,

$latex P(|z| > 1) = 0.3174,\,\,P(|z| > 2) = 0.0456,\,\,P(|z| > 3) = 0.0028. \qquad (\textrm{Eq. 6.19})$

Proof. Using the general definition of a density and the definition of the density of the standard normal Eq. (6.18) we have

$latex P(|z| > 1) = \int_{ - \infty }^{ - 1} {p_z}(t)dt + \int_1^{ + \infty } {p_z}(t)dt$ (by symmetry)

$latex = 2\int_1^{ + \infty } {p_z}(t)dt = \frac{2}{{\sqrt {2\pi } }}\int_1^\infty {e^{ - \frac{{{t^2}}}{2}}} dt.$

There are no simple rules to calculate integrals like this. Therefore such integrals have been tabulated. Whatever density and table you deal with, try to express the integral through the area given in the table. Table 1 in the appendix of NCT or the standard normal table in this book together with Excel gives the area $latex F(z)$ for $latex z > 0$ (in that table $latex z$ means a real number, not the standard normal variable). By the complement rule and aforementioned table

$latex \int_1^{ + \infty } {{p_z}(t)dt} = 1 - \int_{ - \infty }^1 {{p_z}(t)dt} = 1 - F(1) = 1 - 0.8413$

so by symmetry $latex P(|z| > 1) = 2(1 - 0.8413) = 0.3174$. The proof of the other two equalities in Eq. (6.19) is left as an exercise.

Normal distribution

Normal distribution. A normal variable $latex X$ is defined as a linear transformation of the standard normal: $latex X = \sigma z + \mu$, where $latex \sigma > 0$.

Comment. The usual way to define a normal variable is by using its density $latex {p_X}(t) = \frac{1}{{\sqrt {2\pi {\sigma ^2}} }}{e^{ - \frac{{{{(t - \mu )}^2}}}{{2{\sigma ^2}}}}}$. Working with this definition is more difficult, trust me. Instead, we define the normal distribution in two steps: 1) the standard normal is defined as the special case with $latex \mu = 0,\,\,\sigma = 1$ and 2) the general normal is a linear transformation of the standard normal. In particular, this way makes obvious that $latex z = \frac{{X - \mu }}{\sigma }$ is standard normal when $latex X$ is normal with parameters $latex \mu ,\sigma$.

Property 1. The density of $latex X$ is bell-shaped and centered on $latex \mu$: $latex EX = \mu$.

Proof. By linearity of means $latex EX = \sigma Ez + \mu = \mu$. Recall our discussion of linear transformations from Standardization: multiplication of $latex z$ by $latex \sigma$ stretches/squeezes its density and the subsequent addition of $latex \mu$ moves the result to the left or right.

Property 2. $latex V(X) = {\sigma ^2}$.

Proof. Adding a constant doesn't change variance and variance is homogeneous: $latex V(X) = V(\sigma z + \mu) = V(\sigma z) = {\sigma ^2}V(z) = {\sigma ^2}.$

Property 3. Standardization of a normal distribution gives a standard normal distribution.

Proof. By properties 1 and 2 $latex \frac{{X - EX}}{{\sigma (X)}} = \frac{{\sigma z + \mu - \mu }}{\sigma } = z$.

Property 4. Empirical rule:

$latex P(|X - \mu | \le \sigma ) = 0.68,\,\,P(|X - \mu | \le 2\sigma ) = 0.95,\,\,P(|X - \mu | \le 3\sigma ) = 0.997.$

Proof. This follows from Eq. (6.19).

$latex P(|X - \mu | \le \sigma ) = P\left( \left| \frac{{X - \mu }}{\sigma } \right| \le 1 \right) = P(|z| \le 1) = 1 - 0.3174 \approx 0.68,$

for example.

Notation. Another notation for a normal variable with mean $latex \mu$ and variance $latex {\sigma ^2}$ is $latex N(\mu ,{\sigma ^2})$. The standard normal in this notation is $latex N(0,1)$. We see that the family of normal distributions is two-parametric. Each of the parameters takes an infinite number of values, and it would be impossible to tabulate all normal distributions. This is why we need the standardization procedure.

Previous: The uniform distribution | Next: Normal approximation to the binomial

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.