Raising the bar

Chapter 4. Probability

Unit 4.4. Probability

Before proceeding with the formal definitions, try to answer the following questions (in parentheses I give the answers I usually hear from my students).

Exercise 4.1.

(a) Intuitively, what do you think is the probability of getting a head when tossing a coin? (Some say 0.5, others say 50%. There are also answers like: on average, 50 times out of 100).

(b) When rolling a die, what do you think is the probability of getting 1? (Most say 1/6).

(c) Recall Table 2.1 (see Frequencies and distributions). Suppose you obtain a large sample and calculate that the variable takes values listed in the first column with relative frequencies listed in the third column. For the purposes of predicting the values in future samples would you accept the relative frequencies as the probabilities of the respective values? (Most say "Yes").

The answers to parts (a) and (b) suggest that the coin and die are described by Table 4.3.

Table 4.2. Probability tables for C (Coin) and D (Die).

Values of $latex C$	Probabilities		Values of $latex D$	Probabilities
H	$latex 1/2$		$latex {1}$	$latex 1/6$
T	$latex 1/2$		$latex {2}$	$latex 1/6$
			$latex {3}$	$latex 1/6$
			$latex {4}$	$latex 1/6$
			$latex {5}$	$latex 1/6$
			$latex {6}$	$latex 1/6$

Denoting $latex P(A)$ the probability of event $latex A$, from Table 4.3 we can surmise that

$latex P({2,4,6}) = P({2}) + P({4}) + P({6}) = 3/6 = 1/2.$

This makes sense because $latex \{1,2,3,4,5,6\} = \{1,3,5\} \cup \{2,4,6\}$.

Generalizing upon tables 2.1 and 4.3, in case of a variable with $latex n$ values Table 4.4 should be true.

Table 4.3. Probability table in case of a variable with $latex n$ values.

Values of the variable	Probabilities
$latex {x_1}$	$latex {p_1} = P(X = {x_1})$
...	...
$latex {x_n}$	$latex {p_n} = P(X = {x_n})$

In that table, the sample space is $latex S = {{x_1},...,{x_n}}$. Since the numbers $latex {p_i}$ mean percentages, they should satisfy the conditions

1. $latex 0 < {p_i} < 1$ for all $latex i$ and
2. $latex {p_1} + ... + {p_n} = 1$ (completeness axiom).

The completeness axiom means that we have listed in the table all possible basic outcomes. If that axiom were not satisfied, we would have a sample space that is too small.

Until this point I have been using what is called an inductive argument: by analyzing a simple situation and using an element of guessing, try to come up with a definition that could be a base of a theory. When a definition is difficult to understand, try to pinpoint the underlying inductive argument. In the times of Karl Gauss it was common to show inductive arguments. These days they are mostly omitted, partly to save paper and time.

By skipping inductive arguments, you are not saving your time. You are complicating your task.

Before turning to a deductive argument, in which everything is deduced logically from a few basic definitions and postulates, a brush up on functions is necessary. Whenever you talk about a function $latex f(x)$, you have to realize what type of objects you can plug in as arguments and what type of objects you can get as values. Right now we need to discuss dependence of area on a surface and of volume on a body. In both cases the arguments are sets and the values are numbers. Most importantly, the functions are additive. For instance, if we denote $latex {x_i}$ pieces of a jigsaw puzzle, then

$latex \textrm{area}({x_1} \cup ... \cup {x_n}) = \textrm{area}({x_1}) + ... + \textrm{area}({x_n}).$

Similarly, if $latex {r_i}$ denotes a room in a building, then

$latex \textrm{volume}({r_1} \cup ... \cup {r_n}) = \textrm{volume}({r_1}) + ... + \textrm{volume}({r_n}).$

Note the importance, for these equations to hold, of the facts that the pieces $latex {x_1},...,{x_n}$ do not overlap and the rooms $latex {r_1},...,{r_n}$ are disjoint. Put it simply, if $latex A \cap B = \varnothing$, as in the Disjoint figure below, then $latex \textrm{area}(A \cup B) = \textrm{area}(A) + \textrm{area}(B)$. If, on the other hand, $latex A \cap B \ne \varnothing$, as in the more general case of the Not Disjoint figure, then $latex \textrm{area}(A \cup B) = \textrm{area}(A) + \textrm{area}(B) - \textrm{area}(A \cap B)$ because in the sum $latex \textrm{area}(A) + \textrm{area}(B)$ the area of $latex A \cap B$ is counted twice.

Probability. Let $latex S$ be a sample space. By probability on $latex S$ we mean a numerical function $latex P(A)$ of events $latex A \subset S$ such that

1. if $latex A,B$ are mutually exclusive, then $latex P(A \cup B) = P(A) + P(B)$ (additivity),
2. for any $latex A$, $latex P(A)$ is nonnegative, and
3. $latex P(S) = 1$ (completeness axiom).

Once again, we apply set operations to events and arithmetic operations to probabilities. Equations like $latex P(A + B) = P(A) + P(B),\ P(A \cup B) = P(A) \cup P(B)$ don't make sense!

Any definition as complex as this needs some chewing and digesting. This includes going back, to the motivating examples, going forward, to the consequences, and going sideways, to look at variations or cases when it is not satisfied.

Going back. To put the example from Table 4.4 into a general framework, for any event $latex A \subset S$ define

$latex P(A) = \sum\limits_{{x_i} \in A} P({{x_i}})$

(read like this: sum of probabilities of those $latex {x_i}$ which belong to $latex A$).

For example, $latex P({{x_1},{x_2},{x_3}}) = P({{x_1}}) + P({{x_2}}) + P({{x_3}})$. Later on we will use more summation signs, and you have to get used to them. Always write out the sums completely if you don't understand summation signs. As an exercise, try to show that the $latex P(A)$ defined here is additive, nonnegative and satisfies the completeness axiom.

Going forward.

(1) One way to understand a property is to generalize it. In the case under consideration, additivity is generalized to the case of $latex n$ events. If $latex {A_1},...,{A_n}$ are disjoint, then from item (i) in the definition of probability we have

$latex P({A_1} \cup ... \cup {A_n}) = P(({A_1} \cup ... \cup {A_{n - 1}}) \cup {A_n}) = P({A_1} \cup ... \cup {A_{n - 1}}) + P({A_n})$

$latex = P({A_1} \cup ... \cup {A_{n - 2}}) + P({A_{n - 1}}) + P({A_n}) = ... = P({A_1}) + ... + P({A_n}).$

This type of derivation of a general statement from its special case is called induction.

(2) The complement rule. Representing $latex S$ as $latex S = A \cup \bar A$, from items (i) and (iii) we deduce

$latex P(A) + P(\bar A) = P(S) = 1 \qquad (\textrm{Eq. 4.1})$

which implies the complement rule:

$latex P(\bar A) = 1 - P(A).$

(3) Probability range. From (ii) we know that $latex P(A) \ge 0$. In addition, since in Eq. (4.1) $latex P(\bar A) \ge 0$, we have $latex P(A) \le 1$.

(4) Assembly formula. Fix some event $latex B$. If $latex {A_1},...,{A_n}$ are disjoint, then pieces $latex B \cap {A_1},...,B \cap {A_n}$ are also disjoint. Further, if $latex {A_1},...,{A_n}$ are collectively exhaustive, then those pieces comprise $latex B$. Therefore by additivity we have the assembly formula (also called law of total probability)

$latex P(B) = P(B \cap {A_1}) + ... + P(B \cap {A_n}). \qquad (\textrm{Eq. 4.2})$

Going sideways. While area and volume are additive and nonnegative, they generally do not satisfy condition (iii) of the probability definition.

When a definition is long, after chewing and digesting it you should formulate its short-and-easy-to-remember version.

Probability short definition. A probability is a nonnegative additive function defined on some sample space $latex S$ and such that $latex P(S) = 1$.

Impossible and sure events. An impossible event can be defined by $latex P(A) = 0$. When $latex P(A) = 1$, we say that $latex A$ is a sure event. In the discrete case we are considering the only impossible event is $latex A = \varnothing$ and the only sure event is the sample space.

In many practical applications it is not necessary or possible to check that all requirements of probabilistic definitions are satisfied.

Exercise 4.2.

1. What is the probability that tomorrow the sun will rise?
2. What is the probability of you being at home and at the university at the same time?
3. Probability of which event is higher: that tomorrow's temperature at noon will not exceed $latex 19\,^{\circ}\mathrm{C}$ or that it will not exceed $latex 25\,^{\circ}\mathrm{C}$?

---

Previous: Set versus probabilistic terminology | Next: Ways to find probabilities

---

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.