Raising the bar

Chapter 4. Probability

Unit 4.8. Conditional probabilities

Table 4.9. Leftover joint probabilities.

Example 4.6. In the wheat Example 4.5 (see All about joint events), suppose the summer is over, the weather has been good and the harvesting time is approaching. We are left with one line of Table 4.7 given in Table 4.9. In that table the sum

$latex P(S \cap H) + P(S \cap \bar H) = P(S) \qquad (\textrm{Eq. 4.5})$

is not equal to 1 and the completeness axiom is not satisfied. But we know that there are no missing events. To satisfy the completeness axiom, we can divide both sides of Eq. (4.5) by $latex P(S)$,

$latex \frac{{P(S \cap H)}}{{P(S)}} + \frac{{P(S \cap \bar H)}}{{P(S)}} = 1$

and treat $latex P(S \cap H)/P(S)$ and $latex P(S \cap \bar H)/P(S)$ as probabilities of events $latex H$ and $latex \bar H$, provided that $latex S$ has occurred.

Conditional probability. Let $latex P(S) > 0$. The probability $latex P(H|S) = \frac{{P(H \cap S)}}{{P(S)}}$ is called probability of $latex H$ conditional on $latex S$.

Comments. (1) If we are interested in probabilities of $latex H$ and $latex \bar H$, why not use simply $latex P(H)$ and $latex P(\bar H)$? The reason is that occurrence of $latex S$ provides us with additional information. There may be some dependence between events $latex S$ and $latex H$, in which case conditional probabilities $latex P(H|S)$ and $latex P(\bar H|S)$ will be a better reflection of reality than own probabilities of the pair $latex {H,\,\bar H}$. This process of updating our beliefs as new information arrives is one of the fundamental ideas of the theory of probabilities.

(2) Often random variables are introduced axiomatically. That is, take any numbers $latex {p_i}$ satisfying $latex 0 < {p_i} < 1$ and $latex {p_1} + ... + {p_n} = 1$ and you can be sure that there is a random variable whose values have probabilities exactly equal to those $latex {p_i}$'s. Conditional probabilities are defined axiomatically and the subsequent theory justifies their introduction.

Multiplication rule. The definition of $latex P(H|S)$ immediately implies the multiplication rule

$latex P(H \cap S) = P(H|S)P(S).$

Try to see the probabilistic meaning of this equation. Denote $latex M$ the event that you obtain a passing grade in Math and by $latex S$ the event that you obtain a passing grade in Stats. Suppose that Math is a prerequisite for Stats. Then the equation $latex P(S \cap M) = P(S|M)P(M)$ basically tells us that to obtain a passing grade in both Math and Stats you have to pass Math first and, with that prerequisite satisfied, to obtain a passing grade in Stats. The fact that you have not failed Math tells us something about your abilities and increases your chance of passing Stats.

Independence of events.

Intuitive definition. Events $latex A$ and $latex B$ are called independent if occurrence of one of them does not influence occurrence of the other. For example, what happens to the coin has nothing to do with what happens to the die.

Formal definition. Events $latex A$ and $latex B$ are called independent if $latex P(A \cap B) = P(A)P(B)$. When $latex P(B) > 0$, this can be equivalently written as $latex P(A|B) = P(A)$.

Comments. This definition is applied in two ways.

(1) From independence to equation. If you know that $latex A$ and $latex B$ are independent, you can multiply $latex P(A)$ and $latex P(B)$ to get $latex P(A \cap B)$. For example,

$latex P(\textrm{Coin} = 1,\,\textrm{Die} = 1) = P(\textrm{Coin} = 1)P(\textrm{Die} = 1) = 1/12.$

(2) From equation to independence (or dependence). If you find that $latex P(A \cap B) \ne P(A)P(B)$, you conclude that $latex A$ and $latex B$ are dependent.

Exercise 4.7. Prove that if $latex A$ and $latex B$ are independent, then their complements are also independent.

Always write down what is given and what you need to get. When rearranging expressions, indicate what rule you are applying, until everything becomes trivial.

Proof. We know that $latex P(A \cap B) = P(A)P(B)$ and we need to prove that $latex P(\bar A \cap \bar B) = P(\bar A)P(\bar B).$

$latex P(\bar A)P(\bar B) = [1 - P(A)][1 - P(B)]$ (complement rule)

$latex = 1 - P(A) - P(B) + P(A)P(B)$ (multiplying out)

$latex = 1 - P(A) - P(B) + P(A \cap B)$ (by independence)

$latex = 1 - P(A \cup B)$ (additivity rule)

$latex = P(\overline {A \cup B})$ (complement rule)

$latex = P(\bar A \cap \bar B)$ (de Morgan).

Figure 4.1. Example of independent events

Example. Take the square $latex S = [0,1] \times [0,1]$ as the sample space, see Table 4.2 (in Set versus probabilistic terminology) for the definition. For any set $latex A \subset S$ define its probability to be its area: $latex P(A) = \textrm{area}(A)$. Since area is additive, nonnegative and $latex P(S) = 1$, this will be really a probability.

For any $latex [a,b] \subset [0,1]$ define $latex A$ as the vertical strip $latex A = {(x,y):\,a \le x \le b,\,0 \le y \le 1}$ (refer to Figure 4.1, see also this video). For any $latex [c,d] \subset [0,1]$ define $latex B$ as the horizontal strip $latex B = {(x,y):\,0 \le x \le 1,\,c \le y \le d}$. Then

$latex P(A) = b - a,\ P(B) = d - c,$

$latex A \cap B = {(x,y):\,a \le x \le b,\,c \le y \le d},$

$latex P(A \cap B) = (b - a)(d - c) = P(A)P(B).$

Thus, $latex A$ and $latex B$ are independent. Now try to move the strip $latex A$ left and right and the strip $latex B$ up and down. Independence takes a physical meaning: movements in mutually orthogonal directions do not affect each other.

Bayes theorem. This theorem is one of realizations of the idea about updating one's beliefs on the basis of new information. The easiest way to remember it is to express $latex P(B|A)$ through $latex P(A|B)$:

$latex P(B|A) = \frac{{P(B \cap A)}}{{P(A)}} = \frac{{P(B \cap A)}}{{P(B)}}\frac{{P(B)}}{{P(A)}} = P(B)\frac{{P(A|B)}}{{P(A)}}.$

Interpretation in terms of subjective probabilities. Before occurrence of the event $latex A$, an expert forms an opinion about likelihood of the event $latex B$. That opinion, embodied in $latex P(B)$, is called a prior probability. After occurrence of $latex A$, the expert's belief is updated to obtain $latex P(B|A)$, called a posterior probability. By the Bayes theorem, updating is accomplished through multiplication of the prior probability by the factor $latex \frac{{P(A|B)}}{{P(A)}}$.

Complex form of Bayes theorem (simple way to complicate the matters). Suppose $latex {B_1},...,{B_K}$ form a disjoint covering of the sample space. By the Bayes theorem for each $latex {B_i}$

$latex P({B_i}|A) = \frac{{P(A|{B_i})P({B_i})}}{{P(A)}}.$

Here $latex P(A)$ can be replaced by

$latex P(A) = P(A \cap {B_1}) + ... + P(A \cap {B_K})$ (assembly formula)

$latex = P(A|{B_1})P({B_1}) + ... + P(A|{B_K})P({B_K})$ (multiplication rule).

The result is

$latex P({B_i}|A) = \frac{{P(A|{B_i})P({B_i})}}{{P(A|{B_1})P({B_1}) + ... + P(A|{B_K})P({B_K})}}. \qquad (\textrm{Eq. 4.6})$

To me this is a good proof that knowing the derivation is better than remembering the result.

Exercise 4.8. (Wikipedia: Bayes' theorem) Suppose a man told you he had a nice conversation with someone on the train. Assume the speaker was as likely to strike up a conversation with a man as with a woman. Now suppose he also told you that his conversational partner had long hair. Let $latex W$ represent the event that the conversation was held with a woman, and $latex L$ denote the event that the conversation was held with a long-haired person. Suppose it is also known that 75% of women have long hair and that 15% of men have long hair. Find the probability that the conversation was held with a woman, given the fact that the person had long hair, or, in our notation, $latex P(W|L)$.

Previous: All about joint events | Next: Problem solving strategy

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.