Raising the bar

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.5. Independence and its consequences

Background. You have to recall the definition of independence of two events $latex P(A \cap B) = P(A)P(B)$ and realize that a random variable is a much more complex object than an event. Visually, a random variable is a 1-D table with values and probabilities. A pair of random variables is represented by a 2-D table similar to Table 4.9 (see Conditional probabilities). Since we want to work with variables whose probability distributions are not necessarily the same, a table of type Table 5.2 (see Linear combinations of random variables) with the same probabilities for $latex X$ and $latex Y$ is insufficient. We deal with two random variables separately represented by the tables 5.3.

Table 5.3. Separate probability tables.

In general, $latex n$ is different from $latex m$. The probabilities are also different, that's why they are provided with superscripts. If you remember, in this situation we cannot directly define the sum $latex X + Y$ and the product $latex XY$. This is why we need to work with the pair $latex (X,Y)$ which has values $latex ({x_i},{y_j}),\,\,i = 1,...,n,\,\,j = 1,...,m$ (you can think about throwing a coin and die at the same time; any value on the coin can occur in combination with any value on the die). The probabilities of these values are denoted $latex {p_{ij}},\,\,i = 1,...,n,\,\,j = 1,...,m$. Now we arrange a 2-D table putting a pair of values $latex ({x_i},{y_j})$ and corresponding probability $latex {p_{ij}}$ in the same cell of Table 5.4.

Exercise 5.2. One and the same random variable can have many different (equivalent) realizations. The variable $latex X$ is realized in Table 5.3 on the sample space $latex { {x_1},...,{x_n}}$. What is its realization in terms of Table 5.4 (what are the sample space, probability distribution and values)?

Table 5.4. Joint values and probabilities.

Independence of variables

Intuitive definition. We say that $latex X$ and $latex Y$ are independent if what happens to $latex X$ does not influence $latex Y$ in any way.

Formal definition. We say that $latex X$ and $latex Y$ are independent if the events independence condition is satisfied for every pair of their values:

$latex {p_{ij}} = p_i^X p_j^Y,\,\,i = 1,...,n,\,\,j = 1,...,m. \qquad (\textrm{Eq. 5.3})$

Comments. (1) This is an opportune moment to discuss the relation between universal statements and existence statements. Consider a universal statement: all students in a given class are younger than 30. Its opposite is: there are students in a given class who are at least 30 years old, which is an existence statement. This is a general rule of logic: rejecting a universal statement you obtain an existence one, and vice versa.

(2) Rejecting the definition of independence, we obtain the definition of dependence, and it should be an existence statement. Thus, we say that $latex X$ and $latex Y$ are dependent if at least one of the equations Eq. (5.3) is not true.

Sum and product of random variables. The sum $latex X + Y$ and the product $latex XY$ are defined by Table 5.5.

Table 5.5. General definitions of the sum and product.

Exercise 5.3. How is this definition related to the earlier one?

Exercise 5.4. Prove multiplicativity of expected values: If $latex X,\,Y$ are independent, then their mean is multiplicative: $latex E(XY) = (EX)(EY)$.

Proof. Earlier you were asked to rewrite the proof of linearity of means using summation signs. Here we need a special case of that property:

$latex \sum\limits_{j = 1}^m {a{z_j} = } a\sum\limits_{j = 1}^m {{z_j}}.$

Verbally, if a number does not depend on the summation index, it can be pulled out of the summation sign.

Using Table 5.5 and probabilities from Table 5.4 we have

$latex EXY = \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^m {{x_i}{y_j}{p_{ij}}} }$ (using independence assumption)

$latex = \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^m {{x_i}{y_j}p_i^X p_j^Y} }$ ($latex a = {x_i}p_i^X$ is a constant for the inner sum)

$latex = \sum\limits_{i = 1}^n {{x_i}p_i^X \sum\limits_{j = 1}^m {{y_j}p_j^Y} } = (EX)(EY).$

Comments. (1) If you don't understand this, take $latex n = m = 2$, write out everything without summation signs and then write out equivalents with summation signs.

(2) Multiplicativity is not a linear property. All operators in math that are linear may have nonlinear properties only under special conditions. Independence is such a special condition. Recapitulating, $latex E(aX + bY) = aEX + bEY$ is true for ANY $latex X,Y$ and $latex EXY = (EX)(EY)$ holds ONLY under independence.

(3) The complete notation for the fact that $latex X$ takes the value $latex {x_i}$ with probability $latex p_i^X$ is $latex P(X = {x_i}) = p_i^X$. Similarly, the complete expression for the probabilities from Table 5.4 is $latex P(X = {x_i},\,Y = {y_j}) = p_{ij}^{}$. This formalism is necessary to understand some equations in NCT.

(4) Two other characteristics of random variables, variance and covariance, are defined in terms of means. Therefore we will use linearity of means very often.

Previous: Linearity | Next: Covariance

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.