Raising the bar

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.8. Standard deviation

Standard deviation. For a random variable $latex X$, the quantity $latex \sigma (X) = \sqrt {V(X)}$ is called its standard deviation.

In general, there are two square roots of a positive number, one positive and the other negative. The positive one is called an arithmetic square root. An arithmetic root is applied here to $latex V(X) \ge 0$, so standard deviation is always nonnegative.

Absolute values. An absolute value of a real number $latex a$ is defined by

$latex |a| = a,\ \textrm{if } a \textrm{ is nonnegative}, |a|=-a\ \textrm{if } a \textrm{ is negative}. \qquad (\textrm{Eq. 5.7})$

It is introduced to measure the distance from point $latex a$ to the origin. For example, $latex \textrm{dist}(3,0) = |3| = 3$ and $latex \textrm{dist}(-3,0) = |-3| = 3$. More generally, for any points $latex a,b$ on the real line the distance between them is given by $latex \textrm{dist}(a,b) = |a - b|$.

The distance interpretation of the absolute value makes it clear that

1. the condition $latex a \in [-b,b]$ is equivalent to $latex |a| = \textrm{dist}(a,0) \le b$ and
2. the condition $latex a \notin [-b,b]$ is equivalent to $latex |a| = \textrm{dist}(a,0) > b$.

By squaring both sides in Eq. (5.7) we obtain $latex |a|^2 = {a^2}$. Application of the arithmetic square root gives $latex |a| = \sqrt {a^2}$. This is the equation we need right now.

Property 1. Standard deviation is homogeneous of degree 1:

$latex \sigma (aX) = \sqrt {V(aX)} = \sqrt {{a^2}V(X)} = |a|\sigma (X).$

Property 2. Cauchy-Schwarz inequality. (1) For any random variables $latex X,Y$ one has $latex |{\mathop{\rm cov}} (X,Y)| \le \sigma (X)\sigma (Y)$.

(2) If the inequality sign turns into equality, $latex |{\mathop{\rm cov}} (X,Y)| = \sigma (X)\sigma (Y)$, then $latex Y$ is a linear function of $latex X$: $latex Y = aX + b$.

Proof. (1) If at least one of the variables is constant, both sides of the inequality are 0 and there is nothing to prove. To exclude the trivial case, let $latex X,Y$ be nonconstant and, therefore, $latex V(X),\,\,V(Y)$ are positive. Consider a real-valued function of a real number $latex t$ defined by $latex f(t) = V(tX + Y)$. Here we have variance of a linear combination $latex f(t) = {t^2}V(X) + 2t{\mathop{\rm cov}} (X,Y) + V(Y)$.

We see that $latex f(t)$ is a parabola with branches looking upward (because the senior coefficient $latex V(X)$ is positive). By nonnegativity of variance, $latex f(t) \ge 0$ and the parabola lies above the horizontal axis in the $latex (f,t)$ plane. Hence, the quadratic equation $latex f(t) = 0$ may have at most one real root. This means that the discriminant of the equation is non-positive: $latex D = {[{\mathop{\rm cov}} (X,Y)]^2} - V(X)V(Y) \le 0.$ Applying square roots to both sides of $latex {[{\mathop{\rm cov}} (X,Y)]^2} \le V(X)V(Y)$ we finish the proof of the first part.

(2) In case of the equality sign the discriminant is 0. Therefore the parabola touches the horizontal axis where $latex f(t) = V(tX + Y) = 0$. But we know that this implies $latex tX + Y = c$ which is just another way of writing $latex Y = aX + b$.

Do you think this proof is tricky? During the long history of development of mathematics mathematicians have invented many tricks, small and large. No matter how smart you are, you cannot reinvent all of them. Studying the existing tricks is a necessity. By the way, the definition of what is tricky and what is not is strictly personal and time-dependent.

Previous: Variance | Next: Correlation coefficient

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.7. Variance

Variance. Variance of a variable $latex X$ is defined by $latex V(X) = E{(X - EX)^2}$.

Comments. (1) Variance measures the spread of $latex X$ around its mean.

(2) For linear operations, like the mean $latex EX$, it is common to write the argument without parentheses, unless it is complex, as in $latex E(aX + bY)$. For nonlinear operations, of type $latex V(X)$, the argument must be put in parentheses.

Property 1. Relation to covariance: $latex V(X) = {\mathop{\rm cov}} (X,X)$ (covariance of $latex X$ with itself).

Property 2. Variance of a linear combination. For any random variables $latex X,Y$ and numbers $latex a,b$ one has

$latex V(aX + bY) = {a^2}V(X) + 2ab{\mathop{\rm cov}} (X,Y) + {b^2}V(Y). \qquad (\textrm{Eq. 5.5})$

The term $latex 2ab{\mathop{\rm cov}} (X,Y)$ in Eq. (5.5) is called an interaction term.

Proof. Use Property 1:

$latex V(aX + bY) = {\mathop{\rm cov}} (aX + bY,aX + bY)$ (using linearity with respect to the first variable)

$latex = a{\mathop{\rm cov}} (X,aX + bY) + b{\mathop{\rm cov}} (Y,aX + bY)$ (using linearity with respect to the second variable)

$latex = {a^2}{\mathop{\rm cov}} (X,X) + ab{\mathop{\rm cov}} (X,Y) + ab{\mathop{\rm cov}} (Y,X) + {b^2}{\mathop{\rm cov}} (Y,Y)$ (using covariance symmetry and collecting similar terms)

$latex = {a^2}V(X) + 2ab{\mathop{\rm cov}} (X,Y) + {b^2}V(Y).$

Exercise 5.6.

1. Prove Property 2 directly, without appealing to covariance.
2. How do you obtain from this property two special cases:
        - (a) $latex V(X + Y) = V(X) + 2{\mathop{\rm cov}} (X,Y) + V(Y)$ (variance of a sum) and
        - (b) $latex V(aX) = {a^2}V(X)$ (homogeneity of degree 2)?
3. What do you think is larger: $latex V(X + Y)$ or $latex V(X - Y)$?
4. If we add a constant to a variable, does its variance change?

Property 3. Variance of a constant: $latex V(c) = E{(c - Ec)^2} = 0$.

Property 4. Nonnegativity: $latex V(X) \ge 0$ for any $latex X$.

Proof. By definition,

$latex V(X) = E{(X - EX)^2} = {p_1}{({x_1} - EX)^2} + ... + {p_n}{({x_n} - EX)^2}. \qquad (\textrm{Eq. 5.6})$

This is a weighted sum of squared deviations, each of which is nonnegative, and the weights are positive, so the whole thing is nonnegative.

Property 5. Characterization of all variables with vanishing variance. We want to know exactly which variables have zero variance. We know that constants have this property, so $latex X = c$ is sufficient for $latex V(X) = 0$. Conversely, suppose that $latex V(X) = 0$. Since all probabilities are positive, from Eq. (5.6) we see that $latex {x_1} - EX = ... = {x_n} - EX = 0$. This implies that all values are equal to the mean, which is a constant. Thus, $latex X = c$ is necessary for $latex V(X) = 0$. Conclusion: $latex V(X) = 0$ if and only if $latex X = c$.

Property 6. Additivity of variance: if $latex X,Y$ are independent, then $latex V(X + Y) = V(X) + V(Y)$ (the interaction term disappears).

Since variance is a nonlinear operator, it is additive only under special circumstances, when the variables are uncorrelated or, in particular, when they are independent.

Previous: Covariance | Next: Standard deviation

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.6. Covariance

Covariance. For two random variables $latex X,Y$ their covariance is defined by $latex {\mathop{\rm cov}} (X,Y) = E(X - EX)(Y - EY)$.

Comments. (1) Covariance has two complex arguments, whereas its value is simple $latex -$ it's just a number.

(2) $latex X - EX$ is the deviation of $latex X$ from its mean; $latex Y - EY$ is the deviation of $latex Y$ from its mean. Therefore $latex {\mathop{\rm cov}} (X,Y)$ is the expected value of the product of these two deviations. Explain all formulas for yourself in this way.

(3) NCT says that covariance is a measure of relationship between two variables. This is true with some caveats. We keep covariance around mainly for its algebraic properties.

Property 1. Linearity. Covariance is linear in the first argument when the second argument is fixed: for any random variables $latex X,Y,Z$ and numbers $latex a,b$ one has

$latex {\mathop{\rm cov}} (aX + bY,Z) = a{\mathop{\rm cov}} (X,Z) + b{\mathop{\rm cov}} (Y,Z). \qquad (\textrm{Eq. 5.4})$

Proof. We start by writing out the left side of Eq. (5.4):

$latex {\mathop{\rm cov}} (aX + bY,Z) = E[(aX + bY) - E(aX + bY)](Z - EZ)$ (using linearity of means)

$latex = E(aX + bY - aEX - bEY)(Z - EZ)$ (collecting similar terms)

$latex = E[a(X - EX) + b(Y - EY)](Z - EZ)$ (multiplying out)

$latex = E[a(X - EX)(Z - EZ) + b(Y - EY)(Z - EZ)]$ (using linearity of means)

$latex = aE(X - EX)(Z - EZ) + bE(Y - EY)(Z - EZ)$ (by definition of covariance)

$latex = a{\mathop{\rm cov}} (X,Z) + b{\mathop{\rm cov}} (Y,Z).$

Exercise 5.5. Covariance is also linear in the second argument when the first argument is fixed. Write out and prove this property. You can notice the importance of using parentheses and brackets.

Learning longer proofs will develop your logic and intuition to the level that you'll be able to produce results like this on demand.

Property 2. Shortcut for covariance: $latex {\mathop{\rm cov}} (X,Y) = EXY - (EX)(EY)$.

Proof.

$latex {\mathop{\rm cov}} (X,Y) = E(X - EX)(Y - EY)$ (multiplying out)

$latex = E[XY - X(EY) - (EX)Y + (EX)(EY)]$ ($latex EX,EY$ are constants; use linearity)

$latex = EXY - (EX)(EY) - (EX)(EY) + (EX)(EY)$

$latex = EXY - (EX)(EY).$

Uncorrelated random variables. Random variables $latex X,Y$ are uncorrelated if $latex {\mathop{\rm cov}} (X,Y) = 0$.

Uncorrelatedness is close to independence, so the intuition is the same: one variable does not influence the other. You can also say that there is no statistical relationship between uncorrelated variables. The mathematical side is not the same: uncorrelatedness is a more general property than independence.

Property 3. Independent variables are uncorrelated: if $latex X,Y$ are independent, then $latex {\mathop{\rm cov}} (X,Y) = 0$.

Proof. By the shortcut for covariance and multiplicativity of means for independent variables $latex {\mathop{\rm cov}} (X,Y) = EXY - (EX)(EY) = 0$.

Comments. (1) This is a good place to discuss necessary and sufficient conditions. We say that condition A is sufficient for condition B if A implies B: A $latex \Rightarrow$ B. When this is true, we also say that B is necessary for A. Visually, the set of all objects satisfying A is a subset of all objects satisfying B when A is sufficient for B. For example, the statement on relationship between independence and uncorrelatedness is visualized in Figure 5.3.

Figure 5.3. Independent versus uncorrelated pairs of variables.

(2) When A is both necessary and sufficient for B, we say that A is equivalent to B and write A $latex \Leftrightarrow$ B. An interesting question is whether independence is equivalent to uncorrelatedness or, put it differently, if there exist uncorrelated pairs which are not independent. The answer is Yes. Take any variable $latex X$ such that $latex EX = E{X^3} = 0$ and put $latex Y = {X^2}$. This $latex Y$ is not independent of $latex X$ because knowledge of the latter implies knowledge of the former. However, they are uncorrelated: $latex {\mathop{\rm cov}} (X,Y) = EXY - (EX)(EY) = E{X^3} - (EX)(EY) = 0.$

(3) Here we show that the condition $latex EX = E{X^3} = 0$ from the previous paragraph is satisfied for any random variable that is symmetric around zero. The symmetry definition implies that any negative value $latex x_i^-$ can be written as $latex x_i^- = -x_i^+$ where $latex x_i^+$ is a positive value of the same variable and the probabilities are the same: $latex P(X = x_i^-) = P(X = x_i^+)$. Therefore

$latex EX = \sum {{x_i}{p_i}}$ (sorting out positive and negative values)

$latex = \sum\limits_{x_i^- < 0} {x_i^- {p_i}} + \sum\limits_{x_i^+ > 0} {x_i^+ {p_i}}$ (replacing negative values and their probabilities)

$latex = \sum\limits_{x_i^+ > 0} {[(-x_i^+ ) + x_i^+]{p_i}}$ (the two terms cancel out)

$latex = 0.$

The proof of $latex E{X^3} = 0$ is similar.

Property 4. Correlation with a constant. Any random variable is uncorrelated with any constant: $latex {\mathop{\rm cov}} (X,c) = E(X - EX)(c - Ec) = 0.$

Property 5. Symmetry. Covariance is a symmetric function of its arguments: $latex {\mathop{\rm cov}} (X,Y) = {\mathop{\rm cov}} (Y,X)$.

Previous: Independence and its consequences | Next: Variance

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.5. Independence and its consequences

Background. You have to recall the definition of independence of two events $latex P(A \cap B) = P(A)P(B)$ and realize that a random variable is a much more complex object than an event. Visually, a random variable is a 1-D table with values and probabilities. A pair of random variables is represented by a 2-D table similar to Table 4.9 (see Conditional probabilities). Since we want to work with variables whose probability distributions are not necessarily the same, a table of type Table 5.2 (see Linear combinations of random variables) with the same probabilities for $latex X$ and $latex Y$ is insufficient. We deal with two random variables separately represented by the tables 5.3.

Table 5.3. Separate probability tables.

In general, $latex n$ is different from $latex m$. The probabilities are also different, that's why they are provided with superscripts. If you remember, in this situation we cannot directly define the sum $latex X + Y$ and the product $latex XY$. This is why we need to work with the pair $latex (X,Y)$ which has values $latex ({x_i},{y_j}),\,\,i = 1,...,n,\,\,j = 1,...,m$ (you can think about throwing a coin and die at the same time; any value on the coin can occur in combination with any value on the die). The probabilities of these values are denoted $latex {p_{ij}},\,\,i = 1,...,n,\,\,j = 1,...,m$. Now we arrange a 2-D table putting a pair of values $latex ({x_i},{y_j})$ and corresponding probability $latex {p_{ij}}$ in the same cell of Table 5.4.

Exercise 5.2. One and the same random variable can have many different (equivalent) realizations. The variable $latex X$ is realized in Table 5.3 on the sample space $latex { {x_1},...,{x_n}}$. What is its realization in terms of Table 5.4 (what are the sample space, probability distribution and values)?

Table 5.4. Joint values and probabilities.

Independence of variables

Intuitive definition. We say that $latex X$ and $latex Y$ are independent if what happens to $latex X$ does not influence $latex Y$ in any way.

Formal definition. We say that $latex X$ and $latex Y$ are independent if the events independence condition is satisfied for every pair of their values:

$latex {p_{ij}} = p_i^X p_j^Y,\,\,i = 1,...,n,\,\,j = 1,...,m. \qquad (\textrm{Eq. 5.3})$

Comments. (1) This is an opportune moment to discuss the relation between universal statements and existence statements. Consider a universal statement: all students in a given class are younger than 30. Its opposite is: there are students in a given class who are at least 30 years old, which is an existence statement. This is a general rule of logic: rejecting a universal statement you obtain an existence one, and vice versa.

(2) Rejecting the definition of independence, we obtain the definition of dependence, and it should be an existence statement. Thus, we say that $latex X$ and $latex Y$ are dependent if at least one of the equations Eq. (5.3) is not true.

Sum and product of random variables. The sum $latex X + Y$ and the product $latex XY$ are defined by Table 5.5.

Table 5.5. General definitions of the sum and product.

Exercise 5.3. How is this definition related to the earlier one?

Exercise 5.4. Prove multiplicativity of expected values: If $latex X,\,Y$ are independent, then their mean is multiplicative: $latex E(XY) = (EX)(EY)$.

Proof. Earlier you were asked to rewrite the proof of linearity of means using summation signs. Here we need a special case of that property:

$latex \sum\limits_{j = 1}^m {a{z_j} = } a\sum\limits_{j = 1}^m {{z_j}}.$

Verbally, if a number does not depend on the summation index, it can be pulled out of the summation sign.

Using Table 5.5 and probabilities from Table 5.4 we have

$latex EXY = \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^m {{x_i}{y_j}{p_{ij}}} }$ (using independence assumption)

$latex = \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^m {{x_i}{y_j}p_i^X p_j^Y} }$ ($latex a = {x_i}p_i^X$ is a constant for the inner sum)

$latex = \sum\limits_{i = 1}^n {{x_i}p_i^X \sum\limits_{j = 1}^m {{y_j}p_j^Y} } = (EX)(EY).$

Comments. (1) If you don't understand this, take $latex n = m = 2$, write out everything without summation signs and then write out equivalents with summation signs.

(2) Multiplicativity is not a linear property. All operators in math that are linear may have nonlinear properties only under special conditions. Independence is such a special condition. Recapitulating, $latex E(aX + bY) = aEX + bEY$ is true for ANY $latex X,Y$ and $latex EXY = (EX)(EY)$ holds ONLY under independence.

(3) The complete notation for the fact that $latex X$ takes the value $latex {x_i}$ with probability $latex p_i^X$ is $latex P(X = {x_i}) = p_i^X$. Similarly, the complete expression for the probabilities from Table 5.4 is $latex P(X = {x_i},\,Y = {y_j}) = p_{ij}^{}$. This formalism is necessary to understand some equations in NCT.

(4) Two other characteristics of random variables, variance and covariance, are defined in terms of means. Therefore we will use linearity of means very often.

Previous: Linearity | Next: Covariance

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.4. Linearity

Linearity is one of those many properties in which both the statement of the problem and its solution are mathematical.

Problem statement. What is the relation between $latex E(aX + bY)$ and $latex EX,\,EY$?

Answer. For any random variables $latex X,\,Y$ and any numbers $latex a,\,b$ one has $latex E(aX + bY) = aEX + bEY$.

Proof. This is one of those straightforward proofs when knowing the definitions and starting with the left-hand side is enough to arrive at the result. Using the definitions in Table 5.2 (see Linear combinations of random variables) the mean of the linear combination is

$latex E(aX + bY) = (a{x_1} + b{y_1}){p_1} + ... + (a{x_n} + b{y_n}){p_n}$ (distributing probabilities)

$latex = a{x_1}{p_1} + b{y_1}{p_1} + ... + a{x_n}{p_n} + b{y_n}{p_n}$ (grouping by variables)

$latex = (a{x_1}{p_1} + ... + a{x_n}{p_n}) + (b{y_1}{p_1} + ... + b{y_n}{p_n})$ (pulling out constants)

$latex = a({x_1}{p_1} + ... + {x_n}{p_n}) + b({y_1}{p_1} + ... + {y_n}{p_n})$

$latex = aEX + bEY.$

Chewing and digesting

Application. A portfolio contains $latex {n_1}$ shares of stock 1 whose price is $latex {S_1}$ and $latex {n_2}$ shares of stock 2 whose price is $latex {S_2}$. Stock prices fluctuate and are random variables. Numbers of shares are assumed fixed and are deterministic. What is the expected value of the portfolio?

Answer. The portfolio value is its market price $latex V = {n_1}{S_1} + {n_2}{S_2}$. Since this is a linear combination, the expected value is $latex EV = {n_1}E{S_1} + {n_2}E{S_2}$.

Going sideways. (1) Rewrite the proof of linearity of means using summation signs.

(2) Do you think $latex E({X^2})$ is the same as $latex {(EX)^2}$? Hint: It should be clear that $latex {X^2} = (x_1^2,...,x_n^2)$ and $latex E{X^2} = x_1^2{p_1} + ... + x_n^2{p_n}$, see Linear combinations of random variables.

(3) While I don't give an answer for the previous question, I want to use it to discuss the difference between universal statements (which should be true for a class of objects) and existence statements (which assert existence of objects with special properties). If you say: Yes, I think that $latex E({X^2}) = {(EX)^2}$, it should be a universal statement: for ANY random variable $latex X$ one has $latex E({X^2}) = {(EX)^2}$. If, on the other hand, you say: No, I don't think so, then it should be an existence statement: I am producing $latex X$ such that $latex E({X^2}) \ne {(EX)^2}$ (whereas for other variables the equation may be true). Pay attention to the order of words when you use "for every" and "there exists". In the next two propositions the words are the same, just their order is different:

• (i) In the nursery school X, for every kid K there is a toy T such that K plays with T (T depends on K).
• (ii) In the nursery school X, there is a toy T such that every kid K plays with T (one T for all kids).

Going forward. (1) Use induction to prove the generalization to the case of a linear combination of $latex n$ variables: $latex E({a_1}{X_1} + ... + {a_n}{X_n}) = {a_1}E{X_1} + ... + {a_n}E{X_n}$.

(2) Expected value of a constant. The mean of a constant is that constant, $latex Ec = c{p_1} + ... + c{p_n} = c({p_1} + ... + {p_n}) = c$, because a constant doesn't change, rain or shine.

Homogeneity. Letting in the linearity property $latex b = 0$ we get $latex E(aX) = aEX$. This property is called homogeneity of degree 1 (you can pull the constant out of the expected value sign).

Additivity. Letting $latex a = b = 1$ in the linearity property we get $latex E(X + Y) = EX + EY$. This is called additivity.

Can you deduce linearity from homogeneity and additivity?

Previous: Linear combinations of random variables | Next: Independence and its consequences

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.3. Linear combinations of random variables

Linear combination. Suppose that $latex X,Y$ are two discrete random variables with the same probability distribution $latex {p_1},...,{p_n}$. Let $latex a,b$ be real numbers. The random variable $latex aX + bY$ is called a linear combination of $latex X,Y$ with coefficients $latex a,b$. Its special cases are $latex aX$ ($latex X$ scaled by $latex a$) and $latex X + Y$ (a sum of $latex X$ and $latex Y$). The detailed definition is given by Table 5.2.

Table 5.2. Linear operations with random variables.

Comments. (1) There is only one column of probabilities because $latex X$ and $latex Y$ are assumed to have the same probability distribution. The variables in the last three columns have the same probability distribution by definition. The situation when the probability distributions are different is handled using joint probabilities (see Independence and its consequences).

(2) If you are asked what are vectors, you say the following. By a vector we call an $latex n$-tuple of real numbers $latex X = ({x_1},...,{x_n})$. Operations with vectors are defined by the last three columns of Table 5.2. When $latex n = 1$, vectors are just numbers and those operations are just usual summation and multiplication. Sometimes we need to multiply two vectors according to $latex XY = ({x_1}{y_1},...,{x_n}{y_n})$ (then what would be $latex {X^2}$?) These formal definitions plus the geometric interpretation considered next is all you need to know about vectors.

(3) Geometric interpretation. We live in the space of three-dimensional vectors. All our intuition coming from day-to-day geometrical experience carries over to the $latex n$-dimensional case. Let, for simplicity, $latex n = 2$. A vector $latex X = ({x_1},{x_2})$ is shown on the plane as an arrow starting at the origin and ending at the point $latex ({x_1},{x_2})$. The sum $latex X + Y = ({x_1} + {y_1},{x_2} + {y_2})$ is found using the parallelogram rule in Figure 5.1. The rule itself comes from physics: if two forces are applied to a point, their resultant force is found by the parallelogram rule.

Figure 5.1. Summing vectors using the parallelogram rule.

Figure 5.2. Scaling a vector.

Scaling $latex X$ by a positive number $latex a$ means lengthening it in case $latex a > 1$ and shortening in case $latex a < 1$. Scaling by a negative number means, additionally, reverting the direction of $latex X$, see Figure 5.2. To obtain a linear combination $latex aX + bY$ you first scale $latex X$ and $latex Y$ and then add the results.

Previous: General properties of means | Next: Linearity

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.2. General properties of means

Exercise 5.1. With probability 0.001 a strong earthquake destroys your house this year and then your loss is $latex \$200,000$. Otherwise, your loss is just normal wear and tear, estimated at $latex \$4,000$ per year. What is your expected loss?

Sometimes I ask questions before giving formal definitions, to prompt you to guess. Guessing and inductive reasoning are important parts of our thinking process. In economics there is a French term "tâtonnement" (groping one's way in the dark). There are cases when you clearly see the idea and the solution is straightforward. In more complex situations there may be several competing ideas, and some of them may not be clear, looking like unfamiliar creatures in the dark. Looking for them and nurturing them until they shape up so that you can test them is a great skill. This is where you need to collect all the relevant information, mull it over again and again, until it becomes your second nature. This process is more time-consuming than getting a ready answer from your professor or classmate but the rewards are also greater.

The expected loss is $latex (-4{,}000) \times 0.999 + (-200{,}000) \times 0.001 = -4{,}196$. The general idea is that it is a weighted sum of the values of the variable of interest (the loss function). We attach a higher importance to the value that has a higher probability.

In research ideas go before definitions.

Expected value of a discrete random variable. Using the working definition of a random variable, we define the expected value of $latex X$ (also called a population mean, or an average) to be

$latex EX = {x_1}{p_1} + ... + {x_n}{p_n} = \sum\limits_{i = 1}^n {{x_i}{p_i}}. \qquad (\textrm{Eq. 5.1})$

Comments. (1) Expected value is a function whose argument is a complex object (it is described by Table 5.1, see Random variable) and the value is simple: $latex EX$ is just a number. And it is not a product of $latex E$ and $latex X$!

(2) Relation to the sample mean. A discrete random variable is said to be uniformly distributed if $latex {p_1} = ... = {p_n} = 1/n$. In this case the expected value is just the sample mean:

$latex EX = {x_1}\frac{1}{n} + ... + {x_n}\frac{1}{n} = \frac{{{x_1} + ... + {x_n}}}{n} = \bar X.$

(3) Grouped data formula. Recall the procedure for finding relative frequencies. Denote $latex {y_1},...,{y_n}$ the values in the sample. Equal values are joined in groups. Let $latex {x_1},...,{x_m}$ denote the distinct values and $latex {n_1},...,{n_m}$ their absolute frequencies. Their total is, clearly, $latex n$. The sample mean is

$latex \bar Y = ({y_1} + ... + {y_n})/n \qquad (\textrm{Eq. 5.2})$

(sorting out $latex y$'s into groups with equal values)

$latex = \left( \overbrace{ {x_1} + ... + {x_1} }^{{n_1}\textrm{ times}} + ... + \overbrace{ {x_m} + ... + {x_m} }^{{n_m}\textrm{ times}} \right)/n$

$latex = ({n_1}{x_1} + ... + {n_m}{x_m})/n$ (this is the grouped data formula)

(dividing through by $latex n$ and using relative frequencies)

$latex = {r_1}{x_1} + ... + {r_m}{x_m} = EX.$

In this context, the sample mean $latex \bar Y$, the grouped data formula (in the fourth line) and the expected value are the same thing. The weighted mean formula from NCT is the same thing as the grouped data formula with $latex {n_i} = {w_i}$ and $latex n = \sum {{w_i}}$. Note also that when data are grouped by intervals, in the grouped data formula the midpoints $latex {m_i}$ of the intervals are taken as the values of the variables.

(4) When the sample size becomes very large, the relative frequencies approach the true probabilities by the Law of Large Numbers. Therefore Eq. (5.2) gives rise to the following intuitive interpretation of the expected value: $latex EX$ is what $latex \bar X$ approaches when $latex n$ is very large.

Previous: Random variable | Next: Linear combinations of random variables

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.1. Random variable

At some point you may feel that the amount of new information is overwhelming and you forget many facts. Then it's the time to arrange a global repetition. Repeat everything with proofs. Look for those minor or significant details you missed earlier. Make a list of facts in a logical order. This chapter will be more challenging than all the previous taken together, and you will need to be in a perfect shape to conquer it.

Intuitive definition. A random variable is a variable whose values cannot be predicted.

For example, we can associate random variables with the coin and die. On the other hand, to know the value of $latex \sin x$ it suffices to plug the argument $latex x$ in $latex \sin$. $latex \sin x$ or the number of days in a year are not random variables.

Short formal definition. A random variable is a pair: values plus probabilities.

Working definition of a discrete random variable. A discrete random variable is defined by Table 5.1.

Table 5.1. Discrete random variable with $latex n$ values.

Comments. (1) Most of the time, when people talk about random variables, they assume real-valued variables. Thus, in the first column we have real numbers. This is important to be able to use arithmetic operations.

(2) Capital $latex X$ is used for the variable and lower-case $latex x$'s for its values. The complete form of writing $latex {p_i}$ is $latex P(X = {x_i})$.

(3) Without the second column we would have a deterministic variable.

(4) In the table, $latex X$ takes a finite number of values. This is a simplification. In Poisson distribution we'll see an example of a discrete random variable that takes an infinite number of values.

Previous: Questions for repetition (Chapter 4) | Next: General properties of means

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 4. Probability

Unit 4.10. Questions for repetition

1. What do we mean by set operations?
2. Prove de Morgan laws.
3. Prove and illustrate on a Venn diagram equation Eq. (4.2) (see Probability).
4. Prove $latex P(A \cup B) = P(A) + P(B) - P(A \cap B)$.
5. Prove the formulas for orderings and combinations.
6. Using a simple example, in one block give all definitions and identities related to joint and marginal probabilities.
7. Solve Exercise 4.7 (see Conditional probabilities).
8. Derive the complex form of the Bayes theorem.
9. What can you say about event $latex A$ if $latex P(A) = {P^2}(A)$?
10. If $latex A$ and $latex B$ are independent and $latex B$ and $latex C$ are independent, then $latex A$ and $latex C$ are also independent (transitivity). True or false?
11. There are only two events that are independent of themselves: the empty event and the sample space. True or false?
12. If $latex A$ and $latex B$ are independent and both have positive probability, then they cannot be mutually exclusive. True or false?
13. Any event is independent of its complement. True or false?
14. If $latex A,B$ cover the whole sample space $latex S$, $latex A \cup B = S$, then $latex P(A|C) + P(B|C) \ge 1$.
15. If $latex A$ is independent of $latex B$ and $latex C$, then conditioning $latex A$ on either of $latex B,C$ yields the same result.

Previous: Problem solving strategy | Next: Random variable

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 4. Probability

Unit 4.9. Problem solving strategy

1. Make sure that you solve at least two typical exercises from each section of Chapter 3 in NCT.
2. Some exercises are easier than they look. This is where writing down all the relevant equations is particularly useful. Remember what you need to find. In general, to find $latex n$ unknowns it is sufficient to have $latex n$ equations involving them. One of my students was able to solve those exercises without any knowledge of conditional and joint probabilities, just using common sense and the percentage interpretation of probabilities.
3. To write down the conditions, you need a good notation. Avoid abstract notation and use something that reminds you of what you are dealing with, like $latex A$ for Anthropology class.
4. When looking for a solution of a problem, write down all the relevant theoretical facts. Don't limit yourself to conclusions; repeat everything with proofs and derivations. Omit only what is easy; after a while everything will become easy. If you still don't see the solution, go back by one or two steps and solve simpler similar exercises.
5. When you clearly see the idea, don't calculate the numbers. Learn to save your time by concentrating on ideas.
6. If section exercises are too simple, don't waste your time and go directly to end-of-chapter exercises.
7. Everybody finds difficult exercises that require orderings and combinations. Solve more of them, until you feel comfortable. However, it's not a good idea to solve all of them. Combinatorics is a very narrow area of statistics.

Previous: Conditional probabilities | Next: Questions for repetition (Chapter 4)

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.