Raising the bar

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.7. Variance

Variance. Variance of a variable $latex X$ is defined by $latex V(X) = E{(X - EX)^2}$.

Comments. (1) Variance measures the spread of $latex X$ around its mean.

(2) For linear operations, like the mean $latex EX$, it is common to write the argument without parentheses, unless it is complex, as in $latex E(aX + bY)$. For nonlinear operations, of type $latex V(X)$, the argument must be put in parentheses.

Property 1. Relation to covariance: $latex V(X) = {\mathop{\rm cov}} (X,X)$ (covariance of $latex X$ with itself).

Property 2. Variance of a linear combination. For any random variables $latex X,Y$ and numbers $latex a,b$ one has

$latex V(aX + bY) = {a^2}V(X) + 2ab{\mathop{\rm cov}} (X,Y) + {b^2}V(Y). \qquad (\textrm{Eq. 5.5})$

The term $latex 2ab{\mathop{\rm cov}} (X,Y)$ in Eq. (5.5) is called an interaction term.

Proof. Use Property 1:

$latex V(aX + bY) = {\mathop{\rm cov}} (aX + bY,aX + bY)$ (using linearity with respect to the first variable)

$latex = a{\mathop{\rm cov}} (X,aX + bY) + b{\mathop{\rm cov}} (Y,aX + bY)$ (using linearity with respect to the second variable)

$latex = {a^2}{\mathop{\rm cov}} (X,X) + ab{\mathop{\rm cov}} (X,Y) + ab{\mathop{\rm cov}} (Y,X) + {b^2}{\mathop{\rm cov}} (Y,Y)$ (using covariance symmetry and collecting similar terms)

$latex = {a^2}V(X) + 2ab{\mathop{\rm cov}} (X,Y) + {b^2}V(Y).$

Exercise 5.6.

1. Prove Property 2 directly, without appealing to covariance.
2. How do you obtain from this property two special cases:
        - (a) $latex V(X + Y) = V(X) + 2{\mathop{\rm cov}} (X,Y) + V(Y)$ (variance of a sum) and
        - (b) $latex V(aX) = {a^2}V(X)$ (homogeneity of degree 2)?
3. What do you think is larger: $latex V(X + Y)$ or $latex V(X - Y)$?
4. If we add a constant to a variable, does its variance change?

Property 3. Variance of a constant: $latex V(c) = E{(c - Ec)^2} = 0$.

Property 4. Nonnegativity: $latex V(X) \ge 0$ for any $latex X$.

Proof. By definition,

$latex V(X) = E{(X - EX)^2} = {p_1}{({x_1} - EX)^2} + ... + {p_n}{({x_n} - EX)^2}. \qquad (\textrm{Eq. 5.6})$

This is a weighted sum of squared deviations, each of which is nonnegative, and the weights are positive, so the whole thing is nonnegative.

Property 5. Characterization of all variables with vanishing variance. We want to know exactly which variables have zero variance. We know that constants have this property, so $latex X = c$ is sufficient for $latex V(X) = 0$. Conversely, suppose that $latex V(X) = 0$. Since all probabilities are positive, from Eq. (5.6) we see that $latex {x_1} - EX = ... = {x_n} - EX = 0$. This implies that all values are equal to the mean, which is a constant. Thus, $latex X = c$ is necessary for $latex V(X) = 0$. Conclusion: $latex V(X) = 0$ if and only if $latex X = c$.

Property 6. Additivity of variance: if $latex X,Y$ are independent, then $latex V(X + Y) = V(X) + V(Y)$ (the interaction term disappears).

Since variance is a nonlinear operator, it is additive only under special circumstances, when the variables are uncorrelated or, in particular, when they are independent.

Previous: Covariance | Next: Standard deviation

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.