Raising the bar

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.6. Covariance

Covariance. For two random variables $latex X,Y$ their covariance is defined by $latex {\mathop{\rm cov}} (X,Y) = E(X - EX)(Y - EY)$.

Comments. (1) Covariance has two complex arguments, whereas its value is simple $latex -$ it's just a number.

(2) $latex X - EX$ is the deviation of $latex X$ from its mean; $latex Y - EY$ is the deviation of $latex Y$ from its mean. Therefore $latex {\mathop{\rm cov}} (X,Y)$ is the expected value of the product of these two deviations. Explain all formulas for yourself in this way.

(3) NCT says that covariance is a measure of relationship between two variables. This is true with some caveats. We keep covariance around mainly for its algebraic properties.

Property 1. Linearity. Covariance is linear in the first argument when the second argument is fixed: for any random variables $latex X,Y,Z$ and numbers $latex a,b$ one has

$latex {\mathop{\rm cov}} (aX + bY,Z) = a{\mathop{\rm cov}} (X,Z) + b{\mathop{\rm cov}} (Y,Z). \qquad (\textrm{Eq. 5.4})$

Proof. We start by writing out the left side of Eq. (5.4):

$latex {\mathop{\rm cov}} (aX + bY,Z) = E[(aX + bY) - E(aX + bY)](Z - EZ)$ (using linearity of means)

$latex = E(aX + bY - aEX - bEY)(Z - EZ)$ (collecting similar terms)

$latex = E[a(X - EX) + b(Y - EY)](Z - EZ)$ (multiplying out)

$latex = E[a(X - EX)(Z - EZ) + b(Y - EY)(Z - EZ)]$ (using linearity of means)

$latex = aE(X - EX)(Z - EZ) + bE(Y - EY)(Z - EZ)$ (by definition of covariance)

$latex = a{\mathop{\rm cov}} (X,Z) + b{\mathop{\rm cov}} (Y,Z).$

Exercise 5.5. Covariance is also linear in the second argument when the first argument is fixed. Write out and prove this property. You can notice the importance of using parentheses and brackets.

Learning longer proofs will develop your logic and intuition to the level that you'll be able to produce results like this on demand.

Property 2. Shortcut for covariance: $latex {\mathop{\rm cov}} (X,Y) = EXY - (EX)(EY)$.

Proof.

$latex {\mathop{\rm cov}} (X,Y) = E(X - EX)(Y - EY)$ (multiplying out)

$latex = E[XY - X(EY) - (EX)Y + (EX)(EY)]$ ($latex EX,EY$ are constants; use linearity)

$latex = EXY - (EX)(EY) - (EX)(EY) + (EX)(EY)$

$latex = EXY - (EX)(EY).$

Uncorrelated random variables. Random variables $latex X,Y$ are uncorrelated if $latex {\mathop{\rm cov}} (X,Y) = 0$.

Uncorrelatedness is close to independence, so the intuition is the same: one variable does not influence the other. You can also say that there is no statistical relationship between uncorrelated variables. The mathematical side is not the same: uncorrelatedness is a more general property than independence.

Property 3. Independent variables are uncorrelated: if $latex X,Y$ are independent, then $latex {\mathop{\rm cov}} (X,Y) = 0$.

Proof. By the shortcut for covariance and multiplicativity of means for independent variables $latex {\mathop{\rm cov}} (X,Y) = EXY - (EX)(EY) = 0$.

Comments. (1) This is a good place to discuss necessary and sufficient conditions. We say that condition A is sufficient for condition B if A implies B: A $latex \Rightarrow$ B. When this is true, we also say that B is necessary for A. Visually, the set of all objects satisfying A is a subset of all objects satisfying B when A is sufficient for B. For example, the statement on relationship between independence and uncorrelatedness is visualized in Figure 5.3.

Figure 5.3. Independent versus uncorrelated pairs of variables.

(2) When A is both necessary and sufficient for B, we say that A is equivalent to B and write A $latex \Leftrightarrow$ B. An interesting question is whether independence is equivalent to uncorrelatedness or, put it differently, if there exist uncorrelated pairs which are not independent. The answer is Yes. Take any variable $latex X$ such that $latex EX = E{X^3} = 0$ and put $latex Y = {X^2}$. This $latex Y$ is not independent of $latex X$ because knowledge of the latter implies knowledge of the former. However, they are uncorrelated: $latex {\mathop{\rm cov}} (X,Y) = EXY - (EX)(EY) = E{X^3} - (EX)(EY) = 0.$

(3) Here we show that the condition $latex EX = E{X^3} = 0$ from the previous paragraph is satisfied for any random variable that is symmetric around zero. The symmetry definition implies that any negative value $latex x_i^-$ can be written as $latex x_i^- = -x_i^+$ where $latex x_i^+$ is a positive value of the same variable and the probabilities are the same: $latex P(X = x_i^-) = P(X = x_i^+)$. Therefore

$latex EX = \sum {{x_i}{p_i}}$ (sorting out positive and negative values)

$latex = \sum\limits_{x_i^- < 0} {x_i^- {p_i}} + \sum\limits_{x_i^+ > 0} {x_i^+ {p_i}}$ (replacing negative values and their probabilities)

$latex = \sum\limits_{x_i^+ > 0} {[(-x_i^+ ) + x_i^+]{p_i}}$ (the two terms cancel out)

$latex = 0.$

The proof of $latex E{X^3} = 0$ is similar.

Property 4. Correlation with a constant. Any random variable is uncorrelated with any constant: $latex {\mathop{\rm cov}} (X,c) = E(X - EX)(c - Ec) = 0.$

Property 5. Symmetry. Covariance is a symmetric function of its arguments: $latex {\mathop{\rm cov}} (X,Y) = {\mathop{\rm cov}} (Y,X)$.

Previous: Independence and its consequences | Next: Variance

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.