Properties of covariance

Nov 16

category: AP Stats and Business Stats, Econometrics, ST104A, ST104B, Statistics 1, Statistics 2, The College Board 12 Comments

Wikipedia says: The magnitude of the covariance is not easy to interpret. I add: We keep the covariance around mainly for its algebraic properties. It deserves studying because it appears in two important formulas: correlation coefficient and slope estimator in simple regression (see derivation, simplified derivation and proof of unbiasedness).

Definition. For two random variables $X,Y$ their covariance is defined by

$Cov (X,Y) = E(X - EX)(Y - EY)$

(it's the mean value of the product of the deviations of two variables from their respective means).

Properties of covariance

Property 1. Linearity. Covariance is linear in the first argument when the second argument is fixed: for any random variables $X,Y,Z$ and numbers $a,b$ one has
(1) $Cov (aX + bY,Z) = aCov(X,Z) + bCov (Y,Z).$
Proof. We start by writing out the left side of Equation (1):
$Cov(aX + bY,Z)=E[(aX + bY)-E(aX + bY)](Z-EZ)$
(using linearity of means)
$= E(aX + bY - aEX - bEY)(Z - EZ)$
(collecting similar terms)
$= E[a(X - EX) + b(Y - EY)](Z - EZ)$
(distributing $(Z - EZ)$ )
$= E[a(X - EX)(Z - EZ) + b(Y - EY)(Z - EZ)]$
(using linearity of means)
$= aE(X - EX)(Z - EZ) + bE(Y - EY)(Z - EZ)$
$= aCov(X,Z) + bCov(Y,Z).$

Exercise. Covariance is also linear in the second argument when the first argument is fixed. Write out and prove this property. You can notice the importance of using parentheses and brackets.

Property 2. Shortcut for covariance: $Cov(X,Y) = EXY - (EX)(EY)$ .
Proof. $Cov(X,Y)= E(X - EX)(Y - EY)$
(multiplying out)
$= E[XY - X(EY) - (EX)Y + (EX)(EY)]$
( $EX,EY$ are constants; use linearity)
$=EXY-(EX)(EY)-(EX)(EY)+(EX)(EY)=EXY-(EX)(EY).$

Definition. Random variables $X,Y$ are called uncorrelated if $Cov(X,Y) = 0$ .

Uncorrelatedness is close to independence, so the intuition is the same: one variable does not influence the other. You can also say that there is no statistical relationship between uncorrelated variables. The mathematical side is not the same: uncorrelatedness is a more general property than independence.

Property 3. Independent variables are uncorrelated: if $X,Y$ are independent, then $Cov(X,Y) = 0$ .
Proof. By the shortcut for covariance and multiplicativity of means for independent variables we have $Cov(X,Y) = EXY - (EX)(EY) = 0$ .