3
Nov 16

Properties of covariance

Wikipedia says: The magnitude of the covariance is not easy to interpret. I add: We keep the covariance around mainly for its algebraic properties. It deserves studying because it appears in two important formulas: correlation coefficient and slope estimator in simple regression (see derivation, simplified derivation and proof of unbiasedness).

Definition. For two random variables X,Y their covariance is defined by

Cov (X,Y) = E(X - EX)(Y - EY)

(it's the mean value of the product of the deviations of two variables from their respective means).

Properties of covariance

Property 1. Linearity. Covariance is linear in the first argument when the second argument is fixed: for any random variables X,Y,Z and numbers a,b one has
(1) Cov (aX + bY,Z) = aCov(X,Z) + bCov (Y,Z).
Proof. We start by writing out the left side of Equation (1):
Cov(aX + bY,Z)=E[(aX + bY)-E(aX + bY)](Z-EZ)
(using linearity of means)
= E(aX + bY - aEX - bEY)(Z - EZ)
(collecting similar terms)
= E[a(X - EX) + b(Y - EY)](Z - EZ)
(distributing (Z - EZ))
= E[a(X - EX)(Z - EZ) + b(Y - EY)(Z - EZ)]
(using linearity of means)
= aE(X - EX)(Z - EZ) + bE(Y - EY)(Z - EZ)
= aCov(X,Z) + bCov(Y,Z).

Exercise. Covariance is also linear in the second argument when the first argument is fixed. Write out and prove this property. You can notice the importance of using parentheses and brackets.

Property 2. Shortcut for covariance: Cov(X,Y) = EXY - (EX)(EY).
ProofCov(X,Y)= E(X - EX)(Y - EY)
(multiplying out)
= E[XY - X(EY) - (EX)Y + (EX)(EY)]
(EX,EY are constants; use linearity)
=EXY-(EX)(EY)-(EX)(EY)+(EX)(EY)=EXY-(EX)(EY).

Definition. Random variables X,Y are called uncorrelated if Cov(X,Y) = 0.

Uncorrelatedness is close to independence, so the intuition is the same: one variable does not influence the other. You can also say that there is no statistical relationship between uncorrelated variables. The mathematical side is not the same: uncorrelatedness is a more general property than independence.

Property 3. Independent variables are uncorrelated: if X,Y are independent, then Cov(X,Y) = 0.
Proof. By the shortcut for covariance and multiplicativity of means for independent variables we have Cov(X,Y) = EXY - (EX)(EY) = 0.

Property 4. Correlation with a constant. Any random variable is uncorrelated with any constant: Cov(X,c) = E(X - EX)(c - Ec) = 0.

Property 5. Symmetry. Covariance is a symmetric function of its arguments: Cov(X,Y)=Cov(Y,X). This is obvious.

Property 6. Relationship between covariance and variance:

Cov(X,X)=E(X-EX)(X-EX)=Var(X).

12 Responses for "Properties of covariance"

  1. […] any random variables and numbers one has (1) The term in (1) is called an interaction term. See this post for the definition and properties of covariance. […]

  2. […] this post for the definition and properties of covariance. As one can see, it can be studied right after the […]

  3. […] of means, covariances and variances are bread and butter of professionals. Here we consider the bread - the […]

  4. […] Property 2. Independent variables are uncorrelated: . This follows immediately from multiplicativity and the shortcut for covariance: […]

  5. […] 3. The sample covariance unbiasedly estimates the population covariance: . Its analog: the sample covariance converges to a spike at the population […]

  6. […] directly to (1). Do separate in (1) what is supposed to be . For this, plug (3) in (1) and use linearity of covariance with respect to each argument when the other argument is […]

  7. […] analogy, which students love. Fourthly, this topic paves the road to properties of the variance and covariance (recall that the slope in simple regression is covariance over […]

  8. […] covariances depend only on the distance in time between two moments […]

  9. […] covariance, standard deviation and correlation: their definitions and properties are deeply rooted in the […]

  10. […] Probably, this change was caused by the fact that students were not familiar with properties of covariances and variances. Dougherty has a review chapter for a […]

  11. […] and bad (all other) processes. How do you define the good ones? (Hint: Properties of means, covariances and variances are bread and butter of […]

  12. […] is good at them and 2) they clog the picture. For this reason alone it is worth using variances and covariances. For example, if are observations, let us call by variance the quantity  where  is the sample […]

Leave a Reply

You must be logged in to post a comment.