Raising the bar

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.9. Correlation coefficient

Correlation coefficient. Suppose random variables $latex X,Y$ are not constant. Then their standard deviations are not zero and we can define

$latex \rho (X,Y) = \frac{{{\mathop{\rm cov}} (X,Y)}}{{\sigma (X)\sigma (Y)}}$

which is called a correlation coefficient of $latex X$ and $latex Y$.

Property 1. Range of the correlation coefficient: for any $latex X,Y$ one has $latex -1 \le \rho (X,Y) \le 1$.

Proof. By the properties of absolute values the Cauchy-Schwarz inequality can be written as $latex -\sigma (X)\sigma (Y) \le {\mathop{\rm cov}} (X,Y) \le \sigma (X)\sigma (Y).$ It remains to divide this inequality by $latex \sigma (X)\sigma (Y)$.

Property 2. Interpretation of extreme cases. (1) If $latex \rho (X,Y) = 1$, then $latex Y = aX + b$ with $latex a > 0.$

(2) If $latex \rho (X,Y) = -1$, then $latex Y = aX + b$ with $latex a < 0$.

Proof. (1) $latex \rho (X,Y) = 1$ implies

$latex {\mathop{\rm cov}} (X,Y) = \sigma (X)\sigma (Y) \qquad (\textrm{Eq. 5.8})$

which, in turn, implies that $latex Y$ is a linear function of $latex X$: $latex Y = aX + b$. Further, we can establish the sign of the number $latex a$. By the properties of variance and covariance

$latex {\mathop{\rm cov}} (X,Y) = {\mathop{\rm cov}} (X,aX + b) = a{\mathop{\rm cov}} (X,X) + {\mathop{\rm cov}} (X,b) = aV(X),$

$latex \sigma (Y) = \sigma (aX + b) = \sigma (aX) = |a|\sigma (X).$

Plugging this in Eq. (5.8) we get $latex aV(X) = |a|{\sigma ^2}(X)$ and see that $latex a$ is positive.

The proof of (2) is left as an exercise.

Property 3. Suppose we want to measure correlation between weight $latex W$ and height $latex H$ of people. The measurements are either in kilos and centimeters $latex {W_k},{H_c}$ or in pounds and feet $latex {W_p},{H_f}$. The correlation coefficient is unit-free in the sense that it does not depend on the units used: $latex \rho ({W_k},{H_c}) = \rho ({W_p},{H_f})$.

Proof. One measurement is proportional to another, $latex {W_k} = a{W_p},\,\,{H_c} = b{H_f}$ with some positive constants $latex a,b$. By homogeneity

$latex \rho ({W_k},{H_c}) = \frac{{{\mathop{\rm cov}} ({W_k},{H_c})}}{{\sigma ({W_k})\sigma ({H_c})}} = \frac{{{\mathop{\rm cov}} (a{W_p},b{H_f})}}{{\sigma (a{W_p})\sigma (b{H_f})}} = \frac{{ab{\mathop{\rm cov}} ({W_p},{H_f})}}{{ab\sigma ({W_p})\sigma ({H_f})}} = \rho ({W_p},{H_f}).$

Correlation as cosine. Because the range of $latex \rho (X,Y)$ is $latex [-1,1]$, we can define the angle $latex {\alpha _{X,Y}}$ between $latex X$ and $latex Y$ by $latex \cos {\alpha _{X,Y}} = \rho (X,Y),\,\,0 \le {\alpha _{X,Y}} \le \pi .$

Table 5.6. Geometric interpretation of the correlation coefficient.

Previous: Standard deviation | Next: Standardization

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.