Raising the bar

All properties of variance in one place

Certainty is the mother of quiet and repose, and uncertainty the cause of variance and contentions. Edward Coke

Preliminaries: study properties of means with proofs.

Definition. Yes, uncertainty leads to variance, and we measure it by . It is useful to use the name deviation from mean for  and realize that , so that the mean of the deviation from mean cannot serve as a measure of variation of  around .

Property 1. Variance of a linear combination. For any random variables and numbers one has
(1)
The term in (1) is called an interaction term. See this post for the definition and properties of covariance.
Proof.

(using linearity of means)

(grouping by variable)

(squaring out)

(using linearity of means and definitions of variance and covariance)

Property 2. Variance of a sum. Letting in (1) we obtain

Property 3. Homogeneity of degree 2. Choose  in (1) to get

Exercise. What do you think is larger: or ?
Property 4. If we add a constant to a variable, its variance does not change: 
Property 5. Variance of a constant is zero: .

Property 6. Nonnegativity. Since the squared deviation from mean is nonnegative, its expectation is nonnegative: .

Property 7. Only a constant can have variance equal to zero: If , then , see the definition of the expected value. Since all probabilities are positive, we conclude that  for all , which means that is identically constant.

Property 8. Shortcut for variance. We have an identity . Indeed, squaring out gives

(distributing expectation)

(expectation of a constant is constant)

.

All of the above properties apply to any random variables. The next one is an exception in the sense that it applies only to uncorrelated variables.

Property 9. If variables are uncorrelated, that is , then from (1) we have  In particular, letting , we get additivity:  Recall that the expected value is always additive.

Generalizations.  and if all are uncorrelated.

Among my posts, where properties of variance are used, I counted 12 so far.

Preliminaries

Review Properties of conditional expectation, especially the summary, where I introduce a new notation for conditional expectation. Everywhere I use the notation  for expectation of conditional on , instead of .

This post and the previous one on conditional expectation show that conditioning is a pretty advanced notion. Many introductory books use the condition  (the expected value of the error term conditional on the regressor is zero). Because of the complexity of conditioning, I think it's better to avoid this kind of assumption as much as possible.

Conditional variance properties

Replacing usual expectations by their conditional counterparts in the definition of variance, we obtain the definition of conditional variance:

(1)

Property 1. If  are independent, then  and  are also independent and conditioning doesn't change variance:

see Conditioning in case of independence.

Property 2. Generalized homogeneity of degree 2: if  is a deterministic function, then  can be pulled out:

Property 3. Shortcut for conditional variance:

(2)

Proof.

(distributing conditional expectation)

(applying Properties 2 and 6 from this Summary with )

Property 4. The law of total variance:

(3) 

Proof. By the shortcut for usual variance and the law of iterated expectations

(replacing from (2))

(the last two terms give the shortcut for variance of )

Before we move further we need to define conditional covariance by

(everywhere usual expectations are replaced by conditional ones). We say that random variables are conditionally uncorrelated if .

Property 5. Conditional variance of a linear combination. For any random variables and functions one has

The proof is quite similar to that in case of usual variances, so we leave it to the reader. In particular, if are conditionally uncorrelated, then the interaction terms disappears:

Wikipedia says: The magnitude of the covariance is not easy to interpret. I add: We keep the covariance around mainly for its algebraic properties. It deserves studying because it appears in two important formulas: correlation coefficient and slope estimator in simple regression (see derivation, simplified derivation and proof of unbiasedness).

Definition. For two random variables their covariance is defined by

(it's the mean value of the product of the deviations of two variables from their respective means).

Properties of covariance

Property 1. Linearity. Covariance is linear in the first argument when the second argument is fixed: for any random variables and numbers one has
(1)
Proof. We start by writing out the left side of Equation (1):

(using linearity of means)

(collecting similar terms)

(distributing )

(using linearity of means)

Exercise. Covariance is also linear in the second argument when the first argument is fixed. Write out and prove this property. You can notice the importance of using parentheses and brackets.

Property 2. Shortcut for covariance: .
Proof. 
(multiplying out)

( are constants; use linearity)

Definition. Random variables are called uncorrelated if .

Uncorrelatedness is close to independence, so the intuition is the same: one variable does not influence the other. You can also say that there is no statistical relationship between uncorrelated variables. The mathematical side is not the same: uncorrelatedness is a more general property than independence.

Property 3. Independent variables are uncorrelated: if are independent, then .
Proof. By the shortcut for covariance and multiplicativity of means for independent variables we have .

Property 4. Correlation with a constant. Any random variable is uncorrelated with any constant:

Property 5. Symmetry. Covariance is a symmetric function of its arguments: . This is obvious.

Property 6. Relationship between covariance and variance:

.

Distribution of the estimator of the error variance

If you are reading the book by Dougherty: this post is about the distribution of the estimator  defined in Chapter 3.

Consider regression

(1)

where the deterministic matrix is of size satisfies (regressors are not collinear) and the error satisfies

(2)

is estimated by Denote Using (1) we see that and the residual is estimated by

(3)

is a projector and has properties which are derived from those of

(4)

If is an eigenvalue of then multiplying by and using the fact that we get Hence eigenvalues of can be only or The equation
tells us that the number of eigenvalues equal to 1 is and the remaining are zeros. Let be the diagonal representation of Here is an orthogonal matrix,

(5)

and is a diagonal matrix with eigenvalues of on the main diagonal. We can assume that the first numbers on the diagonal of are ones and the others are zeros.

Theorem. Let be normal. 1) is distributed as 2) The estimators and are independent.

Proof. 1) We have by (4)

(6)

Denote From (2) and (5)

and is normal as a linear transformation of a normal vector. It follows that where is a standard normal vector with independent standard normal coordinates Hence, (6) implies

(7)

(3) and (7) prove the first statement.

2) First we note that the vectors are independent. Since they are normal, their independence follows from

It's easy to see that This allows us to show that is a function of :

Independence of leads to independence of their functions and

Different faces of vector variance: again visualization helps

In the previous post we defined variance of a column vector  with components by

In terms of elements this is the same as:

(1)

So why knowing the structure of this matrix is so important?

Let be random variables and let be numbers. In the derivation of the variance of the slope estimator for simple regression we have to deal with the expression of type

(2)

Question 1. How do you multiply a sum by a sum? I mean, how do you use summation signs to find the product ?

Answer 1. Whenever you have problems with summation signs, try to do without them. The product

should contain ALL products Again, a matrix visualization will help:

The product we are looking for should contain all elements of this matrix. So the answer is

(3)

Formally, we can write (the sum does not depend on the index of summation, this is another point many students don't understand) and then perform the multiplication in (3).

Question 2. What is the expression for (2) in terms of covariances of components?

Answer 2. If you understand Answer 1 and know the relationship between variances and covariances, it should be clear that

(4)

Question 3. In light of (1), separate variances from covariances in (4).

Answer 3. When we have which are diagonal elements of (1). Otherwise, for we get off-diagonal elements of (1). So the answer is

(5)

Once again, in the first sum on the right we have only variances. In the second sum, the indices are assumed to run from to , excluding the diagonal

Corollary. If are uncorrelated, then the second sum in (5) disappears:

(6)

This fact has been used (with a slightly different explanation) in the derivation of the variance of the slope estimator for simple regression.

Question 4. Note that the matrix (1) is symmetric (elements above the main diagonal equal their mirror siblings below that diagonal). This means that some terms in the second sum on the right of (5) are repeated twice. If you group equal terms in (5), what do you get?

Answer 4. The idea is to write

that is, to join equal elements above and below the main diagonal in (1). For this, you need to figure out how to write a sum of the elements that are above the main diagonal. Make a bigger version of (1) (with more off-diagonal elements) to see that the elements that are above the main diagonal are listed in the sum This sum can also be written as Hence, (5) is the same as

(7)

Unlike (6), this equation is applicable when there is autocorrelation.

Variance of a vector: motivation and visualization

I always show my students the definition of the variance of a vector, and they usually don't pay attention. You need to know what it is, already at the level of simple regression (to understand the derivation of the slope estimator variance), and even more so when you deal with time series. Since I know exactly where students usually stumble, this post is structured as a series of questions and answers.

Think about ideas: how would you define variance of a vector?

Question 1. We know that for a random variable , its variance is defined by

(1)

Now let

be a vector with components, each of which is a random variable. How would you define its variance?

The answer is not straightforward because we don't know how to square a vector. Let denote the transposed vector. There are two ways to multiply a vector by itself: and

Question 2. Find the dimensions of and and their expressions in terms of coordinates of

Answer 2. For a product of matrices there is a compatibility rule that I write in the form

(2)

Recall that in the notation means that the matrix has rows and columns. For example, is of size Verbally, the above rule says that the number of columns of  should be equal to the number of rows of In the product that common number disappears and the unique numbers ( and ) give, respectively, the number of rows and columns of Isn't the the formula
easier to remember than the verbal statement? From (2) we see that is of dimension 1 (it is a scalar) and is an matrix.

For actual multiplication of matrices I use the visualization

(3)

Short formulation. Multiply rows from the first matrix by columns from the second one.

Long Formulation. To find the element of we find a scalar product of the th row of and th column of To find all elements in the th row of we fix the th row in and move right the columns in Alternatively, to find all elements in the th column of we fix the th column in and move down the rows in . Using this rule, we have

(4)

Usually students have problems with the second equation.

Based on (1) and (4), we have two candidates to define variance:

(5)

and

(6)

Answer 1. The second definition contains more information, in the sense to be explained below, so we define variance of a vector by (6).

Question 3. Find the elements of this matrix.

Answer 3. Variance of a vector has variances of its components on the main diagonal and covariances outside it:

(7)

If you can't get this on your own, go back to Answer 2.

There is a matrix operation called trace and denoted . It is defined only for square matrices and gives the sum of diagonal elements of a matrix.

Exercise 1. Show that In this sense definition (6) is more informative than (5).

Exercise 2. Show that if , then (7) becomes

Assumptions about simple regression

We consider the simple regression

(1) 

Here we derived the OLS estimators of the intercept and slope:

(2) ,

(3) .

A1. Existence condition. Since division by zero is not allowed, for (2) to exist we require . If this condition is not satisfied, then there is no variance in  and all observed points are on the vertical line.

A2. Convenience condition. The regressor  is deterministic. This condition is imposed to be able to apply the properties of expectation, see equation (7) in  this post. The time trend and dummy variables are examples of deterministic regressors. However, most real-life regressors are stochastic. Modifying the theory in order to cover stochastic regressors is the subject of two posts: finite-sample theory and large-sample theory.

A3. Unbiasedness condition. . This is the main assumption that makes sure that OLS estimators are unbiased, see equation (7) in  this post.

Unbiasedness is not enough

Unbiasedness characterizes the quality of an estimator, see the intuitive explanation. Unfortunately, unbiasedness is not enough to choose the best estimator because of nonuniqueness: usually, if there is one unbiased estimator of a parameter, then there are infinitely many unbiased estimators of the same parameter. For example, we know that the sample mean unbiasedly estimates the population mean . Since  ( is the first observation), we can easily construct an infinite family of unbiased estimators , assuming . Indeed, using linearity of expectation .

Variance is another measure of an estimator quality: to have a lower spread of estimator values, among competing estimators we choose the one which has the lowest variance. Knowing the estimator variance allows us to find the z-score and use statistical tables.

Slope estimator variance

It is not difficult to find the variance of the slope estimator using representation (6) derived here:

where 

Don't try to apply directly the definition of variance at this point, because there will be a square of a sum, which leads to a double sum upon squaring. We need two new assumptions.

A4. Uncorrelatedness of errors. Assume that  for all  (errors from different equations (1) are uncorrelated). Note that because of the unbiasedness condition, this assumption is equivalent to for all . This assumption is likely to be satisfied if we observe consumption patterns of unrelated individuals.

A5. Homoscedasticity. All errors have the same variances:  for all . Again, because of the unbiasedness condition, this assumption is equivalent to for all .

Now we can derive the variance expression, using properties from this post:

(dropping a constant doesn't affect variance)

(for uncorrelated variables, variance is additive)

(variance is homogeneous of degree 2)

(applying homoscedasticity)

(plugging )

(using the notation of sample variance)

Note that canceling out two variances in the last line is obvious. It is not so obvious for some if instead of the short notation for variances you use summation signs. The case of the intercept variance is left as an exercise.

Conclusion

The above assumptions A1-A5 are called classical. It is necessary to remember their role in derivations because a considerable part of Econometrics is devoted to deviations from classical assumptions. Once you have a certain assumption violated, you should expect the corresponding estimator property invalidated. For example, if , you should expect the estimators to be biased. If any of A4-A5 is not true, the formula we have derived

will not hold. Besides, the Gauss-Markov theorem that the OLS estimators are efficient will not hold (this will be discussed later). The pair A4-A5 can be called an efficiency condition.

Correlation coefficient: the last block of statistical foundation

Correlation has already been mentioned in

Statistical measures and their geometric roots

Properties of standard deviation

The pearls of AP Statistics 35

Properties of covariance

The pearls of AP Statistics 33

The hierarchy of definitions

Suppose random variables are not constant. Then their standard deviations are not zero and we can define their correlation as in Chart 1.

Chart 1. Correlation definition

Properties of correlation

Property 1. Range of the correlation coefficient: for any one has .
This follows from the Cauchy-Schwarz inequality, as explained here.

Recall from this post that correlation is cosine of the angle between and .
Property 2. Interpretation of extreme cases. (Part 1) If , then with

(Part 2) If , then with .

Proof. (Part 1) implies
(1)
which, in turn, implies that is a linear function of : (this is the second part of the Cauchy-Schwarz inequality). Further, we can establish the sign of the number . By the properties of variance and covariance
,

.
Plugging this in Eq. (1) we get and see that is positive.

The proof of Part 2 is left as an exercise.

Property 3. Suppose we want to measure correlation between weight and height of people. The measurements are either in kilos and centimeters or in pounds and feet . The correlation coefficient is unit-free in the sense that it does not depend on the units used: . Mathematically speaking, correlation is homogeneous of degree in both arguments.
Proof. One measurement is proportional to another, with some positive constants . By homogeneity

Properties of standard deviation are divided in two parts. The definitions and consequences are given here. Both variance and standard deviation are used to measure variability of values of a random variable around its mean. Then why use both of them? The why will be explained in another post.

Properties of standard deviation: definitions and consequences

Definition. For a random variable , the quantity is called its standard deviation.

    Digression about square roots and absolute values

In general, there are two square roots of a positive number, one positive and the other negative. The positive one is called an arithmetic square root. The arithmetic root is applied here to (see properties of variance), so standard deviation is always nonnegative.
Definition. An absolute value of a real number is defined by
(1) if is nonnegative and if is negative.
This two-part definition is a stumbling block for many students, so making them plug in a few numbers is a must. It is introduced to measure the distance from point to the origin. For example, and . More generally, for any points on the real line the distance between them is given by .

By squaring both sides in Eq. (1) we obtain . Application of the arithmetic square root gives

(2)

This is the equation we need right now.

Back to standard deviation

Property 1. Standard deviation is homogeneous of degree 1. Indeed, using homogeneity of variance and equation (2), we have

Unlike homogeneity of expected values, here we have an absolute value of the scaling coefficient .

Property 2. Cauchy-Schwarz inequality. (Part 1) For any random variables one has

(3) .

(Part 2) If the inequality sign in (3) turns into equality, , then is a linear function of : , with some constants .
Proof. (Part 1) If at least one of the variables is constant, both sides of the inequality are and there is nothing to prove. To exclude the trivial case, let be non-constant and, therefore, are positive. Consider a real-valued function of a real number defined by . Here we have variance of a linear combination

.

We see that is a parabola with branches looking upward (because the senior coefficient is positive). By nonnegativity of variance, and the parabola lies above the horizontal axis in the plane. Hence, the quadratic equation may have at most one real root. This means that the discriminant of the equation is non-positive:

Applying square roots to both sides of we finish the proof of the first part.

(Part 2) In case of the equality sign the discriminant is . Therefore the parabola touches the horizontal axis where . But we know that this implies which is just another way of writing .

Comment. (3) explains one of the main properties of the correlation:

.

Properties of means, covariances and variances are bread and butter of professionals. Here we consider the bread - the means

Properties of means: as simple as playing with tables

Definition of a random variable. When my Brazilian students asked for an intuitive definition of a random variable, I said: It is a function whose values are unpredictable. Therefore it is prohibited to work with their values and allowed to work only with their various means. For proofs we need a more technical definition: it is a table values+probabilities of type Table 1.

Table 1.  Random variable definition

Values of	Probabilities
...	...

Note: The complete form of writing is .

Mean (or expected value) value definition.  In words, this is a weighted sum of values, where the weights reflect the importance of corresponding .

Note: The expected value is a function whose argument is a complex object (it is described by Table 1) and the value is simple: is just a number. And it is not a product of and ! See how different means fit this definition.

Definition of a linear combination. See here the financial motivation. Suppose that are two discrete random variables with the same probability distribution . Let be real numbers. The random variable is called a linear combination of with coefficients . Its special cases are ( scaled by ) and (a sum of and ). The detailed definition is given by Table 2.

Table 2.  Linear operations definition

Values of	Values of	Probabilities
...	...	...	...	...	...

Note: The situation when the probability distributions are different is reduced to the case when they are the same, see my book.

Property 1. Linearity of means. For any random variables and any numbers one has

(1) .

Proof. This is one of those straightforward proofs when knowing the definitions and starting with the left-hand side is enough to arrive at the result. Using the definitions in Table 2, the mean of the linear combination is

(distributing probabilities)

(grouping by variables)

(pulling out constants)

See applications: one, and two, and three.

Generalization to the case of a linear combination of variables:

.

Special cases. a) Letting in (1) we get . This is called additivity. See an application. b) Letting in (1) we get . This property is called homogeneity of degree 1 (you can pull the constant out of the expected value sign). Ask your students to deduce linearity from homogeneity and additivity.

Property 2. Expected value of a constant. Everybody knows what a constant is. Ask your students what is a constant in terms of Table 1. The mean of a constant is that constant, because a constant doesn't change, rain or shine: (we have used the completeness axiom). In particular, it follows that .

Property 3. The expectation operator preserves order: if for all , then . In particular, the mean of a nonnegative random variable is nonnegative: if for all , then .

Indeed, using the fact that all probabilities are nonnegative, we get .

Property 4. For independent variables, we have  (multiplicativity), which has important implications on its own.

The best thing about the above properties is that, although we proved them under simplified assumptions, they are always true. We keep in mind that the expectation operator  is the device used by Mother Nature to measure the average, and most of the time she keeps hidden from us both the probabilities and the average .