Variance of a linear combination

May 18

The Newey-West estimator: uncorrelated and correlated data

The Newey-West estimator: uncorrelated and correlated data

I hate long posts but here we by necessity have to go through all ideas and calculations, to understand what is going on. One page of formulas in A. Patton's guide to FN3142 Quantitative Finance in my rendition becomes three posts.

Preliminaries and autocovariance function

Let $X_1,...,X_n$ be random variables. We need to recall that the variance of the vector $X=(X_1,...,X_n)^T$ is

(1) $V(X)=\left(\begin{array}{cccc}V(X_1)&Cov(X_1,X_2)&...&Cov(X_1,X_n) \\ Cov(X_2,X_1)&V(X_2)&...&Cov(X_2,X_n) \\ ...&...&...&... \\ Cov(X_n,X_1)&Cov(X_n,X_2)&...&V(X_n)\end{array}\right).$

With the help of this matrix we derived two expressions for variance of a linear combination:

(2) $V\left(\sum_{i=1}^na_iX_i\right)=\sum_{i=1}^na_i^2V(X_i)$

for uncorrelated variables and

(3) $V\left(\sum_{i=1}^na_iX_i\right)=\sum_{i=1}^na_i^2V(X_i)+2\sum_{i=1}^{n-1}\sum_{j=i+1}^na_ia_jCov(X_i,X_j)$

when there is autocorrelation.

In a time series context $X_1,...,X_n$ are observations along time. $1,...,n$ stand for moments in time and the sequence $X_1,...,X_n$ is called a time series. We need to recall the definition of a stationary process. Of that definition, we will use only the part about covariances: $Cov(X_i,X_j)$ depends only on the distance $|i-j|$ between the time moments $i,j.$ For example, in the top right corner of (1) we have $Cov(X_1,X_n),$ which depends only on $n-1.$

Preamble. Let $X_1,...,X_n$ be a stationary times series. Firstly, $Cov(X_i,X_i+k)$ depends only on $k.$ Secondly, for all integer $k=0,\pm 1,\pm 2,...$ denoting $j=i+k$ we have

(4) $Cov(X_i,X_{i+k})=Cov(X_{j-k},X_j)=Cov(X_j,X_{j-k}).$

Definition. The autocovariance function is defined by

(5) $\gamma_k=Cov(X_i,X_{k+i})$ for all integer $k=0,\pm 1,\pm 2,...$

In particular,

(6) $\gamma_0=Cov(X_i,X_i)=V(X_i)$ for all $i.$

The preamble shows that definition (5) is correct (the right side in (5) depends only on $k$ and not on $i$ ). Because of (4) we have symmetry $\gamma_{-k}=\gamma _k,$ so negative $k$ can be excluded from consideration.

With (5) and (6) for a stationary series (1) becomes

(7) $V(X)=\left(\begin{array}{cccc}\gamma_0&\gamma_1&...&\gamma_{n-1} \\ \gamma_1&\gamma_0&...&\gamma_{n-2} \\ ...&...&...&... \\ \gamma_{n-1}&\gamma_{n-2}&...&\gamma_0\end{array}\right).$

Estimating variance of a sample mean

Uncorrelated observations. Suppose $X_1,...,X_n$ are uncorrelated observations from the same population with variance $\sigma^2.$ From (2)
we get

(8) $V\left(\frac{1}{n}\sum_{i=1}^nX_i\right) =\frac{1}{n^2}\sum_{i=1}^nV(X_i)=\frac{n\sigma^2}{n^2}=\frac{\sigma^2}{n}.$

This is a theoretical relationship. To actually obtain an estimator of the sample variance, we need to replace $\sigma^2$ by some estimator. It is known that

(9) $s^2=\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2$

consistently estimates $\sigma^2.$ Plugging it in (8) we see that variance of the sample mean is consistently estimated by

$\hat{V}=\frac{1}{n}s^2=\frac{1}{n^2}\sum_{i=1}^n(X_i-\bar{X})^2.$

This is the estimator derived on p.151 of Patton's guide.

Correlated observations. In this case we use (3):

$V\left( \frac{1}{n}\sum_{i=1}^nX_i\right) =\frac{1}{n^{2}}\left[\sum_{i=1}^nV(X_i)+2\sum_{i=1}^{n-1}\sum_{j=i+1}^nCov(X_i,X_j)\right]$ .

Here visualization comes in handy. The sums in the square brackets include all terms on the main diagonal of (7) and above it. That is, we have $n$ copies of $\gamma_0,$ $n-1$ copies of $\gamma_{1}$ ,..., 2 copies of $\gamma _{n-2}$ and 1 copy of $\gamma _{n-1}.$ The sum in the brackets is

$\sum_{i=1}^nV(X_i)+2\sum_{i=1}^{n-1}\sum_{j=i+1}^nCov(X_i,X_j)=n\gamma_0+2[(n-1)\gamma_1+...+2\gamma_{n-2}+\gamma _{n-1}]=n\gamma_0+2\sum_{k=1}^{n-1}(n-k)\gamma_k.$

Thus we obtain the first equation on p.152 of Patton's guide (it's up to you to match the notation):

(10) $V(\bar{X})=\frac{1}{n}\gamma_0+\frac{2}{n}\sum_{k=1}^{n-1}(1-\frac{k}{n})\gamma_k.$

As above, this is just a theoretical relationship. $\gamma_0=V(X_i)=\sigma^2$ is estimated by (9). Ideally, the estimator of $\gamma_k=Cov(X_i,X_{k+i})$ is obtained by replacing all population means by sample means:

(11) $\hat{\gamma}_k=\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})(X_{k+i}-\bar{X}).$

There are two problems with this estimator, though. The first problem is that when $i$ runs from $1$ to $n,$ $k+i$ runs from $k+1$ to $k+n.$ To exclude out-of-sample values, the summation in (11) is reduced:

(12) $\hat{\gamma}_k=\frac{1}{n-k}\sum_{i=1}^{n-k}(X_i-\bar{X})(X_{k+i}-\bar{X}).$

The second problem is that the sum in (12) becomes too small when $k$ is close to $n.$ For example, for $k=n-1$ (12) contains just one term (there is no averaging). Therefore the upper limit of summation $n-1$ in (10) is replaced by some function $M(n)$ that tends to infinity slower than $n.$ The result is the estimator

$\hat{V}=\frac{1}{n}\hat{\gamma}_0+\frac{2}{n}\sum_{k=1}^{M(n)}(1-\frac{k}{M(n)})\hat{\gamma}_k$

where $\hat{\gamma}_0$ is given by (9) and $\hat{\gamma}_k$ is given by (12). This is almost the Newey-West estimator from p.152. The only difference is that instead of $\frac{k}{M(n)}$ they use $\frac{k}{M(n)+1}$ , and I have no idea why. One explanation is that for low $n$ , $M(n)$ can be zero, so they just wanted to avoid division by zero.

Tags: autocovariance function, correlated observations, Estimating variance of a sample mean, Newey-West estimator, Uncorrelated observations, University of London International Programmes, uol affiliate centre, Variance of a linear combination

Oct 16

Properties of variance

category: Agresti & Franklin, Agresti and Franklin, AP Stats and Business Stats, Econometrics, ST104A, ST104B, Statistics 1, Statistics 2, The College Board 19 comments

All properties of variance in one place

Certainty is the mother of quiet and repose, and uncertainty the cause of variance and contentions. Edward Coke

Preliminaries: study properties of means with proofs.

Definition. Yes, uncertainty leads to variance, and we measure it by $Var(X)=E(X-EX)^2$ . It is useful to use the name deviation from mean for $X-EX$ and realize that $E(X-EX)=0$ , so that the mean of the deviation from mean cannot serve as a measure of variation of $X$ around $EX$ .

Property 1. Variance of a linear combination. For any random variables $X,Y$ and numbers $a,b$ one has
(1) $Var(aX + bY)=a^2Var(X)+2abCov(X,Y)+b^2Var(Y).$
The term $2abCov(X,Y)$ in (1) is called an interaction term. See this post for the definition and properties of covariance.
Proof.
$Var(aX + bY)=E[aX + bY -E(aX + bY)]^2$

(using linearity of means)
$=E(aX + bY-aEX -bEY)^2$

(grouping by variable)
$=E[a(X-EX)+b(Y-EY)]^2$

(squaring out)
$=E[a^2(X-EX)^2+2ab(X-EX)(Y-EY)+(Y-EY)^2]$

(using linearity of means and definitions of variance and covariance)
$=a^2Var(X) + 2abCov(X,Y) +b^2Var(Y).$
Property 2. Variance of a sum. Letting in (1) $a=b=1$ we obtain
$Var(X + Y) = Var(X) + 2Cov(X,Y)+Var(Y).$

Property 3. Homogeneity of degree 2. Choose $b=0$ in (1) to get
$Var(aX)=a^2Var(X).$
Exercise. What do you think is larger: $Var(X+Y)$ or $Var(X-Y)$ ?
Property 4. If we add a constant to a variable, its variance does not change: $Var(X+c)=E[X+c-E(X+c)]^2=E(X+c-EX-c)^2=E(X-EX)^2=Var(X)$
Property 5. Variance of a constant is zero: $Var(c)=E(c-Ec)^2=0$ .

Property 6. Nonnegativity. Since the squared deviation from mean $(X-EX)^2$ is nonnegative, its expectation is nonnegative: $E(X-EX)^2\ge 0$ .

Property 7. Only a constant can have variance equal to zero: If $Var(X)=0$ , then $E(X-EX)^2 =(x_1-EX)^2p_1 +...+(x_n-EX)^2p_n=0$ , see the definition of the expected value. Since all probabilities are positive, we conclude that $x_i=EX$ for all $i$ , which means that $X$ is identically constant.

Property 8. Shortcut for variance. We have an identity $E(X-EX)^2=EX^2-(EX)^2$ . Indeed, squaring out gives

$E(X-EX)^2 =E(X^2-2XEX+(EX)^2)$

(distributing expectation)

$=EX^2-2E(XEX)+E(EX)^2$

(expectation of a constant is constant)

$=EX^2-2(EX)^2+(EX)^2=EX^2-(EX)^2$ .

All of the above properties apply to any random variables. The next one is an exception in the sense that it applies only to uncorrelated variables.

Property 9. If variables are uncorrelated, that is $Cov(X,Y)=0$ , then from (1) we have $Var(aX + bY)=a^2Var(X)+b^2Var(Y).$ In particular, letting $a=b=1$ , we get additivity: $Var(X+Y)=Var(X)+Var(Y).$ Recall that the expected value is always additive.