Raising the bar

Application: estimating sigma squared

Consider multiple regression

(1) $latex y=X\beta +e$

where

(a) the regressors are assumed deterministic, (b) the number of regressors $latex k$ is smaller than the number of observations $latex n,$ (c) the regressors are linearly independent, $latex \det (X^TX)\neq 0,$ and (d) the errors are homoscedastic and uncorrelated,

(2) $latex Var(e)=\sigma^2I.$

Usually students remember that $latex \beta$ should be estimated and don't pay attention to estimation of $latex \sigma^2.$ Partly this is because $latex \sigma^2$ does not appear in the regression and partly because the result on estimation of error variance is more complex than the result on the OLS estimator of $latex \beta .$

Definition 1. Let $latex \hat{\beta}=(X^TX)^{-1}X^Ty$ be the OLS estimator of $latex \beta$. $latex \hat{y}=X\hat{\beta}$ is called the fitted value and $latex r=y-\hat{y}$ is called the residual.

Exercise 1. Using the projectors $latex P=X(X^TX)^{-1}X^T$ and $latex Q=I-P$ show that $latex \hat{y}=Py$ and $latex r=Qe.$

Proof. The first equation is obvious. From the model we have $latex r=X\beta+e-P(X\beta +e).$ Since $latex PX\beta=X\beta,$ we have further $latex r=e-Pe=Qe.$

Definition 2. The OLS estimator of $latex \sigma^2$ is defined by $latex s^2=\Vert r\Vert^2/(n-k).$

Exercise 2. Prove that $latex s^2$ is unbiased: $latex Es^2=\sigma^2.$

Proof. Using projector properties we have

$latex \Vert r\Vert^2=(Qe)^TQe=e^TQ^TQe=e^TQe.$

Expectations of type $latex Ee^Te$ and $latex Eee^T$ would be easy to find from (2). However, we need to find $latex Ee^TQe$ where there is an obstructing $latex Q.$ See how this difficulty is overcome in the next calculation.

$latex E\Vert r\Vert^2=Ee^TQe$ ($latex e^TQe$ is a scalar, so its trace is equal to itself)

$latex =Etr(e^TQe)$ (applying trace-commuting)

$latex =Etr(Qee^T)$ (the regressors and hence $latex Q$ are deterministic, so we can use linearity of $latex E$)

$latex =tr(QEee^T)$ (applying (2)) $latex =\sigma^2tr(Q).$

$latex tr(P)=k$ because this is the dimension of the image of $latex P.$ Therefore $latex tr(Q)=n-k.$ Thus, $latex E\Vert r\Vert^2=\sigma^2(n-k)$ and $latex Es^2=\sigma^2.$

← Previous: Eigenvalues and eigenvectors of a projector | Next: Distribution of the estimator of the error variance →

Get the PDF book “Matrix Algebra” on Lulu