Raising the bar

Application: distribution of sigma squared estimator

For the formulation of multiple regression and classical conditions on its elements see Application: estimating sigma squared. There we proved unbiasedness of the OLS estimator of $latex \sigma^2.$ Here we do more: we characterize its distribution and obtain unbiasedness as a corollary.

Preliminaries

We need a summary of what we know about the residual $latex r=y-\hat{y}$ and the projector $latex Q=I-P$ where $latex P=X^T(X^TX)^{-1}X^T:$

(1) $latex \Vert r\Vert^2=e^TQe.$

$latex P$ has $latex k$ unities and $latex n-k$ zeros on the diagonal of its diagonal representation, where $latex k$ is the number of regressors. With $latex Q$ it's the opposite: it has $latex n-k$ unities and $latex k$ zeros on the diagonal of its diagonal representation. We can always assume that the unities come first, so in the diagonal representation

(2) $latex Q=UDU^{-1}$

the matrix $latex U$ is orthogonal and $latex D$ can be written as

(3) $latex D=\left(\begin{array}{cc}I_{n-k}&0\\0&0\end{array}\right)$

where $latex I_{n-k}$ is an identity matrix and the zeros are zero matrices of compatible dimensions.

Characterization of the distribution of $latex s^2$

Exercise 1. Suppose the error vector $latex e$ is normal: $latex e\sim N(0,\sigma^2I).$ Prove that the vector $latex \delta =U^{-1}e/\sigma$ is standard normal.

Proof. By the properties of orthogonal matrices

$latex Var(\delta)=E\delta\delta^T=U^{-1}Eee^TU/\sigma^2=U^{-1}U=I.$

This, together with the equation $latex E\delta =0$, proves that $latex \delta $ is standard normal.

Exercise 2. Prove that $latex \Vert r\Vert^2/\sigma^2$ is distributed as $latex \chi _{n-k}^2.$

Proof. From (1) and (2) we have

$latex \Vert r\Vert^2/\sigma^2=e^TUDU^{-1}e/\sigma^2=(U^{-1}e)^TD(U^{-1}e)/\sigma^2=\delta^TD\delta.$

Now (3) shows that $latex \Vert r\Vert^2/\sigma^2=\sum_{i=1}^{n-k}\delta_i^2$ which is the definition of $latex \chi
_{n-k}^2.$

Exercise 3. Find the mean and variance of $latex s^2=\Vert r\Vert^2/(n-k)=\sigma^2\chi _{n-k}^2/(n-k).$

Solution. From Exercise 2 we obtain the result proved earlier in a different way:

$latex Es^2=\sigma^2E\chi _{n-k}^2/(n-k)=\sigma^2.$

Further, using the variance of a standard normal

$latex Var(s^2)=\frac{\sigma^4}{(n-k)^2}\sum_{i=1}^{n-k}Var(\delta_i^2)=\frac{2\sigma^4}{n-k}.$

← Previous: Distributions derived from normal variables | Next: Questions for repetition →

Get the PDF book “Matrix Algebra” on Lulu