Raising the bar

Distribution of the estimator of the error variance

If you are reading the book by Dougherty: this post is about the distribution of the estimator  $latex s^2$ defined in Chapter 3.

Consider regression

(1) $latex y=X\beta +e$

where the deterministic matrix $latex X$ is of size $latex n\times k,$ satisfies $latex \det
\left( X^{T}X\right) \neq 0$ (regressors are not collinear) and the error $latex e$ satisfies

(2) $latex Ee=0,Var(e)=\sigma ^{2}I$

$latex \beta $ is estimated by $latex \hat{\beta}=(X^{T}X)^{-1}X^{T}y.$ Denote $latex P=X(X^{T}X)^{-1}X^{T},$ $latex Q=I-P.$ Using (1) we see that $latex \hat{\beta}=\beta +(X^{T}X)^{-1}X^{T}e$ and the residual $latex r\equiv y-X\hat{\beta}=Qe.$ $latex \sigma^{2}$ is estimated by

(3) $latex s^{2}=\left\Vert r\right\Vert ^{2}/\left( n-k\right) =\left\Vert
Qe\right\Vert ^{2}/\left( n-k\right) .$

$latex Q$ is a projector and has properties which are derived from those of $latex P$

(4) $latex Q^{T}=Q,$ $latex Q^{2}=Q.$

If $latex \lambda $ is an eigenvalue of $latex Q,$ then multiplying $latex Qx=\lambda x$ by $latex Q$ and using the fact that $latex x\neq 0$ we get $latex \lambda ^{2}=\lambda .$ Hence eigenvalues of $latex Q$ can be only $latex 0$ or $latex 1.$ The equation $latex tr\left( Q\right) =n-k$
tells us that the number of eigenvalues equal to 1 is $latex n-k$ and the remaining $latex k$ are zeros. Let $latex Q=U\Lambda U^{T}$ be the diagonal representation of $latex Q.$ Here $latex U$ is an orthogonal matrix,

(5) $latex U^{T}U=I,$

and $latex \Lambda $ is a diagonal matrix with eigenvalues of $latex Q$ on the main diagonal. We can assume that the first $latex n-k$ numbers on the diagonal of $latex Q$ are ones and the others are zeros.

Theorem. Let $latex e$ be normal. 1) $latex s^{2}\left( n-k\right) /\sigma ^{2}$ is distributed as $latex \chi _{n-k}^{2}.$ 2) The estimators $latex \hat{\beta}$ and $latex s^{2}$ are independent.

Proof. 1) We have by (4)

(6) $latex \left\Vert Qe\right\Vert ^{2}=\left( Qe\right) ^{T}Qe=\left(
Q^{T}Qe\right) ^{T}e=\left( Qe\right) ^{T}e=\left( U\Lambda U^{T}e\right)
^{T}e=\left( \Lambda U^{T}e\right) ^{T}U^{T}e.$

Denote $latex S=U^{T}e.$ From (2) and (5)

$latex ES=0,$ $latex Var\left( S\right) =EU^{T}ee^{T}U=\sigma ^{2}U^{T}U=\sigma ^{2}I$

and $latex S$ is normal as a linear transformation of a normal vector. It follows that $latex S=\sigma z$ where $latex z$ is a standard normal vector with independent standard normal coordinates $latex z_{1},...,z_{n}.$ Hence, (6) implies

(7) $latex \left\Vert Qe\right\Vert ^{2}=\sigma ^{2}\left( \Lambda z\right)
^{T}z=\sigma ^{2}\left( z_{1}^{2}+...+z_{n-k}^{2}\right) =\sigma ^{2}\chi
_{n-k}^{2}.$

(3) and (7) prove the first statement.

2) First we note that the vectors $latex Pe,Qe$ are independent. Since they are normal, their independence follows from

$latex cov(Pe,Qe)=EPee^{T}Q^{T}=\sigma ^{2}PQ=0.$

It's easy to see that $latex X^{T}P=X^{T}.$ This allows us to show that $latex \hat{\beta}$ is a function of $latex Pe$:

$latex \hat{\beta}=\beta +(X^{T}X)^{-1}X^{T}e=\beta +(X^{T}X)^{-1}X^{T}Pe.$

Independence of $latex Pe,Qe$ leads to independence of their functions $latex \hat{\beta}$ and $latex s^{2}.$

← Previous: Application: estimating sigma squared | Next: Distributions derived from normal variables →

Get the PDF book “Matrix Algebra” on Lulu