Application: estimating sigma squared
Consider multiple regression
(a) the regressors are assumed deterministic, (b) the number of regressors is smaller than the number of observations (c) the regressors are linearly independent, and (d) the errors are homoscedastic and uncorrelated,
Usually students remember that should be estimated and don't pay attention to estimation of Partly this is because does not appear in the regression and partly because the result on estimation of error variance is more complex than the result on the OLS estimator of
Definition 1. Let be the OLS estimator of . is called the fitted value and is called the residual.
Exercise 1. Using the projectors and show that and
Proof. The first equation is obvious. From the model we have Since we have further
Definition 2. The OLS estimator of is defined by
Exercise 2. Prove that is unbiased:
Proof. Using projector properties we have
Expectations of type and would be easy to find from (2). However, we need to find where there is an obstructing See how this difficulty is overcome in the next calculation.
( is a scalar, so its trace is equal to itself)
(the regressors and hence are deterministic, so we can use linearity of )
because this is the dimension of the image of Therefore Thus, and