Raising the bar

Estimation of parameters of a normal distribution

Here we show that the knowledge of the distribution of $latex s^{2}$ for linear regression allows one to do without long calculations contained in the guide ST 2134 by J. Abdey.

Theorem. Let $latex y_{1},...,y_{n}$ be independent observations from $latex N\left( \mu,\sigma ^{2}\right) $. 1) $latex s^{2}\left( n-1\right) /\sigma ^{2}$ is distributed as $latex \chi _{n-1}^{2}.$ 2) The estimators $latex \bar{y}$ and $latex s^{2}$ are independent. 3) $latex Es^{2}=\sigma ^{2},$ 4) $latex Var\left( s^{2}\right) =\frac{2\sigma ^{4}}{n-1},$ 5) $latex \frac{s^{2}-\sigma ^{2}}{\sqrt{2\sigma ^{4}/\left(n-1\right) }}$ converges in distribution to $latex N\left( 0,1\right) .$

Proof. We can write $latex y_{i}=\mu +e_{i}$ where $latex e_{i}$ is distributed as $latex N\left( 0,\sigma ^{2}\right) .$ Putting $latex \beta =\mu ,\ y=\left(y_{1},...,y_{n}\right) ^{T},$ $latex e=\left( e_{1},...,e_{n}\right) ^{T}$ and $latex X=\left( 1,...,1\right) ^{T}$ (a vector of ones) we satisfy (1) and (2). Since $latex X^{T}X=n,$ we have $latex \hat{\beta}=\bar{y}.$ Further,

$latex r\equiv y-X\hat{
\beta}=\left( y_{1}-\bar{y},...,y_{n}-\bar{y}\right) ^{T}$

and

$latex s^2=\left\Vert r\right\Vert^2/(n-1) =\sum_{i=1}^n(y_i-\bar{y})^2/( n-1$.

Thus 1) and 2) follow from results for linear regression.

3) For a normal variable $latex X$ its moment generating function is $latex M_{X}\left( t\right) =\exp \left(\mu t+\frac{1}{2}\sigma ^{2}t^{2}\right) $ (see Guide ST2133, 2021, p.88). For the standard normal we get

$latex M_{z}^{\prime }\left( t\right) =\exp \left(
\frac{1}{2}t^{2}\right) t,$ $latex M_{z}^{\prime \prime }\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right) (t^{2}+1),$

$latex M_{z}^{\prime \prime \prime}\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right) (t^{3}+2t+t),$ $latex M_{z}^{(4)}\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right)
(t^{4}+6t^{2}+3).$

Applying the general property $latex EX^{r}=M_{X}^{\left(
r\right) }\left( 0\right) $ (same guide, p.84) we see that

$latex Ez=0,$ $latex Ez^{2}=1,$ $latex Ez^{3}=0,$ $latex Ez^{4}=3,$

$latex Var(z)=1,$ $latex Var\left( z^{2}\right) =Ez^{4}-\left( Ez^{2}\right)
^{2}=3-1=2.$

Therefore

$latex Es^{2}=\frac{\sigma ^{2}}{n-1}E\left( z_{1}^{2}+...+z_{n-1}^{2}\right) =\frac{\sigma ^{2}}{n-1}\left( n-1\right) =\sigma ^{2}.$

4) By independence of standard normals

$latex Var\left( s^{2}\right) =$ $latex \left(\frac{\sigma ^{2}}{n-1}\right) ^{2}\left[ Var\left( z_{1}^{2}\right)
+...+Var\left( z_{n-1}^{2}\right) \right] =\frac{\sigma ^{4}}{\left(
n-1\right) ^{2}}2\left( n-1\right) =\frac{2\sigma ^{4}}{n-1}.$

5) By standardizing $latex s^{2}$ we have $latex \frac{s^{2}-Es^{2}}{\sigma \left(s^{2}\right) }=\frac{s^{2}-\sigma ^{2}}{\sqrt{2\sigma ^{4}/\left( n-1\right) }}$ and this converges in distribution to $latex N\left( 0,1\right) $ by the central limit theorem.