Raising the bar Estimation of parameters of a normal distribution

Feb 22

Estimation of parameters of a normal distribution

category: Advanced Statistics ST2133, Advanced Statistics ST2134 No Comments

Estimation of parameters of a normal distribution

Here we show that the knowledge of the distribution of $s^{2}$ for linear regression allows one to do without long calculations contained in the guide ST 2134 by J. Abdey.

Theorem. Let $y_{1},...,y_{n}$ be independent observations from $N\left( \mu,\sigma ^{2}\right)$ . 1) $s^{2}\left( n-1\right) /\sigma ^{2}$ is distributed as $\chi _{n-1}^{2}.$ 2) The estimators $\bar{y}$ and $s^{2}$ are independent. 3) $Es^{2}=\sigma ^{2},$ 4) $Var\left( s^{2}\right) =\frac{2\sigma ^{4}}{n-1},$ 5) $\frac{s^{2}-\sigma ^{2}}{\sqrt{2\sigma ^{4}/\left(n-1\right) }}$ converges in distribution to $N\left( 0,1\right) .$

Proof. We can write $y_{i}=\mu +e_{i}$ where $e_{i}$ is distributed as $N\left( 0,\sigma ^{2}\right) .$ Putting $\beta =\mu ,\ y=\left(y_{1},...,y_{n}\right) ^{T},$ $e=\left( e_{1},...,e_{n}\right) ^{T}$ and $X=\left( 1,...,1\right) ^{T}$ (a vector of ones) we satisfy (1) and (2). Since $X^{T}X=n,$ we have $\hat{\beta}=\bar{y}.$ Further,

$r\equiv y-X\hat{ \beta}=\left( y_{1}-\bar{y},...,y_{n}-\bar{y}\right) ^{T}$

and

$s^{2}=\left\Vert r\right\Vert ^{2}/\left( n-1\right) =\sum_{i=1}^{n}\left( y_{i}-\bar{y}\right) ^{2}/\left( n-1\right) .$

Thus 1) and 2) follow from results for linear regression.

3) For a normal variable $X$ its moment generating function is $M_{X}\left( t\right) =\exp \left(\mu t+\frac{1}{2}\sigma ^{2}t^{2}\right)$ (see Guide ST2133, 2021, p.88). For the standard normal we get

$M_{z}^{\prime }\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right) t,$ $M_{z}^{\prime \prime }\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right) (t^{2}+1),$

$M_{z}^{\prime \prime \prime}\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right) (t^{3}+2t+t),$ $M_{z}^{(4)}\left( t\right) =\exp \left( \frac{1}{2}t^{2}\right) (t^{4}+6t^{2}+3).$

Applying the general property $EX^{r}=M_{X}^{\left( r\right) }\left( 0\right)$ (same guide, p.84) we see that

$Ez=0,$ $Ez^{2}=1,$ $Ez^{3}=0,$ $Ez^{4}=3,$

$Var(z)=1,$ $Var\left( z^{2}\right) =Ez^{4}-\left( Ez^{2}\right) ^{2}=3-1=2.$

Therefore

$Es^{2}=\frac{\sigma ^{2}}{n-1}E\left( z_{1}^{2}+...+z_{n-1}^{2}\right) =\frac{\sigma ^{2}}{n-1}\left( n-1\right) =\sigma ^{2}.$

4) By independence of standard normals

$Var\left( s^{2}\right) =$ $\left(\frac{\sigma ^{2}}{n-1}\right) ^{2}\left[ Var\left( z_{1}^{2}\right) +...+Var\left( z_{n-1}^{2}\right) \right] =\frac{\sigma ^{4}}{\left( n-1\right) ^{2}}2\left( n-1\right) =\frac{2\sigma ^{4}}{n-1}.$

5) By standardizing $s^{2}$ we have $\frac{s^{2}-Es^{2}}{\sigma \left(s^{2}\right) }=\frac{s^{2}-\sigma ^{2}}{\sqrt{2\sigma ^{4}/\left( n-1\right) }}$ and this converges in distribution to $N\left( 0,1\right)$ by the central limit theorem.