Raising the bar

Blueprint for exam versions

This is the exam I administered in my class in Spring 2022. By replacing the Poisson distribution with other random variables the UoL examiners can obtain a large variety of versions with which to torture Advanced Statistics students. On the other hand, for the students the answers below can be a blueprint to fend off any assaults.

During the semester my students were encouraged to analyze and collect information in documents typed in Scientific Word or LyX. The exam was an open-book online assessment. Papers typed in Scientific Word or LyX were preferred and copying from previous analysis was welcomed. This policy would be my preference if I were to study a subject as complex as Advanced Statistics. The students were given just two hours on the assumption that they had done the preparations diligently. Below I give the model answers right after the questions.

Midterm Spring 2022

You have to clearly state all required theoretical facts. Number all equations that you need to use in later calculations and reference them as necessary. Answer the questions in the order they are asked. When you don't know the answer, leave some space. For each unexplained fact I subtract one point. Put your name in the file name.

In questions 1-9 $latex X$ is the Poisson variable.

Question 1

Define $latex X$ and derive the population mean and population variance of the sum $latex S_{n}=\sum_{i=1}^{n}X_{i}$ where $latex X_{i}$ is an i.i.d. sample from $latex X$.

Answer. $latex X$ is defined by $latex P\left( X=x\right) =e^{-\lambda }\frac{\lambda ^{x}}{x!},\
x=0,1,...$ Using $latex EX=\lambda $ and $latex Var\left( X\right) =\lambda $ (ST2133 p.80) we have

$latex ES_{n}=\sum EX_{i}=n\lambda ,$ $latex Var\left( S_{n}\right) =\sum V\left(
X_{i}\right) =n\lambda $

(by independence and identical distribution). [Some students derived $latex EX=\lambda ,$ $latex Var\left( X\right) =\lambda $ instead of respective equations for sample means].

Question 2

Derive the MGF of the standardized sample mean.

Answer. Knowing this derivation is a must because it is a combination of three important facts.

a) Let $latex z_{n}=\frac{\bar{X}-E\bar{X}}{\sigma \left( \bar{X}\right) }.$ Then $latex z_n=\frac{nS_{n}-EnS_{n}}{\sigma(nS_n) }=\frac{S_n-ES_n}{\sigma(S_n)}$, so standardizing $latex \bar{X}$ and $latex S_{n}$ gives the same result.

b) The MGF of $latex S_{n}$ is expressed through the MGF of $latex X$:

$latex M_{S_{n}}\left( t\right)
=Ee^{S_{n}t}=Ee^{X_{1}t+...+X_{n}t}=Ee^{X_{1}t}...e^{X_{n}t}=$

(independence) $latex =Ee^{X_{1}t}...Ee^{X_{n}t}=$ (identical distribution) $latex =\left[ M_{X}\left( t\right) \right] ^{n}.$

c) If $latex X$ is a linear transformation of $latex Y,$ $latex X=a+bY,$ then

$latex M_{X}\left( t\right) =Ee^{X}=Ee^{\left( a+bY\right) t}=e^{at}Ee^{Y\left(
bt\right) }=e^{at}M_{Y}\left( bt\right) .$

When answering the question we assume any i.i.d. sample from a population with mean $latex \mu $ and population variance $latex \sigma ^{2}$:

Putting in c) $latex a=-\frac{ES_{n}}{\sigma \left( S_{n}\right) },$ $latex b=\frac{1}{\sigma \left( S_{n}\right) }$ and using a) we get

$latex M_{z_{n}}\left( t\right) =E\exp \left( \frac{S_{n}-ES_{n}}{\sigma \left(
S_{n}\right) }t\right) =e^{-ES_{n}t/\sigma \left( S_{n}\right)
}M_{S_{n}}\left( t/\sigma \left( S_{n}\right) \right) $

(using b) and $latex ES_{n}=n\mu ,$ $latex Var\left( S_{n}\right) =n\sigma ^{2}$)

$latex =e^{-ES_{n}t/\sigma \left( S_{n}\right) }\left[ M_{X}\left( t/\sigma \left(
S_{n}\right) \right) \right] ^{n}=e^{-n\mu t/\left( \sqrt{n}\sigma \right) }%
\left[ M_{X}\left( t/\left( \sqrt{n}\sigma \right) \right) \right] ^{n}.$

This is a general result which for the Poisson distribution can be specified as follows. From ST2133, example 3.38 we know that $latex M_{X}\left( t\right)=\exp \left( \lambda \left( e^{t}-1\right) \right) $. Therefore, we obtain

$latex M_{z_{n}}\left( t\right) =e^{-\sqrt{\lambda }t}\left[ \exp \left( \lambda
\left( e^{t/\left( n\sqrt{\lambda }\right) }-1\right) \right) \right] ^{n}=
e^{-t\sqrt{\lambda }+n\lambda \left( e^{t/\left( n\sqrt{\lambda }\right)
}-1\right) }.$

[Instead of $latex M_{z_n}$ some students gave $latex M_X$.]

Question 3

Derive the cumulant generating function of the standardized sample mean.

Answer. Again, there are a couple of useful general facts.

I) Decomposition of MGF around zero. The series $latex e^{x}=\sum_{i=0}^{\infty }
\frac{x^{i}}{i!}$ leads to

$latex M_{X}\left( t\right) =Ee^{tX}=E\left( \sum_{i=0}^{\infty }\frac{t^{i}X^{i}}{
i!}\right) =\sum_{i=0}^{\infty }\frac{t^{i}}{i!}E\left( X^{i}\right)
=\sum_{i=0}^{\infty }\frac{t^{i}}{i!}\mu _{i}$

where $latex \mu _{i}=E\left( X^{i}\right) $ are moments of $latex X$ and $latex \mu
_{0}=EX^{0}=1.$ Differentiating this equation yields

$latex M_{X}^{(k)}\left( t\right) =\sum_{i=k}^{\infty }\frac{t^{i-k}}{\left(
i-k\right) !}\mu _{i}$

and setting $latex t=0$ gives the rule for finding moments from MGF: $latex \mu_k=M_X^{(k)}(0)$.

II) Decomposition of the cumulant generating function around zero. $latex K_{X}\left( t\right) =\log M_{X}\left( t\right) $ can also be decomposed into its Taylor series:

$latex K_{X}\left( t\right) =\sum_{i=0}^{\infty }\frac{t^{i}}{i!}\kappa _{i}$

where the coefficients $latex \kappa_i$ are called cumulants and can be found using $latex \kappa_ k=K_X^{(k)}(0)$. Since

$latex K_{X}^{\prime }\left( t\right) =\frac{M_{X}^{\prime }\left( t\right) }{
M_{X}\left( t\right) }$ and $latex K_{X}^{\prime \prime }\left( t\right) =\frac{
M_{X}^{\prime \prime }\left( t\right) M_{X}\left( t\right) -\left(
M_{X}^{\prime }\left( t\right) \right) ^{2}}{M_{X}^{2}\left( t\right) }$

we have

$latex \kappa_0=\log M_X(0) =0,$ $latex \kappa_1=\frac{M_X^{\prime}(0)}{M_X(0)}=\mu_1$,

$latex \kappa_2=\mu_2-\mu_1^2=EX^2-(EX)^2=Var(X)$.

Thus, for any random variable $latex X$ with mean $latex \mu $ and variance $latex \sigma ^{2}$ we have

$latex K_{X}\left( t\right) =\mu t+\frac{\sigma ^{2}t^{2}}{2}+$ terms of higher order for $latex t$ small.

III) If $latex X=a+bY$ then by c)

$latex K_{X}\left( t\right) =K_{a+bY}\left( t\right) =\log \left[
e^{at}M_{Y}\left( bt\right) \right] =at+K_{X}\left( bt\right) .$

IV) By b)

$latex K_{S_{n}}\left( t\right) =\log \left[ M_{X}\left( t\right) \right]
^{n}=nK_{X}\left( t\right) .$

Using III), $latex z_{n}=\frac{S_{n}-ES_{n}}{\sigma \left( S_{n}\right) }$ and then IV) we have

$latex K_{z_{n}}\left( t\right) =\frac{-ES_{n}}{\sigma \left( S_{n}\right) }
t+K_{S_{n}}\left( \frac{t}{\sigma \left( S_{n}\right) }\right) =\frac{-ES_{n}
}{\sigma \left( S_{n}\right) }t+nK_{X}\left( \frac{t}{\sigma \left(
S_{n}\right) }\right) .$

For the last term on the right we use the approximation around zero from II):

$latex K_{z_{n}}\left( t\right) =\frac{-ES_{n}}{\sigma \left( S_{n}\right) }
t+nK_{X}\left( \frac{t}{\sigma \left( S_{n}\right) }\right) \approx \frac{
-ES_{n}}{\sigma \left( S_{n}\right) }t+n\mu \frac{t}{\sigma \left(
S_{n}\right) }+n\frac{\sigma ^{2}}{2}\left( \frac{t}{\sigma \left(
S_{n}\right) }\right) ^{2}$

$latex =-\frac{n\mu }{\sqrt{n}\sigma }t+n\mu \frac{t}{\sqrt{n}\sigma }+n\frac{
\sigma ^{2}}{2}\left( \frac{t}{\sigma \left( S_{n}\right) }\right)
^{2}=t^{2}/2.$

[Important. Why the above steps are necessary? Passing from the series $latex M_X(t) =\sum_{i=0}^\infty\frac{t^i}{i!}\mu_i$ to the series for $latex K_X(t) =\log M_X(t)$ is not straightforward and can easily lead to errors. It is not advisable in case of the Poisson to derive $latex K_{z_n}$ from $latex M_{z_n}(t) =e^{-t\sqrt{\lambda}+n\lambda\left(e^{t/\left(n\sqrt{\lambda}\right)
}-1\right) }$.]

Question 4

Prove the central limit theorem using the cumulant generating function you obtained.

Answer. In the previous question we proved that around zero

$latex K_{z_{n}}\left( t\right) \rightarrow \frac{t^{2}}{2}.$

This implies that

(1) $latex M_{z_{n}}\left( t\right) \rightarrow e^{t^{2}/2}$ for each $latex t$ around zero.

But we know that for a standard normal $latex X$ its MGF is $latex M_{X}\left( t\right)
=\exp \left( \mu t+\frac{\sigma ^{2}t^{2}}{2}\right) $ (ST2133 example 3.42) and hence for the standard normal

(2) $latex M_{z}\left( t\right) =e^{t^{2}/2}.$

Theorem (link between pointwise convergence of MGFs of $latex \left\{X_{n}\right\} $ and convergence in distribution of $latex \left\{ X_n\right\}$) Let $latex \left\{ X_{n}\right\} $ be a sequence of random variables and let $latex X$ be some random variable. If $latex M_{X_{n}}\left( t\right) $ converges for each $latex t$ from a neighborhood of zero to $latex M_{X}\left( t\right)$, then $latex X_{n}$ converges in distribution to $latex X.$

Using (1), (2) and this theorem we finish the proof that $latex z_{n}$ converges in distribution to the standard normal, which is the central limit theorem.

Question 5

State the factorization theorem and apply it to show that $latex U=\sum_{i=1}^{n}X_{i}$ is a sufficient statistic.

Answer. The solution is given on p.180 of ST2134. For $latex x_{i}=1,...,n$ the joint density is

(3) $latex f_{X}\left( x,\lambda \right) =\prod\limits_{i=1}^{n}e^{-\lambda}\frac{\lambda^{x_{i}}}{x_{i}!}=\frac{\lambda^{\Sigma x_{i}}e^{-n\lambda}}{\Pi_{i=1}^{n}x_{i}!}.$

To satisfy the Fisher-Neyman factorization theorem set

$latex g\left(\sum x_{i},\lambda\right)=\lambda^{\Sigma x_{i}e^{-n\lambda}},\ h\left( x\right) =\frac{1}{\Pi_{i=1}^{n}x_{i}!}$

and then we see that $latex \sum x_{i}$ is a sufficient statistic for $latex \lambda .$

Question 6

Find a minimal sufficient statistic for $latex \lambda $ stating all necessary theoretical facts.

Answer. Characterization of minimal sufficiency A statistic $latex T\left(
X\right) $ is minimal sufficient if and only if level sets of $latex T$ coincide with sets on which the ratio $latex f_{X}\left( x,\theta \right) /f_{X}\left(
y,\theta \right) $ does not depend on $latex \theta .$

From (3)

$latex f_X(x,\lambda)/f_X(y,\lambda) =\frac{\lambda^{\Sigma x_i}e^{-n\lambda}}{\Pi_{i=1}^nx_i!}\left[\frac{\lambda^{\Sigma y_i}e^{-n\lambda}}{\Pi_{i=1}^ny_i!}\right] ^{-1}=\lambda^{\left[\Sigma x_i-\Sigma y_i\right]}\frac{\Pi_{i=1}^ny_i!}{\Pi_{i=1}^nx_i!}.$

The expression on the right does not depend on $latex \lambda $ if and only of $latex \Sigma x_{i}=\Sigma y_{i}0$. The last condition describes level sets of $latex T\left( X\right) =\sum X_{i}.$ Thus it is minimal sufficient.

Question 7

Find the Method of Moments estimator of the population mean.

Answer. The idea of the method is to take some populational property (for example, $latex
EX=\lambda $) and replace the population characteristic (in this case $latex EX$) by its sample analog ($latex \bar{X}$) to obtain a MM estimator. In our case $latex \hat{\lambda}_{MM}=
\bar{X}.$ [Try to do this for the Gamma distribution].

Question 8

Find the Fisher information.

Answer. From Problem 5 the log-likelihood is

$latex l_{X}\left( \lambda ,x\right) =-n\lambda +\sum x_{i}\log \lambda -\sum \log
\left( x_{i}!\right) .$

Hence the score function is (see Example 2.30 in ST2134)

$latex s_{X}\left( \lambda ,x\right) =\frac{\partial }{\partial \lambda }
l_{X}\left( \lambda ,x\right) =-n+\frac{1}{\lambda }\sum x_{i}.$

Then

$latex \frac{\partial ^{2}}{\partial \lambda ^{2}}l_{X}\left( \lambda ,x\right) =-
\frac{1}{\lambda ^{2}}\sum x_{i}$

and the Fisher information is

$latex I_{X}\left( \lambda \right) =-E\left( \frac{\partial ^{2}}{\partial \lambda
^{2}}l_{X}\left( \lambda ,x\right) \right) =\frac{1}{\lambda ^{2}}E\sum
X_{i}=\frac{n\lambda }{\lambda ^{2}}=\frac{n}{\lambda }.$

Question 9

Derive the Cramer-Rao lower bound for $latex V\left( \bar{X}\right) $ for a random sample.

Answer. (See Example 3.17 in ST2134) Since $latex \bar{X}$ is an unbiased estimator of $latex
\lambda $ by Problem 1, from the Cramer-Rao theorem we know that

$latex V\left( \bar{X}\right) \geq \frac{1}{I_{X}\left( \lambda \right) }=\frac{
\lambda }{n}$

and in fact by Problem 1 this lower bound is attained.