Raising the bar

The Rao-Blackwell theorem

This theorem provides one of the uses of sufficient statistics. Roughly speaking, it says that by conditioning any statistic on a sufficient statistic we obtain a better statistic with lower mean squared error.

Notation. Let $latex S (X)$ be a sufficient statistic for $latex \theta$ ($latex S$ for Sufficient). Take any other statistic $latex A (X)$ ($latex A$ for Any) and consider $latex B (X) = E (A (X) | S (X) )$ ($latex B$ for Better). For conditioning on $latex S (X)$ we use the notation $latex E_{S (X)} Y = E (Y | S (X) )$, which will make the proof much clearer.

Theorem (Rao-Blackwell) $latex {MSE} (B (X)) \leqslant {MSE} (A (X)) .$

Proof. Step 1. Obviously, $latex B (X)$ depends on the sample. To be sure that it is a statistic, we have to satisfy ourselves that it does not depend on the parameter to be estimated. By the definition of a sufficient statistic, the distribution of the sample conditional on $latex S (X)$ does not depend on $latex \theta$. Since $latex A (X)$ depends only on the sample, its distribution conditional on $latex S (X)$ also does not depend on $latex \theta.$ Hence, the conditional mean $latex B (X) = E (A (X) | S(X) )$ does  not depend on $latex \theta$.

Step 2. We derive a simple auxiliary inequality. From the alternative expression for variance and its non-negativity $latex {Var} (Y) = E (Y^2) -({EY})^2 \geqslant 0$ we get $latex E (Y^2) \geqslant ({EY})^2 .$ Similarly, using conditional variance $latex Var_{S (X)}(Y)=E_{S (X)}(Y^2)-(E_{S (X)}Y)^2$ we obtain

(1) $latex E_{S (X)} (Y^2) \geqslant (E_{S (X)} Y)^2 .$

Step 3. To prove the theorem, we note that by the Law of Iterated Expectation

$latex {MSE} (A (X)) = E [E_{S (X)} (A (X) - \theta)^2] $

(letting $latex Y = A (X) - \theta$ in (1))

$latex \geqslant E { [E_{S (X)} (A (X) - \theta)]^2 } $

(using linearity of means)

=$latex E { [E_{S (X)}  A (X) - E_{S (X)} \theta)]^2 }$

(the expected value of a constant is that constant and applying the definition of $latex B (X)$)

=$latex E [(B (X) -  \theta)^2] ={MSE} (B (X))$.

Remark. If $latex A (X)$ is unbiased, then $latex B (X)$ is too:
$latex E [B (X)] = E [E_{S (X)} A (X)] = E [A (X)] = \theta$.
Therefore, for unbiased $latex A (X)$, the Rao-Blackwell inequality becomes $latex {Var} (B (X))\leqslant {Var} (A (X))$.