Raising the bar

Chapter 8. Estimation: One or Two Populations

Unit 8.4.3. Independent samples. Case II

When $latex \sigma_X^2 = \sigma_Y^2 = \sigma^2$ from Eq. (8.8) we see that

$latex \sigma(\bar X - \bar Y) = \sigma \sqrt{\frac{1}{n_X} + \frac{1}{n_Y}}. \qquad (\textrm{Eq. 8.9})$

The question therefore is how one combines the information contained in two samples to better estimate $latex \sigma$? Theorem 7.1 provides the leading idea in this and the next section.

Idea. The population variance $latex \sigma^2$, sample variance $latex s^2$ and degrees of freedom $latex r$ should satisfy the equation

$latex \chi_r^2 = \frac{r s^2}{\sigma^2}. \qquad (\textrm{Eq. 8.10})$

To find the right estimator we use an inductive argument. By Theorem 7.1 we have equations

$latex \chi_{n_X - 1}^2 = \frac{(n_X - 1) s_X^2}{\sigma^2}, \quad \chi_{n_Y - 1}^2 = \frac{(n_Y - 1) s_Y^2}{\sigma^2}.$

Summing them and using the fact that the two chi-square distributions involved are independent we get

$latex \frac{(n_X - 1) s_X^2}{\sigma^2} + \frac{(n_Y - 1) s_Y^2}{\sigma^2} = \chi_{n_X - 1}^2 + \chi_{n_Y - 1}^2 = \chi_{n_X + n_Y - 2}^2. \qquad (\textrm{Eq. 8.11})$

From Eq. (8.10) and Eq. (8.11) we see that the estimator of $latex \sigma^2$ should be defined by

$latex s_p^2 = \frac{(n_X - 1) s_X^2 + (n_Y - 1) s_Y^2}{n_X + n_Y - 2}. \qquad (\textrm{Eq. 8.12})$

Definition 8.7 (Pooled estimator). The expression in Eq. (8.12) is called a pooled estimator.

Exercise 8.8. (1) In addition to Eq. (8.10), the pooled estimator has the usual unbiasedness property $latex Es_p^2 = \sigma^2$.

(2) Besides, it gives more weight to the sample with the larger size.

(3) Obviously, Eq. (8.9) is estimated by $latex s_p \sqrt{\frac{1}{n_X} + \frac{1}{n_Y}}$. Following the general statistical idea, in

$latex z_0 = \frac{\bar X - \bar Y}{\sigma(\bar X - \bar Y)} = \frac{\bar X - \bar Y}{\sigma \sqrt{\frac{1}{n_X} + \frac{1}{n_Y}}}$

replace $latex \sigma$ by the pooled estimator to obtain

$latex t = \frac{\bar X - \bar Y}{s_p \sqrt{\frac{1}{n_X} + \frac{1}{n_Y}}}.$

Show that this $latex t$ is distributed as $latex t_{n_X + n_Y - 2}$.

Proof. The proof of (1) is straightforward.

For (2), we write Eq. (8.12) as

$latex s_p^2 = \alpha s_X^2 + \beta s_Y^2, \textrm{ where } \alpha = \frac{n_X - 1}{n_X + n_Y - 2},\ \beta = \frac{n_Y - 1}{n_X + n_Y - 2}.$

Here $latex \alpha + \beta = 1$, which means that $latex s_p^2$ belongs to the segment connecting $latex s_X^2, s_Y^2$, and that $latex \alpha$ is close to 1 (and large relative to $latex \beta$) if $latex n_X$ is large relative to $latex n_Y$. This can be seen by rearranging $latex \alpha$ as $latex \alpha = \frac{1 - 1/n_X}{1 + (n_Y - 2)/n_X}$.

The proof of (3) follows Exercise 8.5. $latex \blacksquare$

Exercise. Find the confidence interval for $latex \mu_X - \mu_Y$.

Solution.

Step 1. Calculate $latex \bar X, \bar Y, s_X^2, s_Y^2, s_p^2$.

Step 2. Choose $latex \alpha$ and find the critical value of the $latex t$ statistic $latex t_{n_X + n_Y - 2, \alpha/2}$.

Step 3. The confidence interval is

$latex P(\bar X - \bar Y - ME < \mu_X - \mu_Y < \bar X - \bar Y + ME) = 1 - \alpha,$

where $latex ME = t_{n_X + n_Y - 2, \alpha/2} s_p \sqrt{\frac{1}{n_X} + \frac{1}{n_Y}}. \quad \blacksquare$

Previous: Independent samples, Case I | Next: Independent samples, Case III (Satterthwaite's formula)

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 8. Estimation: One or Two Populations

Unit 8.3.3. Confidence interval for population proportion

From Eq. (5.14) we know that $latex E\hat p = p,\,\, V(\hat p) = \frac{p(1-p)}{n}.$ By the central limit theorem the variable

$latex \frac{\hat p - E\hat p}{\sigma_{\hat p}} = \frac{\hat p - p}{\sqrt{p(1-p)/n}} \qquad (\textrm{Eq. 8.6})$

is approximately normal if $latex n$ is large enough (the rule of thumb is $latex np(1-p) > 9$). The problem with Eq. (8.6) is that in the denominator we have an unknown $latex p$. Following the general statistical idea, we replace it by $latex \hat p$ and obtain

$latex z_n = \frac{\hat p - p}{\sqrt{\hat p(1-\hat p)/n}}. \qquad (\textrm{Eq. 8.7})$

Exercise 8.7. Using Eq. (8.7) derive the confidence interval for $latex p$. What is the expression for the margin of error?

Previous: Confidence interval for the population mean: unknown variance | Next: Matched pairs

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.14. Distribution function

The full name from NCT is a cumulative distribution function but I am going to stick to the short name (used in more advanced texts). This is one of the topics most students don't get on the first attempt.

Exercise 5.9. Electricity consumption sharply increases when everybody starts using air conditioners, and this happens when temperature exceeds $latex 20\,^{\circ}\mathrm{C}$. The utility company would like to know the likelihood of a jump in electricity consumption tomorrow.

1. Consider the probability $latex P(T \le 15)$ that $latex T$ will not exceed $latex 15\,^{\circ}\mathrm{C}$. How does it relate to the probability $latex P(T \le 20)$?
2. Suppose in the expression $latex P(T \le t)$ the real number $latex t$ increases to $latex +\infty$. What happens to the probability?
3. Now think about $latex t$ going to $latex -\infty$. Then what happens to $latex P(T \le t)$?

Distribution function. Let $latex X$ be a random variable and $latex x$ a real number. The distribution function $latex {F_X}$ of $latex X$ is defined by $latex {F_X}(x) = P(X \le x).$

Properties

1. $latex {F_X}(x)$ is the probability of the event $latex \{ X \le x\}$, so the value $latex {F_X}(x)$ belongs to $latex [0,1]$.
2. As the event becomes wider, the probability increases. This property is called monotonicity and is formally written as follows: if $latex {x_1} \le {x_2}$, then $latex \{ X \le {x_1}\} \subset \{ X \le {x_2}\}$ and $latex {F_X}({x_1}) \le {F_X}({x_2})$.
3. As $latex x$ goes to $latex +\infty$, the event $latex { X \le x}$ approaches a sure event $latex \{X < +\infty\} = \mathbb{R}$ and $latex {F_X}(x)$ tends to 1.
4. As $latex x$ goes to $latex -\infty$, the event $latex \{ X \le x\}$ approaches an impossible event $latex \{X < -\infty\} = \varnothing$ and $latex {F_X}(x)$ tends to 0.
5. From (i)-(iv) the graph of the distribution function in Figure 5.4 emerges.

Figure 5.4. Distribution function shape.Most values of $latex X$ are concentrated where the growth of $latex {F_X}(x)$ is the fastest (say, between $latex a$ and $latex b$).

1. In many applications we are interested in probability of an event that $latex X$ takes values in an interval $latex \{ a < X \le b\}$. Such probability can be expressed in terms of the distribution function. Just apply the additivity rule to the set equation $latex \{ -\infty < X \le b\} = \{ -\infty < X \le a\} \cup \{ a < X \le b\}$ to get $latex {F_X}(b) = {F_X}(a) + P(a < X \le b)$ and, finally,

$latex P(a < X \le b) = {F_X}(b) - {F_X}(a). \qquad (\textrm{Eq. 5.16})$

Interval formula. Eq. (5.16) can be called an interval formula. In terms of Figure 5.4 this means subtracting one area from another.

1. The definition of $latex {F_X}(x)$ equally applies to discrete and continuous variables. However, in case of a discrete random variable the argument $latex x$ is often restricted to the values taken by the variable. From the working definition of a discrete random variable it should be clear that, if the values are ordered, then

$latex {F_X}({x_1}) = P(X \le {x_1}) = P(X = {x_1}) = {p_1},$

$latex {F_X}({x_2}) = P(X \le {x_2}) = P(X = {x_1}) + P(X = {x_2}) = {p_1} + {p_2},$

$latex ...$

$latex {F_X}({x_n}) = {p_1} + ... + {p_n}. \qquad (\textrm{Eq. 5.16.1})$

The sums arising in Eq. (5.16.1) are called cumulative probabilities. We summarize our findings in Table 5.10.

Table 5.10. Probabilities and cumulative probabilities.

Probabilities can be restored from the cumulative probabilities:

$latex {p_1} = {c_1},\,\,{p_2} = {c_2} - {c_1},\,...,\,{p_n} = {c_n} - {c_{n - 1}}.$

1. In Chapter 6 we shall see that a continuous random variable takes any fixed value with probability zero. Therefore for such variables $latex {F_X}(x) = P(X \le x) = P(X < x).$

Previous: Distribution of the binomial variable | Next: Poisson distribution

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.13. Distribution of the binomial variable

Case $latex n = 3$. When you need to recall what a binomial variable is, you start with "it is a sum" and then disclose the rest. Let us derive the distribution of $latex {S_3}$. The idea is to list all possible combinations of values of $latex {X_1},\,{X_2},\,{X_3}$, find the corresponding values of $latex {S_3}$ and then calculate their probabilities. This is done in two steps. Remember that the objective is to get the "values plus probabilities" table.

In mathematical exposition ideas sometimes are not stated. The presumption is that you get the idea after reading the proof. In your mind, ideas should always go first.

Step 1. The first step is realized in Table 5.8.

Table 5.8. Possible outcomes and their probabilities.

Here we used the multiplication rule. For example, the probability $latex p{q^2}$ in the second row is obtained like this: by independence

$latex P({X_1} = 1,\,{X_2} = 0,\,{X_3} = 0) = P({X_1} = 1)P({X_2} = 0)P({X_3} = 0) = p{q^2}.$

Step 2. The next step is to collect equal values of $latex {S_3}$. This is done using the additivity rule. For example, the events (1,0,0), (0,1,0) and (0,0,1) in rows 2--4 are mutually exclusive and therefore

$latex P({S_3} = 1) = P((1,0,0) \cup (0,1,0) \cup (0,0,1))$

$latex = P(1,0,0) + P(0,1,0) + P(0,0,1) = 3p{q^2}.$

Thus the result is Table 5.9.

Table 5.9. Distribution of $latex {S_3}$.

Exercise 5.8. Using Table 5.9 find $latex E{S_3},\,\,V({S_3})$.

Case of general $latex n$. The first thing to understand is that $latex {S_n}$ takes all integer values $latex x$ from 0 to $latex n$, inclusive, and for each of these values we need to figure out the probability $latex P({S_n} = x)$ (that's what Table 5.9 is about in case $latex n = 3$).

Step 1. The equation $latex {S_n} = x$ tells us that in a sequence $latex ({X_1},...,{X_n})$ of zeros and unities there are $latex x$ unities and $latex n - x$ zeros. The probability of ONE such sequence is $latex {p^x}{q^{n - x}}$ by the multiplication rule.

Step 2. We have to find out how many such sequences exist for the given $latex x$. This question can be rephrased as follows: in how many different ways $latex x$ places (where the unities will be put) can be chosen out of $latex n$ places? Since the order doesn't matter, the answer is: in $latex C_x^n$ ways. Because each of these ways has the same probability $latex {p^x}{q^{n - x}}$ and they are mutually exclusive, by the additivity rule

$latex P({S_n} = x) = C_x^n{p^x}{q^{n - x}},\,\,x = 0,...,n. \qquad (\textrm{Eq. 5.15})$

Previous: An obvious and handy result | Next: Distribution function

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.11. Bernoulli variable

Intuitive definition. A Bernoulli variable is an unfair coin.

Formal definition. A Bernoulli variable is defined by Table 5.7 where $latex 0 < p < 1$. Often it is convenient to denote $latex q = 1 - p$.

Table 5.7. Bernoulli variable definition.

Property 1. Mean of the Bernoulli variable: $latex EB = 1 \cdot p + 0 \cdot (1 - p) = p$.

Property 2. Variance of the Bernoulli variable. Using the shortcut for variance,

$latex V(B) = E{B^2} - {(EB)^2} = {1^2}p + {0^2}(1 - p) - {p^2} = p - {p^2} = p(1 - p).$

$latex V(B)$ is a parabola with branches looking downward. It vanishes at $latex p = 0$ and $latex p = 1$ and is maximized at $latex p = 1/2$, which corresponds to a fair coin.

Some facts about sampling not many people will tell you.

Here we provide an example of an unfair coin and use it to discuss the notions of population and sample.

Take any book in English. All alphabetic characters in it are categorized as either consonants or vowels. Denote by $latex p$ the percentage of consonants. We assign 1 to a consonant and 0 to a vowel. The experiment consists in randomly selecting an alphabetic character from the book. Then the outcome is 1 with probability $latex p$ and 0 with probability $latex q$. We have a random variable described by Table 5.7. In this context, the population is the set of consonants and vowels, along with their percentages $latex p$ and $latex q$. This description fits the definition from NCT:

Population and sample. A population is a complete set of all items that interest the investigator. A sample is an observed subset of population values with sample size given by $latex n$.

It is important to imagine the sampling process step by step. To obtain a sample in our example we can randomly select $latex n$ characters and write the sample as $latex {X_1},...,{X_n}$ where some $latex X$'s are unities and others are zeros. The problem is that "random selection" can be understood in different ways. We consider the most important two.

Case I. We randomly select $latex {X_1}$ from the whole book. Suppose it's the first character in the table of contents, which is "C" in NCT. Then $latex {X_1} = 1$. Then we randomly select $latex {X_2}$ again from the whole book. It may well be the same "C" in the table of contents, in which case $latex {X_2} = 1$. We continue like this following two rules: (1) each selection is random and (2) the selection is made from the whole book.

Sampling with replacement. The second condition defines what is called a sampling with replacement: to restore the population, selected elements are returned to it.

After we have obtained a sample of size $latex n$, we can form a sum of the observed values

$latex {S_n} = {X_1} + ... + {X_n} \qquad (\textrm{Eq. 5.10})$

and a sample mean

$latex \bar X = \frac{{{X_1} + ... + {X_n}}}{n} = \frac{{{S_n}}}{n}. \qquad (\textrm{Eq. 5.11})$

After we obtain an actual sample, there will be nothing random. $latex {S_n}$, being a sum of unities and zeros, will be some integer from the range $latex [0,n]$ and $latex \bar X$ will be some fraction from $latex [0,1]$.

The subtlety of the theoretical argument in statistics is that $latex {X_1},...,{X_n}$ are treated not as realized (ex-post) values but as theoretical (ex-ante) random variables that are associated with the drawings from the population.

Let us rewind the movie and think about the selection process again. The first time you don't actually select a character, you just THINK about randomly selecting it from the whole book. The associated random variable is denoted $latex {X_1}$ and, clearly, is described by the same table as the Bernoulli variable $latex B$. Similarly, you think about what possibilities you would face in all other cases and come to the conclusion that all drawings are described by the same table, just the names of the variables will be different.

Identically distributed random variables. Random variables that have the same distributions (that is, values plus probabilities) are called identically distributed.

Comments. (1) Since "values plus probabilities" is all one needs to find the mean and variance of a random variable, identically distributed variables have identical means and variances.

(2) Instead of saying "$latex {X_1},...,{X_n}$ are identically distributed" we can equivalently say "$latex {X_1},...,{X_n}$ are drawn from the same population". In this case the population we draw the sample from is called a parent population.

(3) By construction, in Case I the variables are independent. The common abbreviation for "independent identically distributed" is i.i.d.

(4) In the sum Eq. (5.10) the variables $latex {X_1},...,{X_n}$ may take independently values 0 and 1. If all are zeros, $latex {S_n}$ is zero. If all are unities, $latex {S_n}$ will be equal to $latex n$. In general, $latex {S_n}$ may take any integer value from 0 to $latex n$, inclusive. To emphasize that we deal with variables, in the equation

$latex \mathop {{S_n}}\limits_{0,...,n} = \mathop {{X_1}}\limits_{0,1} + ... + \mathop {{X_n}}\limits_{0,1}$

below the variables I write their possible values. Now the next definition must be clear.

Binomial variable. A binomial variable is a sum of $latex n$ independent identically distributed Bernoulli variables.

Case II. Think about an opinion poll on a simple question requiring a simple answer: Yes or No. Suppose everybody has a definite opinion on the matter, and denote $latex p$ the percentage of voters who have an affirmative answer. A team is sent out to conduct a survey. They record 1 for Yes and 0 for No. The population will be again Bernoulli but there are problems with the sample. It doesn't make sense to ask the same person twice, so the elements in the sample cannot be repeated (this is sampling without replacement). In essence, after having randomly selected the first individual, you cross out that individual from the population before selecting the next one. This implies two things: the observations are dependent and their distributions are not identical.

The general statistical principle is that removing one element from a large population does not change its characteristics very much. So in the case under consideration, the observations can be assumed approximately i.i.d.

Conclusion. If $latex {X_1},...,{X_n}$ are drawn from the same parent population and $latex {\mu _X}$ and $latex {\sigma _X}$ denote the mean and standard deviation of the parent population, then

$latex E{X_1} = ... = E{X_n} = {\mu _X},\,\,\sigma ({X_1}) = ... = \sigma ({X_n}) = {\sigma _X}. \qquad (\textrm{Eq. 5.12})$

Previous: Standardization | Next: An obvious and handy result

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.

Conditional expectation as a projector

This topic was discussed here and here. Looks like my explanations were not sufficient for some students, so here I give it a second try.

Examples of projectors

The name "projector" is almost self-explanatory. Imagine a point and a plane in the three-dimensional space (see the Wolfram embedded notebook below). Draw a perpendicular from the point to the plane. The intersection of the perpendicular with the plane is the point's projection onto that plane. Note that if the point already belongs to the plane, its projection equals the point itself. Besides, instead of projecting onto a plane we can project onto a straight line.

The above description translates into the following equations. For any point $latex (x,y,z)\in R^{3}$ define

(1) $latex P_{1}(x,y,z)=(x,0,0)$ and $latex P_{2}(x,y,z)=(x,y,0).$

In words: $latex P_{1}$ leaves the first two coordinates of the point unchanged and sets the third one to zero. (Describe yourself what $latex P_{2}$ does.)

$latex P_{2}$ projects $latex R^{3}$ onto the plane $latex L_{2}=\{(x,y,0):x,y\in R\}$ (which is two-dimensional) and $latex P_{1}$ projects $latex R^{3}$ onto the straight line $latex L_{1}=\{(x,0,0):x\in R\}$ (which is one-dimensional).

Property 1. Double application of a projector amounts to single application.

Proof. Using (1) three times we get

(1) $latex P_{2}[P_{2}(x,y,z)]=P_{2}(x,y,0)=(x,y,0)=P_{2}(x,y,z).$

Similarly, $latex P_{1}[P_{1}(x,y,z)]=P_{1}(x,y,z).$

Property 2. A successive application of two different projectors yields the projection onto a subspace of a smaller dimension.

Proof. If we apply first $latex P_{2}$ and then $latex P_{1}$, the result is

(2) $latex P_{1}[P_{2}(x,y,z)]=P_{1}(x,y,0)=(x,0,0)=P_{1}(x,y,z).$

If we change the order of projectors, we have

(3) $latex P_{2}[P_{1}(x,y,z)]=P_{2}(x,0,0)=(x,0,0)=P_{1}(x,y,z).$

Informational interpretation

If we know $latex P_{2}(x,y,z)$ then we know more about the original point $latex (x,y,z)$ than if we know $latex P_{1}(x,y,z).$ Thus, $latex P_{1}$ projects onto a smaller information set than $latex P_{2}.$ Equations (2) and (3) can be summarized as follows: the smaller info set wins.

Exercise 1. Show that both projectors are linear.

Exercise 2. Like any other linear operator in a Euclidean space, these projectors are given by some matrices. What are they?

The simple truth about conditional expectation

In the time series setup, we have a sequence of information sets

$latex ... \subset I{t}\subset I_{t+1}\subset ...$

where $latex t$ are points in time (naturally, with time the amount of available information increases). Denote

$latex E_tX=E(X|I_t)$

the expectation of $latex X$ conditional on $latex I_{t}$.

Statement. For each $latex t$, $latex E_{t}$ is a projector onto the space of random functions that depend only on the information set $latex I_{t}$.

The proof is too complex to discuss here but the implications are immediate.

Property 3. Applying Property 1, we see that double application of conditional expectation gives the same result as single application:

(4) $latex E_t(E_tX)=E_tX$

($latex E_{t}X$ is already a function of $latex I_{t}$, so conditioning it on $latex I_{t}$ doesn't change it).

Property 4. A successive conditioning on two different information sets is the same as conditioning on the smaller one:

(5) $latex E_tE_{t+1}X=E_tX,$

(6) $latex E_{t+1}E_{t}X=E_{t}X.$

(5) and (6) apply also in the case $latex E_t=E$ because for the unconditional expectation the information set is empty.

These lead to more general equations. Suppose, $latex s\leq t-1<t.$ Then using (5) many times, we get

$latex E_{s}E_{t}X=E_{s}E_{s+1}E_{t}X=...=E_{s}E_{s+1}...E_{t-1}E_{t}X=E_{s}E_{s+1}...E_{t-1}X=...=E_{s}X.$

Property 5. Conditional expectation is a linear operator: for any random variables $latex X,Y$ and random variables $latex a,b$ that depend only on $latex I_t$ we have

$latex E_t(aX+bY)=aE_tX+bE_tY.$

I prefer to use different names for (4)-(6). I call (4) a projector property. (5) is known as the Law of Iterated Expectations, see my post on the informational aspect for more intuition. (6) holds simply because at time $latex t+1$ the expectation $latex E_{t}X$ is known and behaves like a constant.

Summary. (4)-(6) can be combined into one property. The smaller information set wins: for any points in time $latex s,t$ we have $latex E_{s}E_{t}X=E_{\min \{s,t\}}X.$

In the next Wolfram embedded notebook you can manipulate both graphs to get a better angle of view.

Distribution of the estimator of the error variance

If you are reading the book by Dougherty: this post is about the distribution of the estimator  $latex s^2$ defined in Chapter 3.

Consider regression

(1) $latex y=X\beta +e$

where the deterministic matrix $latex X$ is of size $latex n\times k,$ satisfies $latex \det
\left( X^{T}X\right) \neq 0$ (regressors are not collinear) and the error $latex e$ satisfies

(2) $latex Ee=0,Var(e)=\sigma ^{2}I$

$latex \beta $ is estimated by $latex \hat{\beta}=(X^{T}X)^{-1}X^{T}y.$ Denote $latex P=X(X^{T}X)^{-1}X^{T},$ $latex Q=I-P.$ Using (1) we see that $latex \hat{\beta}=\beta +(X^{T}X)^{-1}X^{T}e$ and the residual $latex r\equiv y-X\hat{\beta}=Qe.$ $latex \sigma^{2}$ is estimated by

(3) $latex s^{2}=\left\Vert r\right\Vert ^{2}/\left( n-k\right) =\left\Vert
Qe\right\Vert ^{2}/\left( n-k\right) .$

$latex Q$ is a projector and has properties which are derived from those of $latex P$

(4) $latex Q^{T}=Q,$ $latex Q^{2}=Q.$

If $latex \lambda $ is an eigenvalue of $latex Q,$ then multiplying $latex Qx=\lambda x$ by $latex Q$ and using the fact that $latex x\neq 0$ we get $latex \lambda ^{2}=\lambda .$ Hence eigenvalues of $latex Q$ can be only $latex 0$ or $latex 1.$ The equation $latex tr\left( Q\right) =n-k$
tells us that the number of eigenvalues equal to 1 is $latex n-k$ and the remaining $latex k$ are zeros. Let $latex Q=U\Lambda U^{T}$ be the diagonal representation of $latex Q.$ Here $latex U$ is an orthogonal matrix,

(5) $latex U^{T}U=I,$

and $latex \Lambda $ is a diagonal matrix with eigenvalues of $latex Q$ on the main diagonal. We can assume that the first $latex n-k$ numbers on the diagonal of $latex Q$ are ones and the others are zeros.

Theorem. Let $latex e$ be normal. 1) $latex s^{2}\left( n-k\right) /\sigma ^{2}$ is distributed as $latex \chi _{n-k}^{2}.$ 2) The estimators $latex \hat{\beta}$ and $latex s^{2}$ are independent.

Proof. 1) We have by (4)

(6) $latex \left\Vert Qe\right\Vert ^{2}=\left( Qe\right) ^{T}Qe=\left(
Q^{T}Qe\right) ^{T}e=\left( Qe\right) ^{T}e=\left( U\Lambda U^{T}e\right)
^{T}e=\left( \Lambda U^{T}e\right) ^{T}U^{T}e.$

Denote $latex S=U^{T}e.$ From (2) and (5)

$latex ES=0,$ $latex Var\left( S\right) =EU^{T}ee^{T}U=\sigma ^{2}U^{T}U=\sigma ^{2}I$

and $latex S$ is normal as a linear transformation of a normal vector. It follows that $latex S=\sigma z$ where $latex z$ is a standard normal vector with independent standard normal coordinates $latex z_{1},...,z_{n}.$ Hence, (6) implies

(7) $latex \left\Vert Qe\right\Vert ^{2}=\sigma ^{2}\left( \Lambda z\right)
^{T}z=\sigma ^{2}\left( z_{1}^{2}+...+z_{n-k}^{2}\right) =\sigma ^{2}\chi
_{n-k}^{2}.$

(3) and (7) prove the first statement.

2) First we note that the vectors $latex Pe,Qe$ are independent. Since they are normal, their independence follows from

$latex cov(Pe,Qe)=EPee^{T}Q^{T}=\sigma ^{2}PQ=0.$

It's easy to see that $latex X^{T}P=X^{T}.$ This allows us to show that $latex \hat{\beta}$ is a function of $latex Pe$:

$latex \hat{\beta}=\beta +(X^{T}X)^{-1}X^{T}e=\beta +(X^{T}X)^{-1}X^{T}Pe.$

Independence of $latex Pe,Qe$ leads to independence of their functions $latex \hat{\beta}$ and $latex s^{2}.$

Sylvester's criterion

Exercise 1. Suppose $latex A=CBC^{T},$ where $latex B=diag[d_{1},...,d_{n}]$ and $latex \det C\neq 0$. Then $latex A$ is positive if and only if all $latex d_{i}$ are positive.

Proof. For any $latex x\neq 0$ we have $latex x^{T}Ax=x^{T}CBC^{T}x=(C^{T}x)^{T}B(C^{T}x).$ Let $latex y=C^{T}x\neq 0.$ Then $latex x^{T}Ax=y^{T}By=\sum_{j}d_{j}y_{j}^{2}.$ This is positive for all $latex y\neq 0$ if and only if $latex \min_{j}d_{j}>0.$

Exercise 2 (modified Gaussian elimination). Suppose that $latex A$ is a real symmetric matrix with nonzero leading principal minors $latex d_{1},...,d_{n}$. Then $latex B=CAC^{T},$ where $latex B=diag[d_{1},d_{2}/d_{1},...,d_{n}/d_{n-1}]$ and $latex \det C=1$.

Proof. Review the transformation applied in Exercise 1 to obtain a triangular form. In that exercise, we eliminated element $latex a_{21}$ below $latex a_{11}$ by premultiplying $latex A$ by the matrix $latex C=I-\frac{a_{21}}{a_{11}} e_{2}^{T}e_{1}.$ Now after this we can post-multiply $latex A$ by the matrix $latex C^{T}=I-\frac{a_{21}}{a_{11}}e_{1}^{T}e_{2}.$ Because of the assumed symmetry of $latex A,$ we have $latex C^{T}=I-\frac{a_{12}}{a_{11}}e_{1}^{T}e_{2},$ so this will eliminate element $latex a_{12}$ to the right of $latex a_{11}$, see Exercise 2. Since in the first column $latex a_{21}$ is already $latex 0$, the diagonal element $latex a_{22}$ will not change.

We can modify Exercise 1 by eliminating $latex a_{1j}$ immediately after eliminating $latex a_{j1}.$ The right sequencing of transformations is necessary to be able to apply Exercise 1: the matrix used for post-multiplication should be the transpose of the matrix used for premultiplication. If $latex C=C_{m}...C_{1},$ then $latex C^{T}=C_{1}^{T}...C_{m}^{T},$ which means that premultiplication by $latex C_{i}$ should be followed by post-multiplication by $latex C_{i}^{T}.$ In this way we can make zero all off-diagonal elements. The resulting matrix $latex B=diag[d_{1},d_{2}/d_{1},$ $latex ...,d_{n}/d_{n-1}]$ is related to $latex A$ through $latex B=CAC^{T}.$

Theorem (Sylvester) Suppose that $latex A$ is a real symmetric matrix. Then $latex A$ is positive if and only if all its leading principal minors are positive.

Proof. Let's assume that all leading principal minors are positive. By Exercise 2, we have $latex A=C^{-1}B(C^{-1})^{T}$ where $latex \det C=1.$ It remains to apply Exercise 1 above to see that $latex A$ is positive.

Now suppose that $latex A$ is positive, that is $latex x^{T}Ax=\sum_{i,j=1}^{n}a_{ij}x_{i}x_{j}>0$ for any $latex x\neq 0.$ Consider cut-off matrices $latex A_{k}=\left( a_{ij}\right) _{i,j=1}^{k}.$ The corresponding cut-off quadratic forms $latex x^{T}A_{k}x=\sum_{i,j=1}^{k}a_{ij}x_{i}x_{j},$ $latex k=1,...,n,$ are positive for nonzero $latex x\in R^{k}.$ It follows that $latex A_{k}$ are non-singular because if $latex x\in N(A_{k}),$ then $latex x^{T}A_{k}x=0.$ Hence their determinants $latex d_{k}=\det A_{k},$ $latex k=1,...,n,$ are nonzero . This allows us to apply the modified Gaussian elimination (Exercise 2) and then Exercise 1 with $latex B=diag[d_{1},...,d_{n}/d_{n-1}].$ By Exercise 1 consecutively $latex d_{1}>0,$ $latex d_{2}>0,...,$ $latex d_{n}>0.$

Exercise 3. $latex A$ is negative if and only if the leading principal minors change signs, starting with minus: $latex d_{1}<0,$ $latex d_{2}>0,$ $latex d_{3}<0,...$

Proof. By definition, $latex A$ is negative if $latex -A$ is positive. Because of homogeneity of determinants, when we pass from $latex A$ to $latex -A,$ the minor of order $latex k$ gets multiplied by $latex (-1)^{k}.$ Thus, by Sylvester's criterion $latex A$ is negative if and only if $latex (-1)^{k}d_{k}>0,$ as required.

Gaussian elimination method

Consider the system

$latex a_{11}x+a_{12}y+a_{13}z=a_1,$

$latex a_{21}x+a_{22}y+a_{23}z=a_2,$

$latex a_{31}x+a_{32}y+a_{33}z=a_3.$

If the first equation is nontrivial, then one of the coefficients is different from zero. Suppose it is $latex a_{11}.$ Adding the first equation multiplied by $latex -a_{21}/a_{11}$ to the second one we eliminate $latex x$ from it. Similarly, adding the first equation multiplied by $latex -a_{31}/a_{11}$ to the third one we eliminate $latex x$ from it. The system becomes

$latex a_{11}x+a_{12}y+a_{13}z=a_1,$

$latex b_{22}y+b_{23}z=b_{2},$

$latex b_{32}y+b_{33}z=b_{3},$

where $latex b$ with indexes stands for new numbers.

If $latex b_{22}\neq 0,$ we can use the second equation to eliminate $latex y$ from the third equation and the result will be

$latex a_{11}x+a_{12}y+a_{13}z=a_{1},$

$latex b_{22}y+b_{23}z=b_{2},$

$latex c_{33}z=c_{3}$

with some new $latex c$'s. If $latex c_{33}\neq 0,$ we can solve the system backwards, finding first $latex z$ from the third equation, then $latex y$ from the second and, finally $latex x$ from the first.

Notice that for the method to work it does not matter what happens with the vector at the right of the system. It only matters what happens to $latex A.$ That's why we focus on transformations of $latex A.$

Theoretical treatment

Exercise 1. Denote

$latex d_{1}=a_{11},\ d_{2}=\det \left(\begin{array}{cc}
a_{11} & a_{12} \\a_{21} & a_{22}\end{array}\right) ,..., \ d_{n}=\det A$

the leading principal minors of $latex A.$ If all of them are different from zero, then the Gaussian method reduces $latex A$ (by way of premultiplication of $latex A$ by elementary matrices) to the triangular matrix

$latex B=\left(\begin{array}{cccc}b_{11} & b_{12} & ... & b_{1n} \\
0 & b_{22} & ... & b_{2n} \\... & ... & ... & ... \\0 & 0 & ... & b_{nn}\end{array}\right) $

where $latex b_{11}=d_{1},\ b_{22}=d_{2}/d_{1},...,\ b_{nn}=d_{n}/d_{n-1}.$

Proof by induction. Let $latex n=2.$ Premultiplying $latex A=\left(\begin{array}{cc}
a_{11} & a_{12} \\a_{21} & a_{22}\end{array}\right) $ by $latex I-\frac{a_{21}}{a_{11}}e_{2}^{T}e_{1}$ we get the desired result:

$latex B=\left(\begin{array}{cc}a_{11} & a_{12} \\0 & a_{22}-\frac{a_{21}a_{12}}{a_{11}}
\end{array}\right) =\left(\begin{array}{cc}d_{1} & a_{12} \\0 & \frac{d_{2}}{d_{1}}
\end{array}\right) .$

Now let the statement hold for $latex n-1.$ By the induction assumption we can start with the matrix of form

$latex A=\left(\begin{array}{ccccc}d_{1} & b_{12} & ... & b_{1n} & b_{1n} \\
0 & d_{2}/d_{1} & ... & b_{2n} & b_{2n} \\... & ... & ... & ... & ... \\0 & 0 & ... & d_{n-1}/d_{n-2} & b_{2n} \\
a_{n1} & a_{n2}&... & a_{n-1,n} & a_{nn} \end{array}\right) .$

Using the condition that all $latex d_{1},...,d_{n-1}$ are different from zero, we can make zero the elements $latex a_{n1}, ..., a_{n-1,n}$. The result is:

$latex B=\left(\begin{array}{ccccc}d_{1} & b_{12} & ... & b_{1n} & b_{1n} \\
0 & d_{2}/d_{1} & ... & b_{2n} & b_{2n} \\... & ... & ... & ... & ... \\0 & 0 & ... & d_{n-1}/d_{n-2} & b_{2n} \\
0 & 0 & ... & 0 & b_{nn}\end{array}\right) .$

The determinant of a triangular matrix equals the product of diagonal elements because of the cross-out rule:

(1) $latex \det B=d_{1}(d_{2}/d_{1})...(d_{n-1}/d_{n-2})b_{nn}=d_{n-1}b_{nn}$

(for example, if a product contains $latex b_{1n}$, you should cross out the first row and then the product should contain one of the zeros below $latex d_1$).

On the other hand, $latex B$ is a result of premultiplication by elementary matrices: $latex B=C_{1}...C_{n-1}A$ where by Exercise 1 $latex \det C_{i}=1$ for all $latex i. $ Hence,

(2) $latex \det B=\det A=d_{n}.$

Combining (1) and (2) we get $latex b_{nn}=d_{n}/d_{n-1},$ which concludes the inductive argument.

Elementary transformations

Here we look at matrix representations of transformations called elementary.

Exercise 1. Let $latex e_{i}$ denote the $latex i$-th unit row vector and let $latex A$ be an arbitrary matrix. Then

a) premultiplication of $latex A$ by $latex e_{j}$ cuts out of $latex A$ the $latex j$-th row $latex A_{j}.$

b) Premultiplication of $latex A$ by $latex e_{i}^{T}e_{j}$ puts the row $latex A_{j}$ into the $latex i$-th row of the null matrix.

c) Premultiplication of $latex A$ by $latex I+ce_{i}^{T}e_{j}$ adds row $latex A_{j}$ multiplied by $latex c$ to row $latex A_{i},$ without changing the other rows of $latex A.$

d) The matrix $latex I+ce_{i}^{T}e_{j}$ has determinant 1.

Proof. a) It's easy to see that

$latex e_{j}A=\left( 0...0~1~0...0\right) \left(\begin{array}{ccc}a_{11} & ... & a_{1n} \\
... & ... & ... \\a_{j1} & ... & a_{jn} \\... & ... & ... \\a_{n1} & ... & a_{nn}\end{array}
\right) =\left(\begin{array}{ccc}a_{j1} & ... & a_{jn}\end{array}\right) =A_{j}.$

b) Obviously,

$latex e_{i}^{T}e_{j}A=\left(\begin{array}{c}0 \\... \\0 \\1 \\0 \\... \\0\end{array}
\right) \left(\begin{array}{ccc}a_{j1} & ... & a_{jn}\end{array}
\right) =\left(\begin{array}{ccc}0 & ... & 0 \\... & ... & ... \\a_{j1} & ... & a_{jn} \\... & ... & ... \\
0 & ... & 0\end{array}
\right) =\left(\begin{array}{c}\Theta \\A_{j} \\\Theta\end{array}
\right) $

($latex A_{j}$ in the $latex i$-th row, $latex \Theta$ denotes null matrices of conformable dimensions)

c) $latex (I+ce_{i}^{T}e_{j})A=A+ce_{i}^{T}e_{j}A=\left(\begin{array}{c}
A_{1} \\... \\A_{i} \\... \\A_{n}\end{array}
\right) +\left(\begin{array}{c}\Theta \\... \\cA_{j} \\... \\\Theta\end{array}
\right) =\left(\begin{array}{c}A_{1} \\... \\A_{i}+cA_{j} \\
... \\A_{n}\end{array}\right) .$

d) The matrix $latex A=I+ce_{i}^{T}e_{j}$ has ones on the main diagonal and only one nonzero element $latex a_{ij}=c$ outside it. By the Leibniz formula it's determinant is 1.

The reader can easily solve the next

Exercise 2. a) Postmultiplication of $latex A$ by $latex e_{j}^{T}$ cuts out of $latex A$ the $latex j$-th column $latex A^{(j)}.$

b) Postmultiplication of $latex A$ by $latex e_{j}^{T}e_{i}$ puts the column $latex A^{(j)}$ into the $latex i$-th column of the null matrix.

c) Postmultiplication of $latex A$ by $latex I+ce_{j}^{T}e_{i}$ adds column $latex A^{(j)}$ multiplied by $latex c$ to column $latex A^{(i)},$ without changing the other columns of $latex A.$

d) The matrix $latex I+ce_{j}^{T}e_{i}=(I+ce_{i}^{T}e_{j})^{T}$ has determinant 1.

Exercise 3. a) Premultiplication of $latex A$ by

(1) $latex \left(\begin{array}{c}e_{1} \\... \\e_{j} \\... \\e_{i} \\... \\e_{n}\end{array}
\right) $

permutes rows $latex A_{i},A_{j}.$

b) Postmultiplication of $latex A$ by the transpose of (1) permutes columns $latex A^{(i)},A^{(j)}.$

This is a general property of permutation matrices. Recall also that their determinants can be only $latex \pm 1.$

Definition. 1) Adding some row multiplied by a constant to another row or 2) adding some column multiplied by a constant to another column or 3) permuting rows or columns is called an elementary operation. Accordingly, matrices that realize them are called elementary matrices.