Raising the bar

Preliminaries

Review Properties of conditional expectation, especially the summary, where I introduce a new notation for conditional expectation. Everywhere I use the notation  for expectation of conditional on , instead of .

This post and the previous one on conditional expectation show that conditioning is a pretty advanced notion. Many introductory books use the condition  (the expected value of the error term conditional on the regressor is zero). Because of the complexity of conditioning, I think it's better to avoid this kind of assumption as much as possible.

Conditional variance properties

Replacing usual expectations by their conditional counterparts in the definition of variance, we obtain the definition of conditional variance:

(1)

Property 1. If  are independent, then  and  are also independent and conditioning doesn't change variance:

see Conditioning in case of independence.

Property 2. Generalized homogeneity of degree 2: if  is a deterministic function, then  can be pulled out:

Property 3. Shortcut for conditional variance:

(2)

Proof.

(distributing conditional expectation)

(applying Properties 2 and 6 from this Summary with )

Property 4. The law of total variance:

(3) 

Proof. By the shortcut for usual variance and the law of iterated expectations

(replacing from (2))

(the last two terms give the shortcut for variance of )

Before we move further we need to define conditional covariance by

(everywhere usual expectations are replaced by conditional ones). We say that random variables are conditionally uncorrelated if .

Property 5. Conditional variance of a linear combination. For any random variables and functions one has

The proof is quite similar to that in case of usual variances, so we leave it to the reader. In particular, if are conditionally uncorrelated, then the interaction terms disappears:

Agresti and Franklin on p.658 say: The indicator variable for a particular category is binary. It equals 1 if the observation falls into that category and it equals 0 otherwise. I say: For most students, this is not clear enough.

Problem statement

Figure 1. Residential power consumption in 2014 and 2015. Source: http://www.eia.gov/electricity/data.cfm

Residential power consumption in the US has a seasonal pattern. Heating in winter and cooling in summer cause the differences. We want to capture the dependence of residential power consumption on the season.

 Visual approach to dummy variables

Seasons of the year are categorical variables. We have to replace them with quantitative variables, to be able to use in any mathematical procedure that involves arithmetic operations. To this end, we define a dummy variable (indicator) for winter such that it equals 1 in winter and 0 in any other period of the year. The dummies for spring, summer and autumn are defined similarly. We provide two visualizations assuming monthly observations.

Table 1. Tabular visualization of dummies

Month
December	1	0	0	0	1
January	1	0	0	0	1
February	1	0	0	0	1
March	0	1	0	0	1
April	0	1	0	0	1
May	0	1	0	0	1
June	0	0	1	0	1
July	0	0	1	0	1
August	0	0	1	0	1
September	0	0	0	1	1
October	0	0	0	1	1
November	0	0	0	1	1

Figure 2. Graphical visualization of D_spr

The first idea may be wrong

The first thing that comes to mind is to regress  on dummies as in

(1)

Not so fast. To see the problem, let us rewrite (1) as

(2)

This shows that, in addition to the four dummies, there is a fifth variable, which equals 1 across all observations. Let us denote it  (for Trivial). Table 1 shows that

(3)

This makes the next definition relevant. Regressors  are called linearly dependent if one of them, say, , can be expressed as a linear combination of the others: .  In case (3), all coefficients are unities, so we have linear dependence. Using (3), let us replace  in (2). The resulting equation is rearranged as

(4)

Now we see what the problem is. When regressors are linearly dependent, the model is not uniquely specified. (1) and (4) are two different representations of the same model.

What is the way out?

If regressors are linearly dependent, drop them one after another until you get linearly independent ones. For example, dropping the winter dummy, we get

(5)

Here is the estimation result for the two-year data in Figure 1:

.

This means that:

in winter,  in spring,

in summer, and  in autumn.

It is revealing that cooling requires more power than heating. However, the summer coefficient is not significant. Here is the Excel file with the data and estimation result.

The category that has been dropped is called a base (or reference) category. Thus, the intercept in (5) measures power consumption in winter. The dummy coefficients in (5) measure deviations of power consumption in respective seasons from that in winter.

Here is the question I ask my students

We want to see how beer consumption depends on gender and income . Let  and  denote the dummies for males and females, resp. Correct the following model and interpret the resulting coefficients:

Final remark

When a researcher includes all categories plus the trivial regressor, he/she falls into what is called a dummy trap. The problem of linear dependence among regressors is usually discussed under the heading of multiple regression. But since the trivial regressor is present in simple regression too, it might be a good idea to discuss it earlier.

Linear dependence/independence of regressors is an exact condition for existence of the OLS estimator. That is, if regressors are linearly dependent, then the OLS estimator doesn't exist, in which case the question about its further properties doesn't make sense. If, on the other hand, regressors are linearly independent, then the OLS estimator exists, and further properties can be studied, such as unbiasedness, variance and efficiency.

Testing for structural changes: a topic suitable for AP Stats

Problem statement

Economic data are volatile but sometimes changes in them look more permanent than transitory.

Figure 1. US GDP from agriculture. Source: http://www.tradingeconomics.com/united-states/gdp-from-agriculture

Figure 1 shows fluctuations of US GDP from agriculture. There have been ups and downs throughout the period of 2005-2016 but overall the trend has been up until 2013 and down since then. We want an objective, statistical confirmation of the fact that in 2013 the change was structural, substantial rather than a random fluctuation.

Chow test steps

1. Divide the observed sample in two parts, A and B, at the point where you suspect the structural change (or break) has occurred. Run three regressions: one for A, another for B and the third one for the whole sample (pooled regression). Get residual sums of squares from each of them, denoted ,  and , respectively.
2. Let  and  be the numbers of observations in the two subsamples and suppose there are coefficients in your regression (for Figure 1, we would regress GDP on a time variable, so the number of coefficients would be 2, including the intercept). The Chow test statistic is defined by

This statistic is distributed as with  degrees of freedom. The null hypothesis is that the coefficients are the same for the two subsamples and the alternative is that they are not. If the statistic is larger than the critical value at your chosen level of significance, splitting the sample in two is beneficial (better describes the data). If the statistic is not larger than the critical value, the pooled regression better describes the data.

Figure 2. Splitting is better (there is a structural change)

In Figure 2, the gray lines are the fitted lines for the two subsamples. They fit the data much better than the orange line (the fitted line for the whole sample).

Figure 3. Pooling is better

In Figure 3, pooling is better because the intercept and slope are about the same and pooling amounts to increasing the sample size.

Download the Excel file used for simulation. See the video.

It's time to modernize the AP Stats curriculum

The suggestions below are based on the College Board AP Statistics Course Description, Effective Fall 2010. Citing this description, “AP teachers are encouraged to develop or maintain their own curriculum that either includes or exceeds each of these expectations; such courses will be authorized to use the “AP” designation.” However, AP teachers are constrained by the statement that “The Advanced Placement Program offers a course description and exam in statistics to secondary school students who wish to complete studies equivalent to a one semester, introductory, non-calculus-based, college course in statistics.”

Too much material for a one-semester course

I tried to teach AP Stats in one semester following the College Board description and methodology. That is, with no derivations, giving only recipes and concentrating on applications. The students were really stretched, didn’t remember anything after completing the course, and usefulness of the course for the subsequent calculus-based course was minimal.

Suggestion. Reduce the number of topics and concentrate on those, which require going all the way from (again citing the description) Exploring Data to Sampling and Experimentation to Anticipating Patterns to Statistical Inference. Simple regression is such a topic.

I would drop the stem-and-leaf plot, because it is stupid; chi-square test for goodness of fit, homogeneity of proportions and independence, including ANOVA, because it is too advanced and looks too vague without the right explanation. Instead of going wide, it is better to go deeper, building upon what students already know. I’ll post a couple of regression applications.

“Introductory” should not mean stupefying

Statistics has its specifics. Even I, with my extensive experience in Math, made quite a few discoveries for myself while studying Stats. Textbook authors, in their attempts to make exposition accessible, often replace the true statistical ideas by after-the-fact intuition or formulas by their verbal description. See, for example, the z score.

Using TI-83+ and TI-84 graphing calculators is like using a Tesla electric car in conjunction with candles for generating electricity. The sole purpose of these calculators is to prevent cheating. The inclination for cheating is a sign of low understanding and the best proof that the College Board strategy is wrong.

Once you say “this course is non-calculus-based”, you close many doors

When we format a document in Word, we don’t care how formatting is implemented technically and we don’t need to know anything about programming. Looks like the same attitude is imparted to students of Stats. Few people notice a big difference. When we format a document, we have an idea of what we want and test the result against that idea. In Stats, the idea has to be translated to a formula, and the software output has to be translated into a formula for interpretation.

I understand that, for the majority of Stats students, the amount of algebra I use in some of my posts is not accessible. However, the opposite tendency of telling students that they don’t need to remember any formulas is unproductive. It’s only by memorizing and reproducing equations that they can augment their algebraic proficiency. Stats is largely a mental science. To improve mental activity, you have to engage in one.

Suggestion. Instead of “this course is non-calculus-based”, say: the course develops the ability to interpret equations and translate ideas to formulas.

Follow a logical sequence

The way most AP Stats books are written does not give any idea as to what comes from where. When I was a bachelor student, I was looking for explanations, and I would hate reading one of today’s AP Stats textbooks. For those who think, memorizing a bunch of recipes, without seeing the logical links, is a nightmare. In some cases, the absence of logic leads to statements that are plain wrong. Just following the logical sequence will put the pieces of the puzzle together.

You don't need statistical tables if you have Excel or Mathematica. Here I give the relevant Excel and Mathematica functions described in Chapter 14 of my book. You can save all the formulas in one spreadsheet or notebook and use it multiple times.

Cumulative Distribution Function of the Standard Normal Distribution

For a given real , the value of the distribution function of the standard normal is

In Excel, use the formula =NORM.S.DIST(z,TRUE).

In Mathematica, enter CDF[NormalDistribution[0,1],z]

Probability Function of the Binomial Distribution

For given number of successes number of trials and probability  the probability is

.

In Excel, use the formula =BINOM.DIST(x,n,p,FALSE)

In Mathematica, enter PDF[BinomialDistribution[n,p],x]

Cumulative Binomial Probabilities

For a given cut-off value number of trials and probability the cumulative probability is

In Excel, use the formula =BINOM.DIST(x,n,p,TRUE).

In Mathematica, enter CDF[BinomialDistribution[n,p],x]

Values of the exponential function

In Excel, use the formula =EXP(-lambda)

In Mathematica, enter Exp[-lambda]

Individual Poisson Probabilities

For given number of successes and arrival rate the probability is

In Excel, use the formula =POISSON.DIST(x,lambda,FALSE)

In Mathematica, enter PDF[PoissonDistribution[lambda],x]

Cumulative Poisson Probabilities

For given cut-off and arrival rate the cumulative probability is

In Excel, use the formula =POISSON.DIST(x,lambda,TRUE)

In Mathematica, enter CDF[PoissonDistribution[lambda],x]

Cutoff Points of the Chi-Square Distribution Function

For given probability of the right tail and degrees of freedom , the cut-off value (critical value) is a solution of the equation

In Excel, use the formula =CHISQ.INV.RT(alpha,v)

In Mathematica, enter InverseCDF[ChiSquareDistribution[v],1-alpha]

Cutoff Points for the Student’s t Distribution

For given probability of the right tail and degrees of freedom , the cut-off value is a solution of the equation .
In Excel, use the formula =T.INV(1-alpha,v)

In Mathematica, enter InverseCDF[StudentTDistribution[v],1-alpha]

Cutoff Points for the F Distribution

For given probability of the right tail , degrees of freedom (numerator) and (denominator), the cut-off value is a solution of the equation .

In Excel, use the formula =F.INV.RT(alpha,v1,v2)

In Mathematica, enter InverseCDF[FRatioDistribution[v1,v2],1-alpha]

Nonparametric estimation is the right topic for expanding the Stats agenda

Figure 1. Dependence of income on age

For the last several years I have been doing research in nonparametric estimation. It is intellectually rewarding and it is the best tool to show Stats students the usefulness of Statistics. Agresti and Franklin have a chapter on nonparametric estimation. However, the choice of the topics (Wilcoxon test and Kruskal-Wallis test) is unfortunate. These two tests are about comparing means from two samples. They provide just numbers (corresponding statistics), which is not very appealing because the students see just another solution to the familiar problem.

Nonparametric technique is best in nonlinear curve fitting, and this is its selling point because it is VISUAL. The following examples explain the difference between parametric and nonparametric estimation.

Example 1

Suppose we want to use simple regression to estimate dependence of consumption on income. This is a parametric model, with two parameters (intercept and slope). Suppose the fitted line is (I just put plausible numbers). The slope is interpreted as the marginal propensity to consume and can be used in economic modeling to find the budget multiplier. The advantage of parametric estimation is that often estimated parameters have economic meaning.

Example 2

This example has been taken from Lecture Notes by John Fox. Now let us look at dependence of income on age. It is clear that income is low for young people, then rises with age until middle age and declines after retirement. The dependence is obviously nonlinear and, a priori, no guesses can be made about the shape of the curve.

Example 3

Figure 2. Density of return on Apple stock

Figure 3. Density of return on MA stock

What would have been the better 5-year investment: Apple or MasterCard? Figure 2 shows that the density of return on Apple stock has a negative mode. The density of return on MasterCard has the mode close to zero. This tells us that MasterCard would be better. Indeed, the annual return on Apple is 18%, while on MasterCard it is 29% (over the last 5 years). Nonparametric estimates of densities (kernel density estimates) are used by financial analysts to simulate stock prices to predict their future movements.

Remark. For simple statistical tasks I recommend Eviews student version for two reasons. 1) It has excellent Help. When I want my students to understand just the essence and avoid proofs, I tell them to read Eviews Help. 2) The student version is just $39.95. Figures 2 and 3 have been produced using Eviews.

Correlation coefficient: the last block of statistical foundation

Correlation has already been mentioned in

Statistical measures and their geometric roots

Properties of standard deviation

The pearls of AP Statistics 35

Properties of covariance

The pearls of AP Statistics 33

The hierarchy of definitions

Suppose random variables are not constant. Then their standard deviations are not zero and we can define their correlation as in Chart 1.

Chart 1. Correlation definition

Properties of correlation

Property 1. Range of the correlation coefficient: for any one has .
This follows from the Cauchy-Schwarz inequality, as explained here.

Recall from this post that correlation is cosine of the angle between and .
Property 2. Interpretation of extreme cases. (Part 1) If , then with

(Part 2) If , then with .

Proof. (Part 1) implies
(1)
which, in turn, implies that is a linear function of : (this is the second part of the Cauchy-Schwarz inequality). Further, we can establish the sign of the number . By the properties of variance and covariance
,

.
Plugging this in Eq. (1) we get and see that is positive.

The proof of Part 2 is left as an exercise.

Property 3. Suppose we want to measure correlation between weight and height of people. The measurements are either in kilos and centimeters or in pounds and feet . The correlation coefficient is unit-free in the sense that it does not depend on the units used: . Mathematically speaking, correlation is homogeneous of degree in both arguments.
Proof. One measurement is proportional to another, with some positive constants . By homogeneity

ANOVA: the artefact that survives because of the College Board

Why ANOVA should be dropped from AP Statistics

1. The common argument in favor of using ANOVA is that "The methods introduced in this chapter [Comparing Groups: Analysis of Variance Methods] apply when a quantitative response variable has a categorical explanatory variable" (Agresti and Franklin, p. 680). However, categorical explanatory variables can be replaced by indicator (dummy) variables, and then regression methods can be used to study dependences involving categorical variables. On p. 695, the authors admit that "ANOVA can be presented as a special case of multiple regression".
2. In terms of knowledge of basic statistical ideas (hypothesis testing, F statistics, significance level), ANOVA doesn't add any value. Those, who have mastered these basic ideas, will not have problems learning ANOVA at their workplace if they have to. There is no need to burden everybody with this stuff "just in case".
3. The explanation of ANOVA is accompanied with definitions of the within-groups variance estimate and between-groups variance estimate (Agresti and Franklin, p. 686). Even in my courses, where I give a lot of algebra, the students don't get them unless they do a couple of theoretical exercises. At the AP Stats level, the usefulness of these definitions is nil.
4. The requirement to remember how the F statistics and degrees of freedom are calculated, for the purpose of being able to interpret just one table with output from a statistical package, doesn't make sense. In my book, I have a whole chapter on ANOVA, with most derivations, and I don't remember a thing. Why torture the students?
5. In the 90 years since R. Fisher has invented ANOVA, many other, more precise and versatile, statistical methods have been developed.

Conclusion

There are two suggestions

1) Explain just the intuition and then jump to the interpretation of output, indicating the statistic to look at, as in Table 14.14.

2) The theory of ANOVA is useful for two reasons: there is a lot of manipulation with summation signs and there is a link to regressions. Learning all this may be the only justification to study ANOVA with definitions. In my classes, this takes 6 hours.

Variance, covariance, standard deviation and correlation: their definitions and properties are deeply rooted in the Euclidean geometry.

Here is the why: analogy with Euclidean geometry

Euclid axiomatically described the space we live in. What we have known about the geometry of this space since the ancient times has never failed us. Therefore, statistical definitions based on the Euclidean geometry are sure to work.

   1. Analogy between scalar product and covariance

Geometry. See Table 2 here for operations with vectors. The scalar product of two vectors is defined by

Statistical analog: Covariance of two random variables is defined by

Both the scalar product and covariance are linear in one argument when the other argument is fixed.

   2. Analogy between orthogonality and uncorrelatedness

Geometry. Two vectors are called orthogonal (or perpendicular) if

(1)

Exercise. How do you draw on the plane the vectors ? Check that they are orthogonal.

Statistical analog: Two random variables are called uncorrelated if .

   3. Measuring lengths

Figure 1. Length of a vector

Geometry: the length of a vector is , see Figure 1.

Statistical analog: the standard deviation of a random variable is

This explains the square root in the definition of the standard deviation.

   4. Cauchy-Schwarz inequality

Geometry: .

Statistical analog: . See the proof here. The proof of its geometric counterpart is similar.

   5. Triangle inequality

Figure 2. Triangle inequality

Geometry: , see Figure 2 where the length of X+Y does not exceed the sum of lengths of X and Y.

Statistical analog: using the Cauchy-Schwarz inequality we have

   4. The Pythagorean theorem

Geometry: In a right triangle, the squared hypotenuse is equal to the sum of the squares of the two legs. The illustration is similar to Figure 2, except that the angle between X and Y should be right.

Proof. Taking two orthogonal vectors as legs, we have

Squared hypotenuse =

(squaring out and using orthogonality (1))

= Sum of squared legs

Statistical analog: If two random variables are uncorrelated, then variance of their sum is a sum of variances

   5. The most important analogy: measuring angles

Geometry: the cosine of the angle between two vectors is defined by

Cosine between X,Y =

Statistical analog: the correlation coefficient between two random variables is defined by

This intuitively explains why the correlation coefficient takes values between -1 and +1.

Remark. My colleague Alisher Aldashev noticed that the correlation coefficient is the cosine of the angle between the deviations and and not between  themselves.

Properties of standard deviation are divided in two parts. The definitions and consequences are given here. Both variance and standard deviation are used to measure variability of values of a random variable around its mean. Then why use both of them? The why will be explained in another post.

Properties of standard deviation: definitions and consequences

Definition. For a random variable , the quantity is called its standard deviation.

    Digression about square roots and absolute values

In general, there are two square roots of a positive number, one positive and the other negative. The positive one is called an arithmetic square root. The arithmetic root is applied here to (see properties of variance), so standard deviation is always nonnegative.
Definition. An absolute value of a real number is defined by
(1) if is nonnegative and if is negative.
This two-part definition is a stumbling block for many students, so making them plug in a few numbers is a must. It is introduced to measure the distance from point to the origin. For example, and . More generally, for any points on the real line the distance between them is given by .

By squaring both sides in Eq. (1) we obtain . Application of the arithmetic square root gives

(2)

This is the equation we need right now.

Back to standard deviation

Property 1. Standard deviation is homogeneous of degree 1. Indeed, using homogeneity of variance and equation (2), we have

Unlike homogeneity of expected values, here we have an absolute value of the scaling coefficient .

Property 2. Cauchy-Schwarz inequality. (Part 1) For any random variables one has

(3) .

(Part 2) If the inequality sign in (3) turns into equality, , then is a linear function of : , with some constants .
Proof. (Part 1) If at least one of the variables is constant, both sides of the inequality are and there is nothing to prove. To exclude the trivial case, let be non-constant and, therefore, are positive. Consider a real-valued function of a real number defined by . Here we have variance of a linear combination

.

We see that is a parabola with branches looking upward (because the senior coefficient is positive). By nonnegativity of variance, and the parabola lies above the horizontal axis in the plane. Hence, the quadratic equation may have at most one real root. This means that the discriminant of the equation is non-positive:

Applying square roots to both sides of we finish the proof of the first part.

(Part 2) In case of the equality sign the discriminant is . Therefore the parabola touches the horizontal axis where . But we know that this implies which is just another way of writing .

Comment. (3) explains one of the main properties of the correlation:

.