Raising the bar Residual Sum of Squares Archives

Nov 16

The pearls of AP Statistics 35

category: Agresti & Franklin, Agresti and Franklin, AP Stats and Business Stats, Econometrics, ST104A, ST104B, Statistics 1, Statistics 2, The College Board 1 comment

The disturbance term: To hide or not to hide? In an introductory Stats course, some part of the theory should be hidden. Where to draw the line is an interesting question. Here I discuss the ideas that look definitely bad to me.

How disturbing is the disturbance term?

In the main text, Agresti and Franklin never mention the disturbance term $u_i$ in the regression model

(1) $y_i=a+bx_i+u_i$

(it is hidden in Exercise 12.105). Instead, they write the equation for the mean $\mu_y=a+bx$ that follows from (1) under the standard assumption $Eu_i=0$ . This would be fine if the exposition stopped right there. However, one has to explain the random source of variability in $y_i$ . On p. 583 the authors say: "The probability distribution of y values at a fixed value of x is a conditional distribution. At each value of x, there is a conditional distribution of y values. A regression model also describes these distributions. An additional parameter σ describes the standard deviation of each conditional distribution."

Further, Figure 12.4 illustrates distributions of errors at different points and asks: "What do the bell-shaped curves around the line at x = 12 and at x = 16 represent?"

Figure 12.4. Illustration of error distributions

Besides, explanations of heteroscedasticity and of the residual sum of squares are impossible without explicitly referring to the disturbance term.

Attributing a regression property to the correlation is not good

On p.589 I encountered a statement that puzzled me: "An important property of the correlation is that at any particular x value, the predicted value of y is relatively closer to its mean than x is to its mean. If an x value is a certain number of standard deviations from its mean, then the predicted y is r times that many standard deviations from its mean."

Firstly, this is a verbal interpretation of some formula, so why not give the formula itself? How good must be a student to guess what is behind the verbal formulation?

Secondly, as I stressed in this post, the correlation coefficient does not entail any prediction about the magnitude of a change in one variable caused by a change in another. The above statement about the predicted value of y must be a property of regression. Attributing a regression property to the correlation is not in the best interests of those who want to study Stats at a more advanced level.

Thirdly, I felt challenged to see something new in the area I thought I knew everything about. So here is the derivation. By definition, the fitted value is

(2) $\hat{y_i}=\hat{a}+\hat{b}x_i$

where the hats stand for estimators. The fitted line passes through the point $(\bar{x},\bar{y})$ :

(3) $\bar{y}=\hat{a}+\hat{b}\bar{x}$

(this will be proved elsewhere). Subtracting (3) from (2) we get

(4) $\hat{y_i}-\bar{y}=\hat{b}(x_i-\bar{x})$

(using equation (4) from this post)

$=\rho\frac{\sigma(y)}{\sigma(x)}(x_i-\bar{x}).$

It is helpful to rewrite (4) in a more symmetric form:

(5) $\frac{\hat{y_i}-\bar{y}}{\sigma(y)}=\rho\frac{x_i-\bar{x}}{\sigma(x)}.$

This is the equation we need. Suppose an x value is a certain number of standard deviations from its mean: $x_i-\bar{x}=k\sigma(x)$ . Plug this into (5) to get $\hat{y_i}-\bar{y}=\rho k\sigma(y)$ , that is, the predicted y is $\rho$ times that many standard deviations from its mean.

Tags: AP Statistics, Business Statistics, conditional distribution, disturbance term, fitted line, fitted value, heteroscedasticity, OLS estimator, regression, Residual Sum of Squares, sample mean, simple regression, standard deviation, University of London International Programmes, uol affiliate centre

Oct 16

The pearls of AP Statistics 34

category: Agresti & Franklin, Agresti and Franklin, AP Stats and Business Stats, EC2020, EC2020 Elements of econometrics, Econometrics, ST104A, ST104B, Statistics 1, Statistics 2, The College Board 1 comment

Coefficient of determination: an inductive introduction to R squared

I know a person, who did not understand this topic, even though he had a PhD in Math. That was me more than twenty years ago, and the reason was that the topic was given formally, without explaining the leading idea.

Leading idea

Step 1. We want to describe the relationship between observed y's and x's using the simple regression

$y_i=a+bx_i+u_i$ .

Let us start with the simple case when there is no variability in y's, that is the slope and the errors are zero. Since $y_i=a$ for all $i$ , we have $y_i=\bar{y}$ and, of course,

(1) $\sum(y_i-\bar{y})^2=0.$

In the general case, we start with the decomposition

(2) $y_i=\hat{y}_i+e_i$

where $\hat{y}_i$ is the fitted value and $e_i$ is the residual, see this post. We still want to see how $y_i$ is far from $\bar{y}.$ With this purpose, from both sides of equation (2) we subtract $\bar{y},$ obtaining $y_i-\bar{y}=(\hat{y}_i-\bar{y})+e_i.$ Squaring this equation, for the sum in (1) we get

(3) $\sum(y_i-\bar{y})^2=\sum(\hat{y}_i-\bar{y})^2+2\sum(\hat{y}_i-\bar{y})e_i+\sum e^2_i.$

Whoever was the first to do this, discovered that the cross product is zero and (3) simplifies to

(4) $\sum(y_i-\bar{y})^2=\sum(\hat{y}_i-\bar{y})^2+\sum e^2_i.$

The rest is a matter of definitions

Total Sum of Squares $TSS=\sum(y_i-\bar{y})^2.$ (I prefer to call this a total variation around $\bar{y}$ )

Explained Sum of Squares $TSS=\sum(\hat{y}_i-\bar{y})^2$ (to me this is explained variation around $\bar{y}$ )

Residual Sum of Squares $TSS=\sum e^2_i$ (unexplained variation around $\bar{y}$ , caused by the error term)

Thus from (4) we have

(5) $TSS=ESS+RSS$ and $1=ESS/TSS+RSS/TSS.$

Step 2. It is desirable to have RSS close to zero and ESS close to TSS. Therefore we can use the ratio ESS/TSS as a measure of how good the regression describes the relationship between y's and x's. From (5) it follows that this ratio takes values between zero and 1. Hence, the coefficient of determination

$R^2=ESS/TSS$

can be interpreted as the percentage of total variation of y's around $\bar{y}$ explained by regression. From (5) an equivalent definition is

$R^2=1-RSS/TSS.$

Back to the pearls of AP Statistics

How much of the above can be explained without algebra? Stats without algebra is a crippled creature. I am afraid, any concepts requiring substantial algebra should be dropped from AP Stats curriculum. Compare this post with the explanation on p. 592 of Agresti and Franklin.

Tags: AP Statistics, Business Statistics, Coefficient of determination, Explained Sum of Squares, explained variation, fitted value, interpretation of coefficient of determination, OLS estimator, R squared, residual, Residual Sum of Squares, simple regression, Total Sum of Squares, total variation, unexplained variation, University of London International Programmes, uol affiliate centre

Raising the bar

Ins and outs of quantitative courses of University of London

The pearls of AP Statistics 35

How disturbing is the disturbance term?

Attributing a regression property to the correlation is not good

The pearls of AP Statistics 34

Coefficient of determination: an inductive introduction to R squared

Leading idea

The rest is a matter of definitions

Back to the pearls of AP Statistics

Subscribe to Blog via Email

Recent Posts

Menu

Archives