Raising the bar

Assumptions about simple regression

We consider the simple regression

(1) $latex y_i=a+bx_i+e_i$

Here we derived the OLS estimators of the intercept and slope:

(2) $latex \hat{b}=\frac{Cov_u(x,y)}{Var_u(x)}$,

(3) $latex \hat{a}=\bar{y}-\hat{b}\bar{x}$.

A1. Existence condition. Since division by zero is not allowed, for (2) to exist we require $latex Var_u(x)\ne 0$. If this condition is not satisfied, then there is no variance in $latex x$ and all observed points are on the vertical line.

A2. Convenience condition. The regressor $latex x$ is deterministic. This condition is imposed to be able to apply the properties of expectation, see equation (7) in  this post. The time trend and dummy variables are examples of deterministic regressors. However, most real-life regressors are stochastic. Modifying the theory in order to cover stochastic regressors is the subject of two posts: finite-sample theory and large-sample theory.

A3. Unbiasedness condition. $latex Ee_i=0$. This is the main assumption that makes sure that OLS estimators are unbiased, see equation (7) in  this post.

Unbiasedness is not enough

Unbiasedness characterizes the quality of an estimator, see the intuitive explanation. Unfortunately, unbiasedness is not enough to choose the best estimator because of nonuniqueness: usually, if there is one unbiased estimator of a parameter, then there are infinitely many unbiased estimators of the same parameter. For example, we know that the sample mean $latex \bar{X}$ unbiasedly estimates the population mean $latex E\bar{X}=EX$. Since $latex EX_1=EX$ ($latex X_1$ is the first observation), we can easily construct an infinite family of unbiased estimators $latex Y=(\bar{X}+aX_1)/(1+a)$, assuming $latex a\ne -1$. Indeed, using linearity of expectation $latex EY=(E\bar{X}+aEX_1)/(1+a)=EX$.

Variance is another measure of an estimator quality: to have a lower spread of estimator values, among competing estimators we choose the one which has the lowest variance. Knowing the estimator variance allows us to find the z-score and use statistical tables.

Slope estimator variance

It is not difficult to find the variance of the slope estimator using representation (6) derived here:

$latex \hat{b}=b+\frac{1}{n}\sum a_ie_i$

where $latex a_i=(x_i-\bar{x})/Var_u(x).$

Don't try to apply directly the definition of variance at this point, because there will be a square of a sum, which leads to a double sum upon squaring. We need two new assumptions.

A4. Uncorrelatedness of errors. Assume that $latex Cov(e_i,e_j)=0$ for all $latex i\ne j$ (errors from different equations (1) are uncorrelated). Note that because of the unbiasedness condition, this assumption is equivalent to $latex Ee_ie_j=0$ for all $latex i\ne j$. This assumption is likely to be satisfied if we observe consumption patterns of unrelated individuals.

A5. Homoscedasticity. All errors have the same variances: $latex Var(e_i)=\sigma^2$ for all $latex i$. Again, because of the unbiasedness condition, this assumption is equivalent to $latex Ee_i^2=\sigma^2$ for all $latex i$.

Now we can derive the variance expression, using properties from this post:

$latex Var(\hat{b})=Var(b+\frac{1}{n}\sum_i a_ie_i)$ (dropping a constant doesn't affect variance)

$latex =Var(\frac{1}{n}\sum_i a_ie_i)$ (for uncorrelated variables, variance is additive)

$latex =\sum_i Var(\frac{1}{n}a_ie_i)$ (variance is homogeneous of degree 2)

$latex =\frac{1}{n^2}\sum_i a_i^2Var(e_i)$ (applying homoscedasticity)

$latex =\frac{1}{n^2}\sum_i a_i^2\sigma^2$ (plugging $latex a_i$)

$latex =\frac{1}{n^2}\sum_i(x_i-\bar{x})^2\sigma^2/Var^2_u(x)$ (using the notation of sample variance)

$latex =\frac{1}{n}Var_u(x)\sigma^2/Var^2_u(x)=\sigma^2/(nVar_u(x)).$

Note that canceling out two variances in the last line is obvious. It is not so obvious for some if instead of the short notation for variances you use summation signs. The case of the intercept variance is left as an exercise.

Conclusion

The above assumptions A1-A5 are called classical. It is necessary to remember their role in derivations because a considerable part of Econometrics is devoted to deviations from classical assumptions. Once you have a certain assumption violated, you should expect the corresponding estimator property invalidated. For example, if $latex Ee_i\ne 0$, you should expect the estimators to be biased. If any of A4-A5 is not true, the formula we have derived

$latex Var(\hat{b})=\sigma^2/(nVar_u(x))$

will not hold. Besides, the Gauss-Markov theorem that the OLS estimators are efficient will not hold (this will be discussed later). The pair A4-A5 can be called an efficiency condition.