Raising the bar

OLS estimator for multiple regression - as simple as possible

Here I try to explain a couple of ideas to folks not familiar with (or afraid of?) matrix algebra.

A matrix is a rectangular table of numbers. Operations with them most of the time are performed like with numbers. For example, for numbers we know that . For matrices this is also true, except that they often are denoted with capital letters: . It is easier to describe differences than similarities.

(1) One of the differences is that for matrices we can define a new operation called transposition: the columns of the original matrix are put into rows of a new matrix, which is called a transposed of . Visualize it like this: if has more rows than columns, then for the transposed the opposite will be true:

(2) We know that the number has the property that . For matrices, the analog is where is a special matrix called identity.

(3) The property we have for nonzero numbers generalizes for matrices except that instead of we write . Thus, the inverse matrix has the property that .

(4) You don't need to worry about how these operations are performed when you are given specific numerical matrices, because they can be easily done in Excel. All you have to do is watch that theoretical requirements are not violated. One of them is that, in general, matrices in a product cannot change places: .

Here is an example that continues my previous post about simplified derivation of the OLS estimator. Consider multiple regression

(5)

where is the dependent variable, is an independent variable, is the parameter to estimate and is the error. Multiplying from the left equation (5) by we get . As in my previous post, we get rid of the term containing the error by formally putting . The resulting equation  we solve for by multiplying by  from the left:

Putting the hat on , we arrive to the OLS estimator for multiple regression: . Like in the previous post, the whole derivation takes just one paragraph!

Caveat. See the rigorous derivation here. My objective is not rigor but to give you something easy to do and remember.