26
Jul 17

Nonlinear least squares: idea, geometry and implementation in Stata

category: Dougherty Introduction to Econometrics, EC2020, EC2020 Elements of econometrics, Econometrics, ST104B, Statistics 2 No Comments

Nonlinear least squares

Here we explain the idea, illustrate the possible problems in Mathematica and, finally, show the implementation in Stata.

Idea: minimize RSS, as in ordinary least squares

Observations come in pairs (x_1,y_1),...,(x_n,y_n). In case of ordinary least squares, we approximated the y's with linear functions of the parameters, possibly nonlinear in x's. Now we use a function f(a,b,x_i) which may be nonlinear in a,b. We still minimize RSS which takes the form

RSS=\sum r_i^2=\sum(y_i-f(a,b,x_i))^2

Nonlinear least squares estimators are the values a,b that minimize RSS. In general, it is difficult to find the formula (closed-form solution), so in practice software, such as Stata, is used for RSS minimization.

Simplified idea and problems in one-dimensional case

Suppose we want to minimize f(x). The Newton algorithm (default in Stata) is an iterative procedure that consists of steps:

  1. Select the initial value x_0.
  2. Find the derivative (or tangent) of RSS at x_0. Make a small step in the descent direction (indicated by the derivative), to obtain the next value x_1.
  3. Repeat Step 2, using x_1 as the starting point, until the difference between the values of the objective function at two successive points becomes small. The last point x_n will approximate the minimizing point.

Problems:

  1. The minimizing point may not exist.
  2. When it exists, it may not be unique. In general, there is no way to find out how many local minimums there are and which ones are global.
  3. The minimizing point depends on the initial point.

See Video 1 for illustration in the one-dimensional case.

NLS geometry

Video 1. NLS geometry

Problems illustrated in Mathematica

Here we look at three examples of nonlinear functions, two of which are considered in Dougherty. The first one is a power functions (it can be linearized applying logs) and the second is an exponential function (it cannot be linearized). The third function gives rise to two minimums. The possibilities are illustrated in Mathematica.

NLS illustrated in Mathematica

Video 2. NLS illustrated in Mathematica

 

Finally, implementation in Stata

Here we show how to 1) generate a random vector, 2) create a vector of initial values, and 3) program a nonlinear dependence.

Nonlinear least squares implemented in Stata

Video 3. NLS implemented in Stata

Post Views: 41

Leave a Reply

You must be logged in to post a comment.