Nonlinear least squares
Here we explain the idea, illustrate the possible problems in Mathematica and, finally, show the implementation in Stata.
Idea: minimize RSS, as in ordinary least squares
Observations come in pairs . In case of ordinary least squares, we approximated the y's with linear functions of the parameters, possibly nonlinear in x's. Now we use a function which may be nonlinear in . We still minimize RSS which takes the form . Nonlinear least squares estimators are the values that minimize RSS. In general, it is difficult to find the formula (closed-form solution), so in practice software, such as Stata, is used for RSS minimization.
Simplified idea and problems in one-dimensional case
Suppose we want to minimize . The Newton algorithm (default in Stata) is an iterative procedure that consists of steps:
- Select the initial value .
- Find the derivative (or tangent) of RSS at . Make a small step in the descent direction (indicated by the derivative), to obtain the next value .
- Repeat Step 2, using as the starting point, until the difference between the values of the objective function at two successive points becomes small. The last point will approximate the minimizing point.
- The minimizing point may not exist.
- When it exists, it may not be unique. In general, there is no way to find out how many local minimums there are and which ones are global.
- The minimizing point depends on the initial point.
See Video 1 for illustration in the one-dimensional case.
Problems illustrated in Mathematica
Here we look at three examples of nonlinear functions, two of which are considered in Dougherty. The first one is a power functions (it can be linearized applying logs) and the second is an exponential function (it cannot be linearized). The third function gives rise to two minimums. The possibilities are illustrated in Mathematica.
Finally, implementation in Stata
Here we show how to 1) generate a random vector, 2) create a vector of initial values, and 3) program a nonlinear dependence.