Unconstrained optimization on the plane: sufficient conditions
Based on what we know about the one-dimensional case we should expect that first-order and second-order conditions jointly are sufficient for a function to have an extreme inside the domain. It remains to find out what those conditions are. Also previous posts provide convincing evidence that one has to work with the Taylor decomposition:
It has been derived here. The necessity of the first order condition for a function to have an extreme
has been shown here. Under this condition (1) gives
Case of a minimum. This approximation makes clear that to have for all small nonzero we need to have for all such . (We require to be nonzero because otherwise we get and we want it to be small because we are looking for a local minimum.)
Case of a maximum. Similarly, to have for all small nonzero we need to have for all such .
These observations lead us to the following definition.
Definition. Let be a square matrix and let be a column-vector of compatible dimension. 1) We say that is positive definite if the quadratic form is positive for all . 2) We say that is negative definite if the quadratic form is negative for all .
Note that in this definition we don't require to be small because in general the matrix is not related to any optimization problem. The definition is stated in terms of the quadratic form which is a function of . It's difficult to apply directly. The question is: how do you know that a matrix is positive or negative definite? The answer is given by Sylvester's criterion. For reasons we cannot discuss here it applies only to symmetric matrices that by definition satisfy . As an exercise, check that a Hessian is symmetric.
These are determinants of upper left corners of the matrix . They are called principal minors.
Sylvester's criterion. Let be symmetric. 1) is positive definite if and only if all principal minors are positive. 2) is negative definite if and only if principal minors have alternating signs, starting with a minus: .
Conclusion. 1) If the first-order conditions (2) are satisfied and the Hessian is positive definite (second-order condition), then is a minimum point. 2) If the first-order conditions (2) are satisfied and the Hessian is negative definite (second-order condition), then is a maximum point.