Unconstrained optimization on the plane: sufficient conditions
Based on what we know about the one-dimensional case we should expect that first-order and second-order conditions jointly are sufficient for a function to have an extreme inside the domain. It remains to find out what those conditions are. Also previous posts provide convincing evidence that one has to work with the Taylor decomposition:
(1)
It has been derived here. The necessity of the first order condition for a function to have an extreme
(2)
has been shown here. Under this condition (1) gives
(3)
Case of a minimum. This approximation makes clear that to have for all small nonzero
we need to have
for all such
. (We require
to be nonzero because otherwise we get
and we want it to be small because we are looking for a local minimum.)
Case of a maximum. Similarly, to have for all small nonzero
we need to have
for all such
.
These observations lead us to the following definition.
Definition. Let be a square matrix and let
be a column-vector of compatible dimension. 1) We say that
is positive definite if the quadratic form
is positive for all
. 2) We say that
is negative definite if the quadratic form
is negative for all
.
Note that in this definition we don't require to be small because in general the matrix
is not related to any optimization problem. The definition is stated in terms of the quadratic form which is a function of
. It's difficult to apply directly. The question is: how do you know that a matrix is positive or negative definite? The answer is given by Sylvester's criterion. For reasons we cannot discuss here it applies only to symmetric matrices that by definition satisfy
. As an exercise, check that a Hessian is symmetric.
Denote
These are determinants of upper left corners of the matrix . They are called leading principal minors.
Sylvester's criterion. Let be symmetric. 1)
is positive definite if and only if all leading principal minors are positive. 2)
is negative definite if and only if principal minors have alternating signs, starting with a minus:
.
Conclusion. 1) If the first-order conditions (2) are satisfied and the Hessian is positive definite (second-order condition), then is a minimum point. 2) If the first-order conditions (2) are satisfied and the Hessian is negative definite (second-order condition), then
is a maximum point.
Leave a Reply
You must be logged in to post a comment.