Law and order in the set of matrices
The law is to feel by touch every little fact. The order is discussed below.
Why do complex numbers come right before this topic?
Once you know this, it's easy to come up with a couple of ideas.
Idea 1. From the characterization of real numbers (see Equation (4)) we see that matrices that satisfy (symmetric matrices) correspond to real numbers and should be better in some way than asymmetric ones.
Idea 2. From Equation (3) we see that the matrix should be symmetric and non-negative.
What is a non-negative matrix?
The set of real numbers is ordered in the sense that for any two real numbers we can say that either or is true. The most important property that we used in my class is this: if and then (the sign of an inequality is preserved if the inequality is multiplied by a positive number). Since any two numbers can be compared like that, it is a complete order. One way in which symmetric matrices are better than more general ones is that for symmetric matrices one can define order. The limitation caused by dimensionality is that this order is not complete (some symmetric matrices are not comparable).
Exercise 1. For the matrix and vector find the expression What is the value of this expression at
Definition 1. The function is called a quadratic form of the matrix Here is symmetric of size and
Discussion. 1) The facts that is in the subscript and the argument is mean that is fixed and is changing.
2) While the argument is a vector, the value of this function is a real number: acts from to
3) does not contain constant or linear terms (of type and It contains only quadratic terms (write to see that the total power is 2), that's why it is called a quadratic form and not a quadratic function.
Definition 2. We say that is positive if for all nonzero and non-negative if for all (Most sources say positive definite instead of just positive and non-negative definite instead of just non-negative. I prefer a shorter terminology. If you don't understand why in the definition of positivity we require nonzero go back to Exercise 1). As with numbers, for two symmetric matrices of the same size, we write or if is positive or non-negative, respectively. Continuing this idea, we can say that is negative if is positive.
More on motivation. A legitimate definition of order would obtain if we compared the two matrices element-wise. Definition 2 is motivated by the fact that quadratic forms arise in the multivariate Taylor decomposition.
Sylvester's criterion is the only practical tool for determining positivity or non-negativity. However, in one case this is simple.
Exercise 2. Show that is symmetric and non-negative.
Solution. The symmetry is straightforward and has been shown before. Non-negativity is not difficult either:
The graph of a quadratic form in good cases is an elliptic paraboloid and has various other names in worse cases. Geometrically, the definition of the inequality means that the graph of is everywhere above the graph of (at the origin they always coincide). In particular, means that the graph of is everywhere above the horizontal plane.
Examples. All examples are matrices of size .
1) The identity matrix is positive because , see Figure 1.
2) The matrix is non-negative. Its quadratic form grows when grows and stays flat when changes and is fixed, see Figure 2.
3) The matrix is not positive or non-negative or negative or non-positive. Its quadratic form is a parabola with branches looking upward when the second argument is fixed and a parabola with branches looking downward when the first argument is fixed, see Figure 3. When a surface behaves like that around some point, that point is called a saddle point.