Matrix properties

In this section, we will look at some of the important properties matrices which are very useful for deep learning applications.

• Norm: Norm is an important property of a vector or a matrix that measures the size of the vector or the matrix. Geometrically it can also be interpreted as the distance of a point, x, from an origin. A L_p norm is therefore defined as follows:

Though a norm can be computed for various orders of p, most popularly known norms are L1 and L₂ norm. L₁ norm is usually considered a good choice for sparse models:

Another norm popular in the deep learning community is the max norm, also referred to as L^∞. This is simply equivalent to the value of the largest element in the vector:

So far, all the previously mentioned norms are applicable to vectors. When we want to compute the size of a matrix, we use Frobenius norm, defined as follows:

Norms are usually used as they can be used to compute the dot product of two vectors directly:

• Trace: Trace is an operator that is defined as the sum of all the diagonal elements of a matrix:

Trace operators are quite useful in computing the Frobenius norm of the matrix, as follows:

Another interesting property of trace operator is that it is invariant to matrix transpose operations. Hence, it is often used to manipulate matrix expressions to yield meaningful identities:

• Determinant: A determinant of a matrix is defined as a scalar value which is simply a product of all the eigenvalues of a matrix. They are generally very useful in the analysis and solution of systems of linear equations. For instance, according to Cramer's rule, a system of linear equations has a unique solution, if and only if, the determinant of the matrix composed of the system of linear equations is non-zero.