In ordinary arithmetic, the number one is the multiplicative identity because multiplying by one changes nothing.
In matrix algebra, the identity matrix, denoted by capital I, plays exactly that multiplicative role.
It is a square matrix whose diagonal entries are all ones and whose off-diagonal entries are all zeros.
Multiplying any compatible matrix A by the identity matrix leaves A unchanged: A times I equals A.
Similarly, in scalar arithmetic, the reciprocal of five is one fifth because five times one fifth equals one.
An inverse matrix, denoted as A with a superscript negative one, acts as the matrix equivalent of a reciprocal.
When a square matrix A multiplies its inverse, the product is the identity matrix I: A times A inverse equals I.
Matrix inverses only exist for square matrices, and not every square matrix possesses an inverse.
A square matrix that lacks an inverse is called a singular matrix or non-invertible matrix.
A matrix is singular when its columns are linearly dependent or its determinant equals zero.
If a matrix is singular, it collapses geometric space, making it impossible to reverse the transformation uniquely.
In regression formulas, solving for optimal model coefficients requires inverting a feature cross-product matrix.
If features are perfectly collinear, the cross-product matrix becomes singular and cannot be inverted.
In practice, computers solve linear equations using numerical decomposition rather than explicitly calculating full inverses.
Explicit inversion is computationally expensive and prone to rounding errors on large datasets.