When a matrix multiplies a vector, it typically rotates the vector and alters its length simultaneously.
However, for certain special vectors, the matrix multiplication does not rotate the vector at all.
It simply stretches, shrinks, or flips the vector along its original line of action.
These invariant direction vectors are called eigenvectors of the matrix.
The scalar factor by which an eigenvector is scaled during this transformation is called its eigenvalue.
The relationship is expressed by the famous equation: matrix A times vector v equals eigenvalue lambda times vector v.
A square n-by-n matrix can have up to n linearly independent eigenvectors and n corresponding eigenvalues.
If an eigenvalue is negative, the matrix reverses the direction of the eigenvector along its line of action.
If an eigenvalue is zero, the matrix collapses that entire direction onto zero, indicating rank deficiency.
For real symmetric matrices, like covariance matrices, all eigenvalues are real numbers and eigenvectors are mutually orthogonal.
Eigendecomposition factors the matrix into a product of its eigenvector matrix, a diagonal eigenvalue matrix, and the inverse eigenvector matrix.
In data analysis, finding the eigenvectors of a covariance matrix uncovers the directions of greatest variance across the data.
The eigenvector with the largest eigenvalue points along the direction where data values vary the most.
Discarding eigenvectors with tiny eigenvalues allows analysts to compress datasets while retaining almost all variation.
Eigendecomposition transforms complex multi-dimensional relationships into independent orthogonal axes of maximum variance.