TracksStudyAbout
Sign in
Start studying
Module 1 · Machine Learning Module contents
Module 1  ·  Card 38 / 55intermediate

PCA

PCA reduces the number of dimensions while keeping as much variance as it can.

It finds new axes, called principal components, that capture the most spread.

You then keep only the first few components and discard the rest.

It is useful for compression, speed and 2D visualisation.

The new components are combinations of old features, so interpretability is lost.

Back Next
Learn in depth ↓
Module 01 · Machine Learning · 55 concepts
In depth  /  15 lines

PCA, in more detail

PCA looks for the directions in which your data spreads out the most.

The first direction captures the most spread of all.

The second captures the most of what is left, at right angles to the first.

It keeps going until it has as many directions as you had columns.

The point is that the last few usually carry almost nothing.

So you keep the first handful and throw the rest away.

Two or three of them let you draw high-dimensional data on a screen.

Fewer columns also means faster training and less room to overfit.

Every new direction is a mixture of all your original features.

That is why interpretability disappears, since component one is not a thing you can name.

Scale the features first, or the column with the biggest numbers dominates everything.

It only finds straight-line structure, so curved patterns are lost.

It has no idea what you are predicting, because it never sees the labels.

So it can quietly discard the one direction that actually mattered for your task.

Fit it on the training split only, or you have leaked the test data.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath PCA

The directions it finds

PCA finds the orthogonal directions along which the data varies most, and describes each point by its coordinates along the first few of them.

Σ=1n−1∑i=1n(xi−xˉ)(xi−xˉ)⊤\Sigma = \frac{1}{n-1} \sum_{i=1}^{n} \left( \mathbf{x}_i - \bar{\mathbf{x}} \right)\left( \mathbf{x}_i - \bar{\mathbf{x}} \right)^{\top}
  • Σ\Sigma: the covariance matrix, d×dd \times d.
  • xˉ\bar{\mathbf{x}}: the mean of the data, which must be subtracted first.
Σvk=λkvk\Sigma \mathbf{v}_k = \lambda_k \mathbf{v}_k

The eigenvectors of that matrix are the principal components. The eigenvalue attached to each one is the variance of the data along it.

Sort by eigenvalue descending, and the first component is the single direction of greatest spread.

What "explained variance" means

explained by k=∑i=1kλi∑i=1dλi\text{explained by } k = \frac{\sum_{i=1}^{k} \lambda_i}{\sum_{i=1}^{d} \lambda_i}

The eigenvalues sum to the total variance in the data, so each one's share is a real fraction rather than a score.

Keeping components until that ratio reaches 0.95 is the usual rule, and it is a decision about how much you are willing to throw away.

Projecting

z=Vk⊤(x−xˉ)\mathbf{z} = V_k^{\top} \left( \mathbf{x} - \bar{\mathbf{x}} \right)
  • VkV_k: the top kk eigenvectors as columns.

Each new coordinate is the dot product of the centred point with one component, so the new features are linear combinations of the old ones.

x^=Vkz+xˉ\hat{\mathbf{x}} = V_k \mathbf{z} + \bar{\mathbf{x}}

Projecting back gives the reconstruction, and the error it leaves is exactly the variance in the components that were dropped.

One run, in order

  1. Scale the features. PCA maximises variance, and variance is measured in the features' own units.
  2. Subtract the mean of each feature.
  3. Compute the covariance matrix, or take the SVD of the centred data directly.
  4. Sort the eigenvectors by their eigenvalues, descending.
  5. Keep the top kk and project.

In practice step three uses the SVD, because forming Σ\Sigma explicitly squares the condition number and loses precision for nothing.

What it cannot do

Every component is a linear combination of all the original features, so nothing is selected and nothing is interpretable. Component one is 0.3 of everything.

It is unsupervised: it maximises variance without knowing the labels, and the direction of greatest variance can easily be the direction that separates the classes least.

And it is linear. Data on a curved manifold is flattened rather than unrolled, which is what the nonlinear methods exist for.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
What do you lose when you replace features with principal components?
Concepts these five lines lean on
Curse of DimensionalityTraining / Validation / Test SetsVariance · Data ScienceOverfittingData Leakage
Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close