Cross-Validation in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Cross-Validation, in more detail

A single train and test split leaves a great deal to luck.

Draw an easy test set and your model looks better than it is.

Cross-validation removes most of that luck.

Split the data into five or ten equal parts, called folds.

Train on all but one fold, then score on the fold you left out.

Rotate until every fold has taken its turn as the test set.

Then average the scores you collected.

You also get the spread, which matters as much as the average does.

A wide spread means your estimate is fragile and your data is small.

It costs five or ten times the training time, which is the trade you make.

It matters most on small datasets, where one split is unreliable.

Every step that learns from data must happen inside each fold.

Scaling or feature selection done beforehand leaks across the folds.

For time-ordered data, use folds that always train on the past.

For grouped data, keep whole groups inside a single fold.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Cross-Validation

One split is a single noisy measurement

A held-out test set gives one number, computed on one arbitrary slice of the data.

\mathrm{Var}(\hat{e}) = \frac{e(1-e)}{m}

$e$ : the model's true error rate.
$m$ : the size of the test set.

A model with a true error of 10 percent, measured on 200 rows, has a standard error of about 2 points.

So two models that differ by a point on such a split are not distinguishable, and the split itself decided which one looked better.

What k-fold averages away

\mathrm{CV}_k = \frac{1}{k} \sum_{j=1}^{k} e_j

$k$ : the number of folds.
$e_j$ : the error on fold $j$ , measured by a model trained on the other $k-1$ .

Every row is used for validation exactly once and for training $k-1$ times, so the estimate uses all the data and the model still never scores itself.

k \in \{5, 10\}

Five or ten is the standard choice, and it is a compromise: higher $k$ trains on more data so bias falls, but the folds overlap more so the average is less stable.

One run, in order

Shuffle the rows, unless they are ordered in time.
Cut them into $k$ folds of equal size.
For each fold, train on the other $k-1$ and score the held-out one.
Average the $k$ scores, and report their spread alongside.
Retrain on all the data for the model you actually ship.

The variants that exist for a reason

Stratified k-fold keeps each class at the same proportion in every fold, which matters as soon as one class is rare enough that a fold could miss it.

Group k-fold keeps all rows sharing an identifier in the same fold, so a patient or a user cannot appear in training and validation at once.

\text{train} = [1 \ldots t], \quad \text{validate} = [t+1 \ldots t+h]

Time series cross-validation never shuffles. Each fold trains on the past and validates on the future, because a shuffled split lets the model see tomorrow.

k = n

Leave-one-out is k-fold taken to its limit: almost unbiased, expensive, and high variance because the $n$ training sets are nearly identical.

The mistake that quietly inflates every number

Any step fitted on the data belongs inside the loop, not before it.

Scaling, imputation, feature selection and target encoding all learn from the rows they see, so fitting them on the full dataset leaks the validation folds into training.

Selecting the best of 200 hyperparameter settings by cross-validation also makes that score optimistic, because the choice was made using it.

A second, outer split is the only honest way to report what such a model scores on data nothing has touched.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why does k-fold beat a single train/test split on a small dataset?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →