Early Stopping in deep learning

Module 02 · Deep Learning · 25 concepts

In depth / 15 lines

Early Stopping, in more detail

Early stopping is the cheapest regulariser there is, since it only means stopping sooner.

Training loss falls for as long as you let it, because memorising always lowers it further.

Validation loss falls too, up to the point where the model runs out of real pattern to find.

After that it starts fitting the noise, and validation loss turns upward.

That turn is the boundary between learning and memorising, and it is visible on the curve.

Early stopping watches for it and ends the run.

The curve is noisy, so a single worse epoch means nothing.

Patience is how many epochs of no improvement you tolerate before calling it, often five or ten.

The weights from the best epoch are kept, not the ones from the epoch you stopped on.

That requires saving a checkpoint whenever validation improves.

The decision must be made on validation data, never on the test set.

Using the test set to choose the stopping point leaks it, and the final score stops being honest.

It saves compute as well as accuracy, since the useless epochs never run.

It works alongside dropout and weight decay rather than replacing them.

If validation loss never turns upward, the model is too small rather than trained too long.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Early Stopping

Stopping is a regularisation decision

\mathcal{L}_{\mathrm{train}} \downarrow \text{ always}, \qquad \mathcal{L}_{\mathrm{val}} \downarrow \text{ then } \uparrow

Training loss falls for as long as you keep training. Validation loss falls, reaches a minimum, and then rises as the model starts fitting noise.

The minimum of the validation curve is the model you want, and every step past it is spent making the model worse.

t^{*} = \arg\min_{t} \; \mathcal{L}_{\mathrm{val}}(t)

Early stopping is the rule that finds that point without knowing in advance where it is.

Patience, because the curve is noisy

\text{stop if } \mathcal{L}_{\mathrm{val}}(t) > \mathcal{L}^{*} - \delta \text{ for } p \text{ evaluations}

$p$ : the patience, how many evaluations of no improvement are tolerated.
$\delta$ : the minimum change that counts as an improvement.

Validation loss wobbles from one evaluation to the next, so stopping at the first rise stops almost immediately and almost always too early.

Patience of five to ten evaluations is typical, and it should be larger when the schedule still has a large learning rate to decay.

The threshold $\delta$ exists so that an improvement of $10^{-6}$ does not reset the counter forever.

One training run, in order

Evaluate on the validation set at a fixed interval.
If the score improved, record it and save the weights.
If not, increment the patience counter.
When the counter reaches $p$ , stop training.
Restore the saved weights, which are not the ones in memory at the moment you stopped.

Step five is the one that gets skipped.

Stopping after $p$ bad evaluations leaves the model $p$ evaluations past its best, and without the restore the procedure hands back a worse model than it found.

Why it is equivalent to a penalty

\|\boldsymbol{\theta}_t - \boldsymbol{\theta}_0\| \leq \eta \, t \, \|\mathbf{g}\|_{\max}

Starting from a small initialisation, the parameters can only travel so far in $t$ steps, so stopping early keeps them inside a ball around where they began.

For a linear model with squared loss that correspondence is exact: stopping at step $t$ gives the same solution as L2 regularisation with a particular $\lambda$ .

So early stopping is not merely a convenience. It is a regularisation dial that happens to be free, tuned on the validation set by construction.

What it interacts badly with

A learning rate schedule that decays to zero already produces much of its improvement in the final steps, so aggressive patience cuts off the part of the run that mattered.

\mathcal{L}_{\mathrm{val}} \uparrow \text{ while accuracy } \uparrow

Validation loss and validation accuracy can move in opposite directions late in training, as the model becomes more confident where it is right and more wrong where it is not.

Stop on the metric you are judged by, not on the loss, whenever the two disagree.

The validation set is now part of the training procedure, so a score read off it is optimistic and a separate test set is required to report anything.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Early stopping decides when to stop by watching which number?

Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →