TracksStudyAbout
Sign in
Start studying
Module 2 · Deep Learning Module contents
Module 2  ·  Card 11 / 25intermediate

Learning Rate Schedule

A learning rate schedule changes the step size as training goes along.

Large steps early cover ground quickly, and small steps later settle into a good spot.

Warmup starts tiny and grows, so the first few batches cannot wreck a fresh model.

Cosine decay then eases the rate smoothly down toward zero.

Nearly every large model is trained with warmup followed by decay.

Back Next
Learn in depth ↓
Module 02 · Deep Learning · 25 concepts
In depth  /  15 lines

Learning Rate Schedule, in more detail

The learning rate is how far the weights move on each update, and it need not stay fixed.

Early on the weights are wrong in obvious ways, so large steps make quick progress.

Later the model is close to a good setting and large steps overshoot it.

A schedule starts high and lowers the rate as training goes on.

Warmup does the opposite for the first few hundred steps, starting near zero and rising.

A fresh model has meaningless weights, and a full-size step on those can wreck it.

Warmup matters most with Adam, whose early estimates are unreliable.

After warmup the rate decays, and cosine decay is the usual shape.

Cosine eases down gently, holding a useful rate for longer than a straight line would.

Step decay is the older approach, cutting the rate by ten at fixed epochs.

Loss that falls then flattens abruptly usually means the rate is now too high.

Loss that crawls from the start usually means it was too low all along.

Nearly every large model is trained with warmup followed by cosine decay to near zero.

The peak rate remains the single most important hyperparameter to get right.

Get it wrong and no schedule will save the run.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath Learning Rate Schedule

Why a constant rate is the wrong answer twice

Early in training the parameters are random, so large steps are cheap and useful. Late in training they are nearly right, and a large step overshoots a minimum it had already found.

ηt=η0\eta_t = \eta_0

One constant rate has to serve both, so it is either too small at the start or too large at the end. A schedule is the admission that the right rate changes.

Warmup, and what it is protecting

ηt=ηmax⁡⋅tTwarm\eta_t = \eta_{\max} \cdot \frac{t}{T_{\mathrm{warm}}}

For the first few hundred or few thousand steps the rate is ramped from near zero up to its peak.

Adaptive optimisers need it most. The second moment estimate is built from almost no samples at step one, so the adaptive denominator is unreliable exactly when the steps are largest.

Twarm≈0.01 to 0.05 of total stepsT_{\mathrm{warm}} \approx 0.01 \text{ to } 0.05 \text{ of total steps}

Skipping warmup on a transformer is a common way to produce a loss that diverges in the first hundred steps and never recovers.

Cosine decay, which is the current default

ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡(πtT))\eta_t = \eta_{\min} + \frac{1}{2}\big(\eta_{\max} - \eta_{\min}\big)\left(1 + \cos\left(\frac{\pi t}{T}\right)\right)
  • TT: the total number of steps planned.
  • ηmin⁡\eta_{\min}: the floor, often a tenth of the peak or zero.

It falls slowly at first, fastest in the middle, and slowly again at the end, which spends most of the budget near useful rates rather than near the extremes.

ηt=η0γ⌊t/s⌋\eta_t = \eta_0 \gamma^{\lfloor t / s \rfloor}

Step decay multiplies by a constant every ss steps instead. It works, and the visible drops in the loss curve at each step are a sign of a rate that had been too large for a while.

One schedule, in order

  1. Fix the total number of steps in advance, because cosine needs to know where the end is.
  2. Ramp linearly from near zero to the peak over the warmup steps.
  3. Decay by cosine from the peak towards the floor over the remaining steps.
  4. Update the rate every step rather than every epoch.
  5. Log it alongside the loss, so a bad schedule is visible rather than inferred.

The interaction nobody expects

η∝B\eta \propto B

The peak rate and the batch size are tied. A run that works at batch 256 will often diverge at batch 1024 unless the peak rate is raised with it.

ηmax⁡∝1dmodel\eta_{\max} \propto \frac{1}{\sqrt{d_{\mathrm{model}}}}

Model width matters too. A wider model needs a smaller peak rate, which is why the same recipe fails when scaled up without adjustment.

Changing the total step count changes the whole curve, so a run stopped early under a cosine schedule did not reach the low rates where most of the final accuracy is won.

That is the one real drawback: the schedule commits to a budget before the first step, and a shortened run is not the same as a shorter schedule.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Why do large models start with a warmup rather than the full learning rate?
Concepts these five lines lean on
HyperparametersEpochs, Batches and Iterations
Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close