Gradient clipping caps how large a single training step is allowed to be.
Occasionally one batch produces an enormous gradient that would wreck the weights.
Clipping scales it back down while keeping its direction.
It is standard practice for recurrent networks and for large language models.
A loss that suddenly becomes nothing at all is the symptom it usually cures.
Occasionally one batch produces a gradient hundreds of times larger than usual.
The optimizer takes it at face value and throws the weights far out of position.
One bad step can undo hours of training and leave the loss stuck at nonsense.
Gradient clipping puts a ceiling on how big a step is allowed to be.
Norm clipping measures the overall length of the gradient across all parameters.
If the length is above the limit, everything is scaled down by the same factor.
Scaling everything together preserves the direction and only shortens the step.
Value clipping instead caps each number individually, which does distort the direction.
Norm clipping is the one nearly everyone uses, with a limit around one.
Recurrent networks need it most, because the same weights are multiplied at every timestep.
Large language model training uses it as a matter of course.
The symptom it cures is a loss that suddenly jumps or becomes not-a-number.
It is a safety rail, not a fix for a badly set learning rate.
If clipping is firing on most batches, the learning rate is too high.
It costs almost nothing, which is why it is left switched on by default.
The gradient of a whole model is one long vector, and its norm is a single number describing how big the step would be.
Below the threshold nothing happens. Above it, the whole vector is scaled down so its norm is exactly .
The direction is untouched, which is the point: the step still goes where the gradient said, just not as far.
Clipping each element separately also bounds the step, and it changes the direction of the update, because the elements that were clipped shrink and the others do not.
On a gradient where one parameter dominates, value clipping turns a step along one axis into a step along a completely different one.
Norm clipping rescales everything by the same factor, so the update remains parallel to the true gradient. It is the default for that reason alone.
Clipping happens between backward and step. Applied after the optimiser it does nothing, and applied before backward there is nothing to clip.
Train for a few hundred steps without clipping, record the norms, and set the threshold near the top of the normal range rather than at the middle.
A threshold that clips most steps is a learning rate reduction wearing a disguise, and it will slow training without saying so.
One is the conventional value for transformers, and it works because the norms of a well-scaled model sit below it in ordinary steps.
It makes exploding gradients survivable. A single bad batch that would have produced a NaN instead produces a normal-sized step.
It does nothing at all for vanishing gradients, since scaling down a tiny gradient is not something the rule ever triggers.
It is essential in recurrent networks, where the same matrix is applied at every timestep and the norm genuinely can explode by orders of magnitude on one sequence.
A sudden spike in the logged pre-clip norm is worth chasing rather than clipping away. It usually means a corrupt batch, a label error, or a learning rate that is already too high.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.