AdamW is Adam with weight decay handled correctly.
Plain Adam mixes L2 penalty into the gradient, which interacts badly with its adaptive scaling.
AdamW instead decays the weights as a separate step.
This gives noticeably better generalisation in practice.
It is the standard optimizer for training transformers and large models.
AdamW is Adam with one bug fixed, and that fix is worth a version of its own.
Weight decay pulls every weight slightly towards zero on each step.
It keeps weights small, which limits how sharply the model can fit its training data.
The old way was to add an L2 penalty to the loss and let the gradient carry it.
That works with plain SGD, where the penalty reaches every weight equally.
It breaks with Adam, which divides each gradient by that weight's own noise estimate.
The decay is divided too, so noisy weights are barely decayed and quiet ones are decayed hard.
The regularisation you asked for lands unevenly and largely at random.
AdamW takes the decay out of the gradient and applies it as a separate subtraction.
Every weight is then decayed by the same fraction, as intended.
The W stands for decoupled weight decay, which is exactly what changed.
The measured result is better generalisation across most tasks.
It is the standard optimizer for transformers and every large model in current use.
Biases and normalisation parameters are usually excluded from decay.
If you are choosing between them today, choose AdamW.
L2 regularisation and weight decay are the same thing for plain SGD, and they are not the same thing for Adam.
L2 adds the penalty to the loss, so the decay term enters as part of the gradient.
In Adam that gradient is then divided by the running root mean square of past gradients, which divides the decay term too.
A parameter with large historical gradients gets a large denominator, so its decay is scaled down. A parameter with tiny gradients is decayed hard.
The amount of regularisation a weight receives therefore depends on its gradient history, which is not what anybody asking for weight decay meant.
AdamW applies the decay to the parameters directly, outside the adaptive rescaling.
Every weight is now pulled towards zero by the same proportion per step, which is what weight decay is supposed to be.
Steps four and five are independent, and keeping them independent is the entire contribution of the method.
Decoupling made the decay rate tunable again. Under Adam with L2, changing the learning rate silently changed the effective regularisation, so the two could not be tuned separately.
AdamW is now the default optimiser for transformers, and every current large language model is trained with it.
Biases, and the gain and shift parameters of normalisation layers, are normally left out of the decay entirely.
They are few, they are not a capacity risk, and shrinking a normalisation gain towards zero attacks the layer's ability to scale rather than its ability to overfit.
Most frameworks do not do this by default, so it is a parameter group that has to be written by hand, and it is worth a few tenths of a point.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.