Pre-norm applies layer normalization before the attention or feed-forward block.
The original transformer applied it afterwards, which is called post-norm.
Post-norm needs a careful warmup, or very deep models fail to train at all.
Pre-norm keeps the residual path clean, so gradients still reach the early layers.
Almost every large model built since uses pre-norm for that reason.
Pre-norm and post-norm differ only in where the normalisation sits.
Post-norm runs the sublayer, adds the residual, then normalises the result.
Pre-norm normalises the input, runs the sublayer, then adds the residual.
The consequence is about what happens to the residual path.
In post-norm the path passes through a normalisation at every layer.
Gradients are rescaled at each step and, deep in the stack, arrive unhelpfully small.
In pre-norm the path is a plain chain of additions from input to output.
A gradient can travel the whole model without being rescaled once.
Post-norm models need a careful warmup schedule or they diverge in the first few hundred steps.
That fragility grows with depth, and beyond a few dozen layers it becomes acute.
Pre-norm trains reliably at eighty layers or more with far less ceremony.
The original 2017 transformer used post-norm, which is why the warmup recipe became folklore.
GPT-2 moved to pre-norm and essentially every large model since has followed.
Post-norm is sometimes argued to reach slightly better final quality when it can be trained.
Its instability is the reason almost nobody takes that bet.
The original transformer normalised after adding the residual. Every model since about 2019 normalises before the sublayer instead.
The difference is one line and it decides whether a deep transformer trains at all.
Under pre-norm the residual stream runs from the input to the output with nothing applied to it, so every block writes into a shared, unmodified channel.
The gradient therefore reaches layer zero through a path that multiplies by the identity, whatever the depth.
Post-norm puts a normalisation on that path at every layer, so the gradient is rescaled times and the product can shrink geometrically again.
Post-norm transformers need a careful learning rate warmup, and without it the loss diverges in the first few hundred steps.
Pre-norm trains with a short warmup or none, tolerates a larger learning rate, and scales past a hundred layers where post-norm stalls near twenty.
The cost is that the residual stream grows as blocks keep adding to it, so activations at the last layer are much larger than at the first.
That is why a pre-norm stack ends with one final normalisation before the output projection. Without it the logits inherit the accumulated scale.
RMSNorm drops the mean subtraction and keeps only the rescaling, which is cheaper and works as well, so most current models use it in place of layer norm.
Some large models scale each residual branch by a small factor at initialisation, so the stack begins close to the identity and the blocks learn to contribute gradually.
Post-norm has not disappeared entirely. It produces slightly better final quality when it can be trained, which is why a few models use hybrid schemes that normalise in both places.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.