Layer normalization is applied around every sublayer.
It normalises across the features of each token independently.
This keeps activation scales stable as depth increases.
Post-norm places it after the residual add, which the original paper used.
Pre-norm places it before the sublayer and is more stable for very deep models.
Normalisation sits around every sublayer in a transformer, and the model does not train without it.
Layer normalization rescales the features of one token using only that token.
Batch normalization would use the other examples in the batch, which suits sequences badly.
Sequences vary in length and batches are often small, so those statistics are unreliable.
Normalising per token sidesteps both problems entirely.
It subtracts the mean across the features and divides by their spread.
A learned scale and shift then let the model undo the normalisation where that is better.
The purpose is to stop activation scales drifting as depth increases.
In a stack of 80 layers a small drift compounds into an unusable range.
The placement relative to the residual addition turns out to matter a great deal.
Post-norm normalises after the addition, which is what the original paper did.
Pre-norm normalises the input before the sublayer and leaves the residual path clean.
Pre-norm trains far more stably at depth and is now standard.
RMSNorm is a cheaper variant that skips the mean and divides by the root mean square.
It performs about as well and most recent large models use it.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.