Self-attention has no built-in sense of order.
Without help it would treat a sentence as an unordered bag of tokens.
Positional encoding injects information about each token's position.
The original paper used fixed sine and cosine waves of different frequencies.
Modern models often use learned or rotary positional embeddings instead.
Attention treats its input as a set, not a sequence.
Every position is compared with every other in the same way, with no notion of order.
Shuffle the words and the raw mechanism computes the same thing.
That is fatal for language, where the dog bit the man is not the man bit the dog.
Position therefore has to be added to the token vectors explicitly.
The original transformer added fixed patterns of sine and cosine waves.
Each dimension uses a different frequency, so every position gets a distinct signature.
Fixed waves need no training and extend in principle to unseen lengths.
Learned positional embeddings are the alternative, with one vector per position.
They fit the data better and stop working past the longest length seen in training.
That limit is one reason context windows were historically hard to extend.
Both approaches encode absolute position, which is a slightly odd thing to care about.
What usually matters is relative distance, since two words being adjacent is the useful fact.
Rotary embeddings encode position as a rotation and give relative distance directly.
That is why nearly every recent model uses them instead.
Self-attention is permutation-equivariant: shuffle the input and the outputs shuffle with it, unchanged.
So order has to be added to the representation itself, before attention ever sees it.
It is added, not concatenated. That keeps the width fixed and lets the model decide how much of each dimension to spend on position.
Each pair of dimensions is a sine and cosine of the same frequency, and the frequency falls geometrically across the pairs.
The fastest pair alternates every couple of positions. The slowest has a wavelength of about 60,000, longer than any sequence it will see.
Together they encode position the way a binary counter does, with each dimension flipping at a different rate.
Feeding in raw would be unbounded and would dominate the embedding at long positions.
The real argument is that a fixed offset becomes a linear transformation.
That follows from the angle-addition identities, and it means the model can learn to attend "three positions back" as one linear operation that works everywhere in the sequence.
It also extrapolates in principle: the function is defined for any , including positions longer than anything in training. In practice that extrapolation degrades quickly.
Learned absolute embeddings simply train a vector per position, which works as well within the training length and not at all beyond it.
Rotary embeddings rotate the queries and keys instead of adding to the input, which makes the dot product depend on relative position directly.
That last property is why almost every recent model uses rotary rather than sinusoidal, and why sinusoidal is now mostly of historical interest.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.