Rotary embeddings encode position by rotating each token's vector by an angle.
The size of the angle depends on where the token sits in the sequence.
Because attention compares rotations, the model sees relative distance rather than absolute position.
That extends to longer sequences far better than adding a fixed position vector.
Most recent open language models use it instead of the original scheme.
Rotary embeddings encode position by rotating a token's vector rather than adding to it.
Pairs of dimensions are treated as coordinates on a plane and turned by an angle.
The angle is proportional to the token's position in the sequence.
Different pairs rotate at different rates, from very fast to very slow.
The clever part is what happens when two rotated vectors are compared.
Attention takes a dot product, and rotating both vectors leaves only the difference in angle.
So the score depends on how far apart the tokens are, not where they sit absolutely.
Relative distance falls out of the geometry rather than being bolted on.
That is what the model actually needs, since adjacency matters and absolute index rarely does.
It also degrades gracefully on sequences longer than those seen in training.
Interpolating or rescaling the frequencies extends a model's context window after the fact.
That trick is how many models gained much longer windows without full retraining.
It adds no parameters, since the rotation is computed rather than learned.
It is applied inside each attention layer to the queries and keys, not once at the input.
Llama, Mistral, Qwen and most recent open models use it.
The original transformer adds a position vector to the token embedding, which mixes what a token is with where it sits, in the same coordinates.
Rotary embeddings instead rotate the query and key vectors by an angle proportional to their position, leaving the embedding itself untouched.
The vector is taken two dimensions at a time and each pair is rotated in its own plane, so the operation is a block-diagonal rotation matrix.
A rotation preserves length, so no information about the token is scaled away, which addition cannot promise.
Two rotations composed give a rotation by the difference of the angles, so the attention score depends only on .
That is the whole result. Absolute positions go in and only the relative distance comes out, without any extra parameters or a bias table.
Each pair of dimensions rotates at a different frequency, fast in the early dimensions and slow in the later ones.
The slow frequencies encode coarse position across a long document and the fast ones encode fine local order, which is the same reasoning as the sinusoidal scheme it replaced.
Values carry content rather than being compared, so rotating them would distort what is retrieved without changing what is matched.
There are no learned position parameters, so nothing has to be resized when the context length changes.
It is applied inside every attention layer rather than once at the input, so the position signal does not have to survive fifty layers of mixing.
Scores decay as grows.
Distant tokens interfere across many frequencies at once, which produces a gentle decay with distance that matches how language actually behaves.
Position interpolation divides every frequency by the scaling factor, so a model trained at 4,000 tokens can read 32,000 by rotating more slowly.
It works because the model has seen every angle already. It is being asked to interpolate rather than to extrapolate, and a short fine-tune closes the remaining gap.
NTK-aware scaling refines this.
The refinement scales the slow dimensions harder than the fast ones, so long-range position stretches while local word order stays exactly as trained.
That is how almost every open model published with a long context reached it, and it is why RoPE is now near-universal.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.