A transformer is one block repeated dozens of times, and the block has only two working parts.
Multi-head attention lets each position gather information from every other position.
The feed-forward layer then processes each position on its own, with no mixing between them.
Both parts sit inside a residual connection and are preceded by layer normalization.
Mixing across positions, then thinking at each position: that alternation is the whole architecture.
Everything in a transformer is one block, repeated, and the depth of a model is simply how many.
Learn the block and you have learned the architecture, because nothing else varies much.
The block does two things in sequence, and they divide the labour cleanly.
Multi-head attention moves information between positions: each token gathers from all the others.
This is the only part of the block where positions can see each other at all.
The feed-forward layer then works on each position independently, the same weights applied everywhere.
It is typically two linear layers with a non-linearity between them, expanding to four times the width and back.
That expansion is where most of the parameters live, usually about two thirds of the model.
A useful reading is that attention decides what to look at and the feed-forward layer decides what to make of it.
Each sub-layer is wrapped in a residual connection, so its output is added to its input rather than replacing it.
Without those residuals the gradient would not survive a stack fifty layers deep.
Layer normalization stabilises the scale, and modern models place it before each sub-layer rather than after.
That pre-norm arrangement is what allows very deep stacks to train without a learning rate warmup baby-sitting them.
Nothing in the block knows the order of the tokens, which is why positional information is injected separately.
A decoder block adds a mask so a position can only attend backwards, and that single change is what makes generation possible.
That is a modern pre-norm block in its entirety. Everything else in a large language model is this, repeated.
Attention mixes information between positions and does almost nothing to each position on its own.
The feed-forward layer transforms each position on its own and mixes nothing between them.
Neither can do the other's job, and a block is exactly one of each.
Attention works across positions, and the feed-forward network works within one.
Every sublayer is wrapped in a residual connection and a normalisation, and both are load-bearing.
The residual gives the gradient a path with an identity in it, which is what lets the stack be 80 layers deep instead of 12.
The normalisation keeps the scale of the activations stable as they pass through those additions, which would otherwise grow without bound down the stack.
Attention contributes four squared, the feed-forward layer eight.
At that is about 200 million parameters per block, and a 70B model is roughly 80 such blocks plus embeddings.
That approximation is accurate to within a few percent for most decoder-only models, and it is worth being able to do in your head.
Very little. The activation function, whether normalisation is pre or post, whether the feed-forward layer is gated, how positions are encoded, and how many key-value heads are shared.
The shape above has been stable since 2017, and almost all of the improvement since has come from scale, data and training procedure rather than from the block.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.