Decoder-Only Architecture in transformers

Module 05 · Transformers · 19 concepts

In depth / 15 lines

Decoder-Only Architecture, in more detail

Decoder-only models drop the encoder and keep a single stack.

Prompt and response live in one continuous stream of tokens.

There is no boundary in the architecture between reading and writing.

The model is trained to do exactly one thing, which is predict the next token.

That simplicity is the reason the design won.

One objective, one stack, and every document on the internet as training data.

Tasks that once needed an encoder are handled by putting the input into the prompt.

Translation becomes a request to translate rather than a specialised architecture.

It scaled better than the alternatives, and scaling turned out to be the deciding factor.

GPT, Claude, Llama, Mistral and Gemini are all built this way.

The cost is that attention is masked, so early tokens never see later ones.

A pure encoder understands its input slightly better for a fixed size.

In practice the size and training scale of these models buries that disadvantage.

It also means one model serves every task, which changed how software is built around it.

The whole industry converged on this shape within about three years.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Decoder-Only Architecture

One stack, one objective

An encoder-decoder model has two stacks and three attention types. A decoder-only model has one stack, one attention type, and one loss.

\mathcal{L} = -\sum_{t=1}^{T} \log p\big(x_t \mid x_{<t}\big)

Every token is a prediction target, and the input and the label are the same sequence shifted by one position.

That is why the architecture won. The training signal is $T$ predictions per sequence rather than one, and it needs no labelled pairs at all.

The mask is the architecture

M_{ij} = \begin{cases} 0 & j \leq i \ -\infty & j > i \end{cases}

\mathrm{Attn} = \mathrm{softmax}\left( \frac{QK^{\top}}{\sqrt{d_k}} + M \right) V

Adding negative infinity before the softmax sends those weights to exactly zero, so a position can attend to itself and everything before it and nothing after.

Without the mask, predicting the next token would be trivial: the answer is one of the inputs.

The mask is added rather than applied afterwards, because zeroing the weights after a softmax would leave them unnormalised.

One training step, in order

Take a sequence of $T$ tokens as the input, and the same sequence shifted left as the target.
Run every block with causal masking, in parallel across all positions.
Project each position's final state to logits over the vocabulary.
Compute cross-entropy at every position at once.
Average across positions and tokens, and step.

The parallelism in step two is the reason this trains where a recurrent network cannot.

Masking makes all $T$ predictions independent given the weights, so one forward pass produces all of them.

Generation is the opposite shape

\text{prefill} : O(T^2 d) \text{ once}, \qquad \text{decode} : O(T d) \text{ per token}

Training processes a whole sequence in one pass. Generation must produce one token, append it, and run again, so it is sequential by construction.

The KV cache is what makes this affordable: the keys and values of earlier tokens do not change, so each new step computes one new column rather than the whole matrix.

Prefill is compute-bound and decode is memory-bound, and that asymmetry is behind almost every serving optimisation.

Why encoders lost the argument

\text{BERT} : 15\% \text{ of tokens are targets}, \qquad \text{GPT} : 100\%

Masked language modelling trains on a fraction of the positions, which is a less efficient use of the same corpus and the same compute.

A bidirectional encoder also cannot generate. Predicting a masked token in place is a different task from continuing a sequence, and only one of them composes into open-ended output.

Translation, summarisation, classification and dialogue are all expressible as continuing a prompt, so one stack and one objective absorbed tasks that used to need separate architectures.

What was given up is real: a decoder's representation of a token cannot depend on what follows it.

For pure understanding tasks with a fixed input, an encoder still measures better per parameter.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What is a decoder-only model trained to do?

Transformers

19 concepts in this module, then a graded test with every answer explained.

Study this module →