Speculative Decoding in LLMs, explained in 5 lines

Module 16 · Advanced AI · 23 concepts

In depth / 15 lines

Speculative Decoding, in more detail

Speculative decoding makes generation faster without changing the output.

Decoding is slow because tokens must be produced one at a time.

Each one needs a full pass over the weights, and the GPU is barely busy during it.

The bottleneck is reading the weights from memory, not the arithmetic.

So checking several tokens costs almost the same as checking one.

A small draft model guesses the next few tokens quickly.

The large model then evaluates all of those positions in a single pass.

It checks each guess against what it would have produced itself.

Guesses that match are accepted, and the first mismatch is corrected.

Everything after that mismatch is discarded and the process repeats.

The output is provably identical to what the large model would have generated alone.

That is what makes it a free optimisation rather than a quality trade.

Speedups of two to three times are typical when the draft model agrees often.

The draft model must be fast and well aligned with the large one, often a distilled version.

Variants such as Medusa predict several tokens from the large model itself, avoiding a second model.

Deep dive

What is underneath Speculative Decoding

Verifying many tokens for the price of one

t_{\mathrm{token}} \approx \frac{\text{bytes of weights}}{\text{bandwidth}}

Decoding is memory-bound, so a forward pass that scores one token and one that scores five cost almost the same.

A small model proposes a short continuation, and the large model checks the whole proposal at once.

Every accepted token was produced without its own pass through the large model, which is where the speedup comes from.

The acceptance rule preserves the distribution

\text{accept } x_i \text{ with probability } \min\left(1, \; \frac{p(x_i)}{q(x_i)}\right)

$p$ : the large model's probability for that token.
$q$ : the draft model's.

If the target likes the token at least as much as the draft did, it is accepted outright. Otherwise it is accepted in proportion.

p'(x) = \frac{\max\big(0, \; p(x) - q(x)\big)}{\sum_{x'} \max\big(0, \; p(x') - q(x')\big)}

On a rejection, the token is resampled from the residual distribution, which is exactly what is left of $p$ after removing what $q$ already covered.

The output distribution is provably identical to sampling from the large model alone.

This is an exact method, not an approximation, which is why it can be switched on without evaluating quality.

The expected gain

\mathbb{E}[\text{tokens per pass}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}

$\alpha$ : the average acceptance rate.
$k$ : the number of drafted tokens.

With $\alpha = 0.8$ and $k = 4$ , that is about 3.4 tokens per verification pass.

\text{speedup} = \frac{\mathbb{E}[\text{tokens}]}{1 + k \, c}

$c$ : the draft model's cost relative to the target's.

The draft is not free, so a draft model a tenth the size and a $k$ of four or five is the usual balance, giving two to three times the throughput.

One step, in order

The draft model generates $k$ tokens autoregressively from the current context.
The target model runs one forward pass over the context plus all $k$ tokens.
Compare the two distributions at each position and accept by the rule, left to right.
Stop at the first rejection and resample that position from the residual.
Append the accepted tokens and the resampled one, then draft again.

Every rejection still yields one token, so a step never makes no progress.

What decides whether it pays

The acceptance rate $\alpha$ depends on agreement, not on quality.

The draft model must be trained on similar data and share the tokeniser, or the acceptance rate collapses and the drafting is pure overhead.

Predictable text accepts well. Code, boilerplate and formatting are drafted almost perfectly, while genuinely uncertain continuations are rejected often.

The larger the batch, the smaller the gain.

At large batch sizes the target model is already compute-bound, so the spare capacity the method exploits no longer exists.

Speculative decoding is therefore mainly a low-latency, small-batch technique.

Variants remove the second model entirely, drafting with a subset of the target's own layers or with several extra prediction heads, which avoids maintaining two models in the serving stack.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What is given up by using speculative decoding?

Advanced AI

23 concepts in this module, then a graded test with every answer explained.

Study this module →