Top-p Sampling in LLMs, explained in 5 lines

Module 06 · LLMs · 22 concepts

In depth / 15 lines

Top-p Sampling, in more detail

Top-p sampling, also called nucleus sampling, cuts by probability mass rather than by count.

Sort the tokens by likelihood and take them until their probabilities sum to p.

With p at 0.9 you keep just enough tokens to cover 90 per cent of the mass.

How many that is changes at every single step.

When the model is confident the leader may hold 95 per cent, so one token survives.

When it is genuinely open the mass is spread, so dozens survive.

That adaptivity is the whole argument for it over top-k.

The tail is still cut, so bizarre tokens remain unreachable.

Common defaults are 0.9 to 0.95, and 1.0 disables it entirely.

Lowering p tightens the output towards the safe and predictable.

Raising it loosens it, at the cost of the occasional strange word.

It is usually paired with temperature, which reshapes before the cut is applied.

Changing both at once makes the effect of either hard to judge.

Adjust one at a time and keep a fixed set of prompts to compare against.

For factual work, low temperature matters more than the exact p you choose.

Deep dive

What is underneath Top-p Sampling

Truncate by mass, not by count

Top-p keeps the smallest set of tokens whose probabilities add up to at least $p$ , and throws the rest away.

V_p = \min \left\{ V' \subseteq V : \sum_{i \in V'} p_i \geq p \right\}

$V$ : the whole vocabulary.
$V_p$ : the nucleus: the tokens that survive.
$p$ : the mass to keep, typically 0.9 or 0.95.

In practice you sort by probability descending and walk down the list adding probabilities until the running total crosses $p$ .

Then renormalise

The surviving probabilities no longer sum to one, so they are rescaled.

p'_i = \begin{cases} \dfrac{p_i}{\sum_{j \in V_p} p_j} & i \in V_p \\[6pt] 0 & \text{otherwise} \end{cases}

Without that division the sampler would be drawing from a distribution summing to about 0.9, and the missing mass has to go somewhere.

Why a mass threshold beats a fixed count

The size of the nucleus changes with how certain the model is, and that is the entire point.

After "the capital of France is" the model may put 0.99 on one token, so the nucleus at $p = 0.9$ contains exactly one token and sampling is effectively greedy.

After "she opened the door and saw" the mass is spread over hundreds of plausible continuations, so the nucleus is large and the sampler has real freedom.

Top-k cannot do this: $k = 50$ admits 49 bad tokens in the first case and cuts off good ones in the second.

One token, in order

Get the logits and apply temperature.
Softmax them into probabilities.
Sort descending and accumulate until the total reaches $p$ .
Zero everything outside that set and rescale what is left to sum to one.
Draw one token from the rescaled distribution.

What the values do

At $p = 1.0$ nothing is truncated and you are sampling from the full distribution, tail included.

At $p \to 0$ only the single most likely token survives, which is greedy decoding again.

The interesting range is narrow: below about 0.8 the text tends to loop, above about 0.98 the tail starts leaking in nonsense.

It composes with temperature rather than replacing it: temperature reshapes the distribution and top-p then cuts it.

The order matters, because cutting first would truncate a distribution that is about to change shape.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why is top-p usually preferred over top-k?

LLMs

22 concepts in this module, then a graded test with every answer explained.

Study this module →