Temperature in LLMs, explained in 5 lines

Module 06 · LLMs · 22 concepts

In depth / 15 lines

Temperature, in more detail

Temperature is the single knob that controls how adventurous the output is.

It divides the logits before softmax, which reshapes the distribution.

A value below one spreads the scores apart and concentrates probability on the front-runner.

A value above one pulls them together and gives unlikely tokens a real chance.

At zero it is effectively deterministic and always takes the most likely token.

That is what you want for classification, extraction, code and anything with a right answer.

Around 0.7 is the usual default for conversation, giving variety without drifting.

Above about 1.2 the output becomes noticeably strange, and higher still it degrades into nonsense.

Higher temperature is not more creative in any deep sense, only less predictable.

The effect compounds, since one unusual token pushes the rest of the sentence somewhere unusual.

Even at zero the output is not guaranteed identical, because of batching and floating-point order.

It is usually combined with top-p rather than used alone.

Temperature reshapes the distribution and top-p truncates it, so they do different jobs.

If output is repetitive, raise it slightly before reaching for anything else.

If output is unreliable or invents details, lower it before rewriting the prompt.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Temperature

Dividing the logits

Temperature does exactly one thing: it divides the logits before the softmax sees them.

p_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}

$z_i$ : the logit for token $i$ .
$T$ : the temperature, a positive number.
$p_i$ : the probability the token ends up being sampled with.

At $T = 1$ the division does nothing and you get the model's own distribution, unmodified.

What the two directions do

Recall that softmax turns logit differences into probability ratios.

\frac{p_i}{p_j} = e^{(z_i - z_j) / T}

Below 1, the division makes every gap bigger, so the ratio between a likely token and an unlikely one grows and the distribution sharpens.

Above 1, the division shrinks every gap, so the ratios move towards one and the distribution flattens.

The model's ranking of tokens never changes. Only how strongly it prefers its favourite does.

The two limits

\lim_{T \to 0^{+}} p_i = \begin{cases} 1 & i = \arg\max_j z_j \\ 0 & \text{otherwise} \end{cases}

At zero the distribution collapses onto the single highest logit, which is greedy decoding.

Implementations special-case this rather than dividing by zero, which is why setting temperature to 0 is a valid setting and not an error.

\lim_{T \to \infty} p_i = \frac{1}{K}

Very high temperature approaches a uniform distribution over the whole vocabulary, which is sampling text at random.

One token, in order

The model produces one logit per vocabulary entry.
Divide every logit by the temperature.
Apply any truncation, such as top-k or top-p.
Softmax what is left into probabilities.
Draw one token from that distribution.

What it is not

Temperature is not a confidence setting and not a creativity dial, even though both are how it is usually described.

It cannot make the model know something it does not, and it cannot change which tokens the model rates low. It can only make an unlikely token more likely to be picked.

Raising it does not produce better ideas, it produces more sampling from the tail.

The tail holds the interesting continuations and the wrong ones together, in whatever ratio the model put them there.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Setting temperature to 0 has what effect?

LLMs

22 concepts in this module, then a graded test with every answer explained.

Study this module →