Logits in LLMs, explained in 5 lines

Module 06 · LLMs · 22 concepts

In depth / 15 lines

Logits, in more detail

Logits are the raw scores the model produces before anything is turned into a probability.

There is one logit per token in the vocabulary, at every position.

With a 100,000-token vocabulary that is 100,000 numbers for a single next-token decision.

They are unnormalised, so they can be negative, and they do not sum to anything in particular.

Only the differences between them matter, since adding a constant to all changes nothing.

A higher logit means the model finds that token more plausible here.

Softmax converts them into probabilities that are positive and sum to one.

Temperature is applied to the logits before that conversion.

Dividing by a low temperature spreads the scores apart and sharpens the result.

Dividing by a high temperature pulls them together and flattens it.

Logit bias lets you add a fixed amount to particular tokens to encourage or forbid them.

Setting a large negative bias is how a token is banned outright.

Structured output enforcement works here, by masking every token the schema disallows.

The gap between the top two logits is a rough signal of how confident the model is.

It is a signal about wording, not about truth, so it is a weak guide to correctness.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Logits

The number before the probability

A logit is the raw, unbounded score a model produces for one option, before anything has turned it into a probability.

z = \mathbf{w}^{\top} \mathbf{h} + b

$\mathbf{h}$ : the model's final hidden state for this position.
$\mathbf{w}$ : one row of the output matrix, the row belonging to this token.
$z$ : the logit. Any real number: 12.4, or minus 30.

A language model produces one of these per vocabulary entry, so the output of a forward pass is a vector as long as the vocabulary.

Where the name comes from

The logit function is the inverse of the logistic sigmoid, and it maps a probability to a log-odds.

\mathrm{logit}(p) = \log \frac{p}{1-p}

So calling the pre-softmax score a logit is the claim that it lives on a log-odds scale, which is exactly what softmax then assumes when it exponentiates.

z_i - z_j = \log \frac{p_i}{p_j}

A difference of logits is a log probability ratio. A gap of 2.3 means one token is about ten times more likely than the other.

Only differences matter

\mathrm{softmax}(z + c)_i = \mathrm{softmax}(z)_i

Adding the same constant to every logit changes nothing at all. The absolute value of a single logit is therefore meaningless on its own.

That is why "the logit was 8.2" tells you nothing, and why reading confidence off a logit requires the whole vector.

Why models expose them at all

Every manipulation of the output distribution happens here, on the logits, and not on the probabilities.

Temperature divides them.
Repetition penalties subtract from the ones already used.
Logit bias adds a constant to chosen tokens, to force or forbid them.
Masking sets forbidden tokens to negative infinity.
Only then does softmax run, once, on the result.

Doing any of that to probabilities instead would require renormalising after every step, and the arithmetic is both slower and less numerically stable.

The practical trap

Every framework's cross-entropy function expects logits, not probabilities, because it fuses the softmax in for stability.

Passing already-softmaxed values applies softmax twice. The model still trains, badly, with flattened gradients and no error message anywhere.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What do logits become after softmax is applied?

LLMs

22 concepts in this module, then a graded test with every answer explained.

Study this module →