Perplexity measures how surprised a language model is by text it has not seen.
It is worked out from the probability the model gave to each word that actually came next.
A perplexity of 20 means it was as unsure as if choosing between 20 equally likely words.
Lower is better, and a model that predicted every word perfectly would score one.
It only compares models sharing a tokenizer, since splitting words differently changes the count.
Perplexity scores a language model on text it has never seen.
At each word the model gave a probability to what actually came next.
A good model gives high probability to the real word and a poor one spreads its bets.
Perplexity turns those probabilities into a single number for the whole test set.
It is the exponential of the average negative log probability per token.
The plainer reading is the effective number of words the model was choosing between.
A perplexity of 20 means it was as unsure as if picking uniformly from 20 words.
A perfect model scores one, and lower is always better.
Early n-gram models scored in the hundreds and modern models score in the low tens.
The number only means anything on the same test set and the same tokenizer.
A model that splits text into smaller pieces predicts each one more easily and scores lower.
So comparing perplexity across tokenizers is a comparison of nothing.
It also says nothing about whether the output is useful, true or safe.
A model can have excellent perplexity and still be unhelpful to talk to.
It remains the standard measure of pretraining progress and a poor measure of a product.
The expression inside the exponential is exactly the average cross-entropy loss, so perplexity is a monotone reparameterisation of the training objective.
A loss of 2.0 is a perplexity of 7.4. A loss of 3.0 is 20.1. Nothing new is measured; the scale is changed.
Written this way it is the inverse geometric mean of the probabilities assigned to the correct tokens.
The usual reading is the branching factor: a perplexity of 20 means the model is as uncertain as if it were choosing uniformly among 20 equally likely tokens at every step.
A perplexity of 1 is a model that predicts every token with certainty. The floor is 1, not 0.
The ceiling is the vocabulary size: a model that assigns uniform probability over tokens has perplexity exactly .
This is the part that gets misused. Perplexity is per token, and the tokenizer decides what a token is.
A model with a 32,000-token vocabulary splits a word into more pieces than one with 128,000, so it makes more predictions over the same text and each is individually easier.
Comparing perplexities across different tokenizers is comparing two different quantities, and the direction of the error is not even predictable.
Bits per byte is reported instead when models must be compared, because a byte is a byte in every scheme.
Perplexity rewards a model for being unsurprised by text. It says nothing about whether the model is useful, truthful, or safe.
An instruction-tuned model usually has worse perplexity than the base model it came from, and is far more useful.
That is not a paradox: fine-tuning moved it away from the raw text distribution on purpose.
One question on this card. Every option explains itself once you pick, right or wrong.
20 concepts in this module, then a graded test with every answer explained.