People are bad at scoring one answer out of ten and good at saying which of two is better.
Pairwise comparison exploits that, showing two outputs and asking only which one wins.
Many such comparisons are turned into a single rating using a system like Elo.
Public model arenas rank models this way, from votes cast by anonymous users.
Position bias is real, so the order of the two answers must be randomised or swapped.
Asking a judge to rate an answer from one to ten produces surprisingly unreliable numbers.
The internal scale drifts between judges, between sessions, and even down a single list.
Asking which of these two is better is a much easier question and gets far more consistent answers.
This is true of human judges and of models used as judges.
Pairwise comparison builds evaluation on that easier question and nothing else.
Two outputs for the same input are shown, and the judge picks a winner or declares a tie.
The result of many such comparisons is a ranking rather than a set of scores.
Elo, borrowed from chess, converts pairwise results into a single number per competitor.
Winning against a strong opponent moves the rating more than winning against a weak one.
The Bradley-Terry model is the statistical version of the same idea and is what most arenas actually fit.
Public leaderboards built this way collect votes from users comparing two anonymous models.
The biases are well documented and must be controlled for.
Position bias means judges favour the first option, so order is randomised on every comparison.
Verbosity bias means longer answers win more often than their content justifies.
And a rating is only meaningful within its pool, so an Elo number cannot be compared across arenas.
Absolute ratings drift. One rater's 7 is another's 5, and the same rater's scale moves across a long session.
A comparison asks only for a win, a loss or a tie.
A comparison needs no scale, so it is far more consistent between raters and over time, which is why model evaluation moved to it.
The cost is that models need pairings to compare directly, and the ratings below are how that is avoided.
A 400-point gap means the stronger side is expected to win about 10 times out of 11, which is what fixes the scale.
The update is the surprise: beating a much stronger opponent moves the rating a lot, and beating a much weaker one barely moves it at all.
An Elo rating depends on the order of the matches.
Elo was designed for players whose strength changes over time, so it is sequential and path dependent.
A model's strength does not change between comparisons, so processing the same battles in a different order should not give a different rating, and under Elo it does.
The Bradley-Terry model fits all the comparisons at once by maximum likelihood, which is order independent and gives confidence intervals.
Modern leaderboards report Bradley-Terry estimates and keep the Elo scale for familiarity, which is why the numbers look like chess ratings but are computed differently.
Raters and model judges both favour the first option shown, so the order must be randomised and the effect measured.
Longer answers win more often at equal quality, for humans and for model judges alike, which is why leaderboards now publish length-controlled ratings.
Ties need a policy decided in advance. Forcing a choice adds noise, and allowing ties freely lets raters avoid the judgement the evaluation exists to collect.
The interval on a model with a few hundred battles is wide enough that adjacent leaderboard positions are usually not separable, and the published intervals say so.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.