LLM-as-judge uses a model to score another model's output.
It is far cheaper and faster than human raters.
It needs a specific rubric, not a vague instruction to rate quality.
Known biases include favouring longer answers and its own style.
Always validate the judge against human ratings before trusting it.
LLM-as-judge uses a model to score another model's output.
It is far cheaper and faster than human raters and can run on every change.
That makes continuous evaluation practical rather than aspirational.
It needs a specific rubric, exactly as a human rater does.
Asking a model to rate quality out of ten gives you a number with no meaning.
Give it criteria, a scale with described levels, and the context it needs to judge.
Ask for reasoning before the score, since the reasoning improves the score.
The known biases are well documented and consistent.
It prefers longer answers, and it prefers answers in its own style.
It shows position bias when comparing two outputs, so the order must be randomised.
It is generous to fluent nonsense, since fluency is what it is best at recognising.
It cannot reliably judge factual accuracy without being given the facts.
Always validate against human ratings on a sample before trusting it.
Agreement of 80 per cent with humans is usable, and 50 per cent is a coin toss with extra steps.
Use a different model from the one being evaluated, since a model marking its own work is not a test.
Human evaluation is slow and expensive, and string metrics cannot tell a correct paraphrase from a wrong one.
A judge model sits between them: it costs cents, runs in seconds, and understands the text well enough to apply a written rubric.
Published comparisons put a strong judge at roughly the level of agreement two human raters reach with each other, which is the bar that matters.
Raw agreement is misleading when one label dominates. On a set that is 90 percent correct, a judge answering correct every time agrees 90 percent of the time and has learned nothing.
Cohen's kappa subtracts the chance level, which is why it is the number to report for a judge.
The judge must be validated on a few hundred human-labelled examples before it is used on thousands, and revalidated whenever the prompt or the model version changes.
The first option shown wins more often.
The fix is to run every pair twice with the order swapped, and to count a disagreement between the two runs as a tie.
Longer answers score higher.
The rubric has to state that length is not quality, and even then the effect survives, so length should be reported alongside the score.
A model prefers its own outputs.
Judging a model with itself or with its own family inflates the result, which is why an independent judge is the standard precaution.
On a 1 to 10 scale, scores cluster at 7 and 8.
A fine-grained scale produces false precision. Three or five levels with written definitions is more reliable and easier to act on.
A judge is not ground truth.
Anything with a checkable answer should be checked: run the code, compare the number, query the database. A judge is for the cases where no such check exists.
Optimising a model against a judge makes the judge's blind spots into the model's behaviour, which is Goodhart's law with a fast feedback loop.
That is why a held-out human evaluation stays in the process, even when the judge is doing almost all the volume.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.