LLM evaluation is hard because many different outputs can be correct.
Exact match works only for closed, single-answer tasks.
Rubric-based scoring judges output against explicit written criteria.
A fixed evaluation set lets you compare versions fairly.
Building that set early is the highest-leverage thing you can do.
Evaluating a language model is harder than evaluating a classifier.
There is usually no single correct output.
Twenty summaries can all be good and share almost no words.
So exact match only works for closed tasks with one right answer.
Extraction, classification and arithmetic can be scored that way, and little else.
For open-ended output, a written rubric is the practical approach.
State the criteria explicitly: is it accurate, does it answer the question, is it the right length.
Vague criteria produce vague scores, whether the rater is a person or a model.
The single most valuable thing is a fixed evaluation set built early.
Twenty to a hundred real cases with the outputs you would accept.
Without it, every change is judged by whether the last example looked better.
Include the failures you have already seen, so a regression is caught.
Run the set on every prompt or model change, not only when something feels wrong.
Track the score over time so slow degradation is visible.
Automated scoring should be validated against human judgement before it is trusted.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.