Human evaluation has people rate outputs directly.
It remains the gold standard for quality, tone and helpfulness.
Clear rubrics and multiple raters are what make it reliable.
Inter-rater agreement tells you whether your rubric is well defined.
It is slow and expensive, so it is best used to validate automated metrics.
Human evaluation means people read outputs and rate them.
It remains the gold standard for quality, tone and helpfulness.
Automated metrics approximate human judgement, so human judgement is the reference.
It is only reliable if the rubric is clear.
Rate this from one to five produces noise, since everyone means something different.
Define what each level looks like, with an example of each.
Use several raters per item and measure how often they agree.
Low agreement means the rubric is unclear, not that the raters are careless.
Fixing the rubric is the fix, and Cohen's kappa is the usual measure.
Comparing two outputs side by side is easier and more reliable than scoring one alone.
People are much better at picking a winner than at assigning an absolute number.
Randomise the order, since position bias is real and consistent.
It is slow and expensive, so use it where it counts.
The best use is validating an automated judge on a sample, then trusting the judge in bulk.
Re-validate periodically, because a judge that drifts is worse than no judge.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.