Prompt evaluation means testing prompts systematically instead of by vibe.
You build a fixed set of inputs with known good outputs.
Every prompt change is run against that whole set.
Scoring can be exact match, a rubric, or another model as judge.
Without it you cannot tell whether a prompt edit helped or quietly broke something.
Prompt evaluation means testing prompt changes instead of judging them by feel.
Without it every edit is a guess, and yesterday's fix is today's regression.
Start with a fixed set of inputs that represent real usage.
Twenty well-chosen cases are worth more than a thousand random ones.
Include the failures that prompted the change, or you will not know it worked.
Record the output you would accept for each case.
Exact match works for classification and extraction, where there is one right answer.
For open-ended output use a rubric with a few clear criteria.
LLM-as-judge automates that scoring and needs its own checking against human ratings.
Run the whole set on every prompt change, not just the case you were fixing.
Track the score over time so a slow decline is visible.
Test with the temperature you actually use, since low temperature hides variance.
Run each case a few times when temperature is above zero.
Version prompts alongside their scores, so a rollback is a decision rather than a scramble.
This is the difference between prompt engineering and prompt guessing.
One question on this card. Every option explains itself once you pick, right or wrong.
17 concepts in this module, then a graded test with every answer explained.