Without a fixed set of test cases, every prompt change is judged by whether it feels better.
Start with real inputs from real usage, not invented examples that flatter the system.
Include the failures you already know about, since those are the cases a change must not break.
Fifty well-chosen cases beat five thousand generated ones you have never read.
Keep it in version control and treat adding a case to it as the standard response to a bug.
Most teams building on a language model begin with no way to tell whether a change helped.
A prompt is edited, three examples are tried by hand, and the change ships because it felt better.
An eval set replaces that with a fixed collection of inputs and a way to score the outputs.
The first question is where the inputs come from, and the answer is production.
Real users ask things nobody on the team would think to write down.
Logged requests, with anything sensitive removed, are the highest-quality source available.
The second source is your bug tracker, since every reported failure is a test case waiting to be added.
Building the set from known failures is what stops the same regression shipping twice.
Coverage matters more than volume, and this is where most effort is wasted.
Fifty cases spanning the real range of inputs are worth more than five thousand near-duplicates.
Deliberately include the edge cases: empty input, wrong language, hostile input, ambiguous requests.
Each case needs a way to be scored, and not every case needs the same one.
Exact match works where there is a right answer, and a rubric with an LLM judge covers open output.
The set belongs in version control next to the code, so a change to it is reviewed like any other.
It should also grow: a bug that reaches production and is not added as a case will return.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.