A prompt is code, and editing it can break a case that has worked for months.
Regression testing runs the eval set on every change and compares against the last known result.
Outputs are not deterministic, so the comparison is over scores across the set, not exact strings.
Report which individual cases flipped, since an unchanged average can hide a real regression.
Model upgrades need the same treatment, because the provider changing something is a change to you.
A prompt that has been tuned for months is a load-bearing part of a system.
It is also a single string that anyone can edit, with no compiler to object.
Regression testing treats it accordingly: every change runs the eval set and compares with the previous run.
The immediate complication is that model output is not deterministic.
Even at temperature zero, wording varies with batching and provider-side changes.
So the comparison cannot be over strings and has to be over scores.
The right unit is the per-case result, not the aggregate.
An average that is unchanged can hide five cases improving and five getting worse.
Reporting which specific cases flipped from pass to fail is what makes the test actionable.
Because scores are noisy, a small difference is not evidence of anything.
Running each case several times and comparing distributions is the honest version.
Cost forces a compromise, so most teams run a small critical set often and the full set before release.
The other trigger is a model change, and it is the one people forget.
A provider updating a model silently changes your system, so the same suite must run against a new version before adopting it.
Pinning to a dated model version and upgrading deliberately is what makes that possible at all.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.