Offline evaluation runs a fixed set of cases before release and tells you whether to ship.
Online evaluation measures the system in production against what users actually do.
Offline is fast, repeatable and cheap, and it only covers the inputs you thought of.
Online covers everything real but is slow, noisy and expensive to run an experiment on.
You need both: offline to catch regressions, online because the offline set is always incomplete.
Evaluation happens in two places and the two answer different questions.
Offline evaluation runs a stored set of inputs through the system and scores the outputs.
It happens before release, in continuous integration, and it gates the change.
Its strengths are speed, repeatability and cost: the same cases, the same scores, in minutes.
Its weakness is fixed by construction, because it can only contain what someone thought to include.
Real users are more inventive than any test author, and the distribution shifts over time.
Online evaluation measures the system as deployed, against the traffic it actually receives.
Some signals are explicit, such as thumbs up and down or a correction to the model's answer.
Most are implicit and more informative: did the user retry, edit the output, or abandon the task.
A rising retry rate is a quality signal long before anyone files a complaint.
Comparing two versions online means an A/B test, with the sample size that implies.
That is slow, and detecting a small improvement can take weeks of traffic.
It is also the only way to know whether a change improved anything a user cares about.
The healthy arrangement is a loop: online failures become offline cases, offline gates the next release.
Teams that run only offline ship confidently into problems they never measured.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.