Evaluate the fine-tuned model on held-out data it never saw during training.
Compare directly against the base model with a good prompt as your baseline.
Check that general ability has not degraded, which is catastrophic forgetting.
Task-specific metrics matter more than the training loss curve.
If it does not beat a well-prompted base model, the fine-tune was not worth it.
A fine-tune is not finished until it has been measured against the alternative.
The baseline is the base model with a good prompt, not the base model with a lazy one.
Many fine-tunes lose to a well-written prompt, which is worth discovering before deploying.
Evaluate on held-out data the model never saw in training.
Training loss falling means it memorised, and says nothing about generalisation.
Use metrics that match the task rather than the loss curve.
Exact match for extraction, a rubric for style, and accuracy for classification.
Check general ability separately, since narrow training degrades it.
Run a broad set of ordinary questions and compare with the base model.
A model that aces your task and has lost the ability to hold a conversation may still be wrong for you.
Test the format under pressure, with unusual and adversarial inputs.
Compare cost and latency too, since a smaller fine-tuned model may win on both.
Have a person read fifty outputs, because metrics miss things that are obvious to read.
Keep the evaluation set fixed so successive runs are comparable.
If it does not clearly beat a prompted base model, do not ship it.
One question on this card. Every option explains itself once you pick, right or wrong.
15 concepts in this module, then a graded test with every answer explained.