Model evaluation measures how well a model performs its task.
For classical ML you compute metrics against held-out labelled data.
The metric must match what you actually care about in the real problem.
One aggregate number always hides variation across subgroups.
Always break results down by segment before trusting an overall score.
Evaluation is how you find out whether a model is any good before users do.
The method is always the same: predict on data the model has not seen and compare with the truth.
The difficulty is choosing what to compare and how to summarise it.
The metric has to match the decision the model actually supports.
Accuracy is the default and is misleading whenever classes are imbalanced.
Optimising a metric that does not reflect the real cost produces a model that is wrong usefully.
Ask what a false positive costs and what a false negative costs, since they are rarely equal.
One aggregate number always hides variation underneath.
A model at 92 per cent overall may be at 60 per cent for one important group.
Break results down by segment before trusting anything.
Compare against a baseline, which might be the current system or simply predicting the majority.
A model that cannot beat the obvious baseline is not a model worth deploying.
Evaluate on data resembling production, including the messy inputs.
Held-out data drawn from the same clean sample flatters the model.
For anything time-based, split by time rather than randomly, or you are predicting the past.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.