You normally split your data into training, validation and test sets.
The training set is what the model actually learns from.
The validation set is what you use to compare models and tune hyperparameters.
The test set is touched once at the very end to estimate real-world performance.
Keeping the three roles separate is what stops you from fooling yourself.
Three splits exist because there are three different questions.
Training data answers: what pattern is in this data?
Validation data answers: which of my choices is the best one?
Test data answers: how will this do on something genuinely new?
The moment you make a decision using a set, it stops being neutral.
Look at the test set twice and it has quietly become a validation set.
A common split is 60, 20 and 20, but it depends on how much data you have.
With very little data, cross-validation beats a fixed validation split.
Split before you clean, scale or fill anything in.
Doing it the other way round is the most common cause of leakage.
If your data has an order in time, split by time and not at random.
Otherwise you train on the future and test on the past.
If rows come in groups, keep whole groups on one side of the split.
The same person appearing on both sides will flatter your score.
The test set gets opened once, at the end, and then it is spent.
One question on this card. Every option explains itself once you pick, right or wrong.
55 concepts in this module, then a graded test with every answer explained.