Dataset preparation is where most fine-tuning success or failure is decided.
Examples must match the exact format and style you want at inference.
Quality and consistency beat raw volume every time.
A few hundred clean examples outperform thousands of noisy ones.
Always hold out a validation split before you start training.
Dataset preparation decides whether a fine-tune succeeds, more than any hyperparameter.
The model learns the pattern in your data, including the parts you did not intend.
Format the examples exactly as requests will arrive at inference.
A mismatch between training format and serving format silently ruins the result.
Consistency matters more than volume.
Five hundred examples in one clear style beat five thousand in three inconsistent ones.
Inconsistent examples teach the model to pick a style at random.
Include the hard cases and the edge cases, since easy ones teach little.
Include examples of refusing or asking for clarification if you want that behaviour.
A model trained only on confident answers will always answer confidently.
Remove duplicates, which inflate the apparent size and encourage memorisation.
Hold out a validation split before training, chosen randomly and never touched.
Read a hundred examples by hand, because problems are visible and rarely detected by script.
Synthetic data from a stronger model is common and needs filtering rather than trust.
Most of the effort in a fine-tuning project should land here.
One question on this card. Every option explains itself once you pick, right or wrong.
15 concepts in this module, then a graded test with every answer explained.