Most failed fine-tunes are data problems rather than training problems.
Too few examples, or inconsistent ones, simply teach the model contradictions.
Fine-tuning teaches behaviour and style far better than it adds new facts.
If the model needs knowledge it never saw, retrieval is the right tool instead.
A prompt change is cheaper to try, so rule that out before touching any weights.
A fine-tune that disappoints is usually a data problem dressed as a training problem.
The first question is how many examples there are and how consistent they are.
A few hundred clean and consistent examples beat ten thousand inconsistent ones.
If two examples answer similar inputs in contradictory ways, the model learns the contradiction.
Labelling drift across a dataset built over months is a common and nearly invisible cause.
The second question is whether the task is really about behaviour or about knowledge.
Fine-tuning is very good at teaching format, tone, structure and how to respond.
It is poor at installing facts that the base model never saw.
Facts learned this way are held weakly and are hard to update when they change.
Retrieval is the right tool for knowledge, because the fact stays outside the weights.
Catastrophic forgetting is the third failure, where the new skill pushes out the old ones.
A model tuned hard on one narrow format often becomes worse at everything else.
Mixing some general data back into the training set reduces that effect.
Evaluate on held-out data and on general benchmarks, or the regression stays invisible.
Before any of this, try the prompt, because it is faster, cheaper and reversible.
One question on this card. Every option explains itself once you pick, right or wrong.
15 concepts in this module, then a graded test with every answer explained.