Synthetic Data Generation

Fine-tuning needs examples, and for most specific tasks nobody has ever written them down.

Synthetic data generation uses a capable model to produce that dataset instead of paying people to.

The standard recipe seeds a few handwritten examples and asks the model for many variations.

Self-Instruct popularised it, and most public instruction datasets were built this way.

Distillation is the same technique aimed at a specific goal: a small model learns from a large one's outputs.

It works because judging and imitating good answers is easier than producing them from nothing.

The dominant risk is that flaws in the generated data become flaws in the trained model.

A fabricated fact in an example is not noise to the learner; it is a target to be reproduced.

So verification matters more than volume, and anything checkable should be checked.

Code can be run, maths can be evaluated, and claims can be validated against a source document.

Diversity is the failure that hides best, because the text always looks varied on inspection.

In reality a model asked repeatedly for examples returns a handful of templates with the nouns changed.

Deliberately varying the prompt, the seed topics and the personas is what restores real spread.

Deduplication by embedding similarity catches the near-copies that exact matching misses.

Finally, read the terms of the model you generate from, since many prohibit training a competing model on its output.