Continued pretraining keeps training a finished model on raw text from a new domain.
It uses the same next-token objective as the original run, not instruction and response pairs.
This is how a model learns unfamiliar vocabulary, notation and house style at scale.
It needs billions of tokens to matter, which puts it far above fine-tuning in cost.
The model comes out fluent in the domain and unable to follow instructions until retuned.
Fine-tuning teaches a model how to behave, and it is poor at teaching it new subject matter.
A few thousand instruction pairs cannot install a vocabulary the model has never encountered.
Continued pretraining, also called domain-adaptive pretraining, is the tool for that job.
It resumes the original training procedure on a large corpus from the target domain.
The objective is unchanged: predict the next token over raw, unlabelled text.
Legal filings, clinical notes, chip design documents and low-resource languages are the usual motivations.
The scale required is the main barrier, since meaningful gains generally need billions of tokens.
That is a serious training run, not an afternoon on a single GPU.
Catastrophic forgetting is the standard hazard: train only on medicine and general ability degrades.
The usual defence is to mix a proportion of general text back into the corpus.
A lower learning rate than the original run also helps preserve what the model already knows.
The result is a base model again, fluent in the domain and unable to follow instructions.
So continued pretraining is almost always followed by instruction tuning and preference training.
That full sequence is what produces domain-specific models rather than domain-specific prompts.
For most teams the honest answer is that retrieval is cheaper and gets most of the benefit.
One question on this card. Every option explains itself once you pick, right or wrong.
15 concepts in this module, then a graded test with every answer explained.