Pretraining is the first and by far the most expensive training stage.
The model reads an enormous corpus of text from the internet and books.
Its only task is predicting the next token, over and over.
Grammar, facts and reasoning patterns all emerge from that single objective.
It costs millions of dollars and takes weeks on very large GPU clusters.
Pretraining is the stage that creates the base model, and it is by far the most expensive.
The model is shown an enormous corpus of text and asked to predict the next token.
That is the only objective, repeated over trillions of tokens.
The corpus is a filtered crawl of the web plus books, code and reference material.
Filtering out junk and duplicates matters as much as adding more data.
Everything the model appears to know comes from this stage.
Grammar, facts, translation and reasoning patterns all emerge from next-token prediction alone.
Nobody labels anything, so the supervision is the text itself.
That is why it can use every document ever written rather than a hand-labelled set.
A frontier run costs millions of dollars and takes weeks on thousands of GPUs.
The result is a base model that continues text rather than answering questions.
Ask it a question and it may well reply with more questions, since that is what documents do.
Instruction tuning and preference training afterwards turn it into something conversational.
The knowledge cutoff is set here, since the model knows nothing published after the crawl.
Everything later, including fine-tuning, is small change against this stage.
Predict the next token, average over every position, and that is the entire training signal for a base model.
No labels are needed, which is the point. The supervision is the text itself, so the training set is as large as the readable internet.
The loss is reported in nats per token, and a well-trained model on English text lands somewhere around 2, meaning a perplexity near 7.
Two operations per parameter for the forward pass and four for the backward, which is where the 6 comes from.
At about useful operations per second per accelerator, that is roughly a thousand accelerator-days, before any inefficiency.
For a fixed compute budget, the Chinchilla result says parameters and tokens should grow together rather than parameters alone.
That gives roughly 20 tokens per parameter as the compute-optimal ratio, which showed that the models of 2020 were several times too large for the data they saw.
Models trained today deliberately overshoot it, often past 100 tokens per parameter, because the budget that matters is inference over the model's life rather than the one-off training cost.
A smaller model trained longer is cheaper to serve forever, and that economics now outranks the compute-optimal point.
Data quality beats data quantity.
Deduplication, filtering and mixture weights move the final loss more than architecture changes of the same effort, and most published gains at fixed scale come from the corpus.
Contamination is the failure that invalidates everything downstream. Benchmark text inside the training corpus produces scores that measure recall rather than ability.
When the loss spikes, roll back and skip the batch.
Long runs diverge. The standard response is not a cleverer optimiser but a checkpoint from before the spike and a different data order, which is why checkpoints are frequent.
The output of all this is a base model, which continues text and does not follow instructions. Turning it into something that answers questions is a separate stage entirely.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.