A language model assigns a probability to text, which amounts to predicting what comes next.
Train it on enough text and the prediction task alone forces it to learn grammar, facts and style.
No labels are needed, since the next word in the corpus is its own answer.
That is why the internet could be used as training data, and why the models grew so large.
Everything a chat model does is still this one operation, run over and over.
A language model answers one question: how likely is this sequence of words?
By the chain rule of probability, that question decomposes into predicting each word given the ones before.
So a language model is, mechanically, a next-word predictor and nothing more.
The reason this matters is that the training target comes free with the data.
Every position in every document supplies both an input and the correct answer.
That makes it self-supervised learning, and it is why the entire web became usable as training data.
No annotation budget bounds the corpus, so the only limits are compute and available text.
The surprise of the last decade is how much is learned in service of that narrow objective.
Predicting the next word well requires syntax, since the answer must be grammatical.
It requires facts, because completing the capital of France is has exactly one good answer.
It requires tracking who is being discussed, and it requires some sense of style and register.
None of that was asked for; all of it is instrumental to lowering the prediction error.
The measure of a language model is perplexity, the effective number of words it was choosing between.
The earliest models counted n-grams, then recurrent networks arrived, then transformers.
Nothing about the objective changed across those generations; only the capacity to fit it did.
A language model assigns a probability to a sequence, and it does so by factorising that probability into a product of conditionals.
This is not an approximation. It is an identity, true of any joint distribution over any ordered variables.
The modelling choice is not the factorisation, it is deciding to learn each conditional with a neural network and to condition on everything before.
The identity holds for any ordering, so left to right is a choice rather than a necessity.
It is chosen because it matches generation: to produce text you need the next token given the past, which is exactly what this factorisation provides.
A model factorised in some other order would assign the same probability to a full sequence and be useless for writing one.
Maximising the likelihood of the training text is minimising this, which is the cross-entropy between the true next-token distribution and the model's.
Every token in the sequence is a training example, so a document of 2000 tokens gives 2000 supervised predictions from one unlabelled document.
That is the property that made the whole field possible: the supervision is free, because the next word is already there.
During training the model always conditions on the real prefix, even when its own prediction for the previous token was wrong.
During generation it conditions on what it produced. One mistake enters the context and every later prediction is made from a prefix the model never saw in training.
That mismatch is called exposure bias, and it is part of why long generations drift while short ones are fine.
Step two is why transformers train so much faster than recurrent models: all predictions are made in one parallel pass, not sequential ones.
One question on this card. Every option explains itself once you pick, right or wrong.
20 concepts in this module, then a graded test with every answer explained.