Self-supervised learning invents its own labels out of unlabelled data.
You hide part of the input and ask the model to predict the missing piece.
Predicting the next word in a sentence is exactly this, and it is how language models are trained.
It unlocks enormous datasets, because no human has to label anything.
The result is a general model that is then fine-tuned on a small labelled set.
Labelled data is expensive and unlabelled data is nearly free.
Self-supervised learning bridges that gap by inventing labels from the data itself.
The recipe is to hide part of the input and ask the model to reconstruct it.
The answer is already in the data, so no person has to write anything down.
Predicting the next word in a sentence is exactly this, and it is how language models are trained.
Masking random words and filling them back in is the same idea, and it is how BERT was trained.
In vision the equivalent is hiding patches of an image and reconstructing them.
Contrastive methods instead show two altered views of one image and ask the model to match them.
To do any of this well the model has to learn how the data is structured.
That structure is what transfers to the task you actually care about.
The result is a general-purpose model, not a solution to any particular problem.
A small labelled set then fine-tunes it, often a few thousand examples instead of millions.
This is why every foundation model exists, since it is what makes internet-scale training possible.
It is unsupervised in that nobody labelled anything, and supervised in that there is a right answer.
That is where the awkward name comes from.
One question on this card. Every option explains itself once you pick, right or wrong.
55 concepts in this module, then a graded test with every answer explained.