BERT is an encoder-only transformer built for understanding rather than generating.
It reads the whole sentence at once, seeing the words on both sides of every position.
It was trained by hiding random words and asking the model to fill them back in.
That makes it strong at classification, search, and pulling answers out of text.
It cannot write text on its own, because it was never trained to continue a sequence.
BERT was the model that made pretraining standard in language work.
It is encoder-only, so it reads text and produces representations rather than continuations.
Bidirectional is the key word in the name, since every position sees both directions.
That is impossible for a model trained to predict the next word, which must not see ahead.
So BERT was trained on a different task: masked language modelling.
Roughly 15 per cent of tokens are hidden and the model fills them back in.
Filling a gap benefits from both sides, which is exactly what forces bidirectional understanding.
It was also trained to judge whether two sentences follow each other, which later work found unhelpful.
Pretraining once and fine-tuning cheaply on many tasks was the change that mattered.
It set records across a broad set of benchmarks in 2018 and reset expectations.
It excels at classification, named entity recognition, and pulling an answer span out of a passage.
Its sentence embeddings underpin an enormous amount of semantic search.
It cannot generate text on its own, which is not a defect but a design.
RoBERTa, DeBERTa and DistilBERT refined it and are usually better choices today.
For a fixed text task with labelled data, this family remains the efficient answer.
A left-to-right model predicts the next token, and it cannot see the future because that would give away the answer.
BERT wants both directions at once, so that the representation of a word is built from everything around it.
That is impossible under next-token prediction, since a bidirectional stack trivially sees the token it is asked to predict.
The fix is to remove the token from the input and ask for it back.
Of those selected positions, 80 percent are replaced by a mask token, 10 percent by a random token, and 10 percent are left alone.
The last two rules exist because the mask token never appears at fine-tuning time. Training only on masked inputs would make the model expect a symbol that production never sends.
Leaving some untouched forces the model to build a useful representation at every position rather than only where it sees the mask.
The masking rate is a direct trade: more targets per sequence against less context to predict them from.
Three special tokens frame every input: [CLS], [SEP] and [MASK].
The class token at the front has no word of its own, so its final state is free to serve as a summary of the whole sequence for classification.
Classification, entity recognition, retrieval and sentence similarity all reduce to reading the encoder's states, and a task head is often a single linear layer.
Pretraining once and fine-tuning many times was the contribution that changed the field's economics, a year before generation did it again.
The objective fills in blanks in a fixed-length sequence. It has no way to extend one, so BERT cannot write a sentence.
For scoring a fixed input, an encoder does one pass with full context in both directions, and a decoder-only model of the same size does more work for a weaker representation.
Embedding models, rerankers and classifiers in production are still overwhelmingly encoders, because they are smaller, faster and measurably better at exactly those jobs.
The next-sentence prediction objective from the original paper was later shown to add nothing, and the models that dropped it in favour of longer training scored higher on everything.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.