The encoder turns an input sequence into rich contextual representations.
Each layer runs self-attention followed by a feed-forward network.
Its attention is bidirectional, so every token sees the whole sequence.
It produces understanding rather than generated text.
BERT is the classic encoder-only model, used for classification and embeddings.
An encoder turns an input sequence into a set of context-rich vectors.
It produces understanding rather than text, which is the whole distinction.
Each layer runs self-attention and then a feed-forward network.
Its attention is bidirectional, so every token sees the tokens before and after it.
That is the defining property, and it is only possible because nothing is being generated.
The word bank can therefore be resolved using the word river four positions later.
The output is one vector per input token, each shaped by the whole sequence.
Those vectors feed a classifier, a tagger, or are pooled into a single embedding.
Encoder-only models are what you want for classification, search and extraction.
BERT is the canonical example and its descendants still power a great deal of production search.
They are small and cheap by modern standards, often a few hundred million parameters.
For a fixed classification task an encoder usually beats a much larger generative model.
It also costs a fraction as much to run and returns an answer in milliseconds.
An encoder cannot write text, since it was never trained to continue a sequence.
Reaching for a chat model to classify support tickets is a common and expensive mistake.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.