Attention lets a model decide which parts of the input matter for each output.
Instead of one fixed summary it computes a weighted blend of all positions.
Positions that are relevant get a high weight, the rest get a low one.
This removed the fixed-vector bottleneck of old seq2seq models.
It is the single idea the entire transformer architecture is built around.
Attention lets a model decide, for each thing it produces, which inputs matter.
The old sequence models had to squeeze the entire input into one fixed vector.
That worked for short sentences and fell apart on long ones.
The information was there in principle and unreachable in practice.
Attention replaces the single summary with a fresh blend for every output step.
At each step the model scores every input position for relevance right now.
Those scores are turned into weights that sum to one.
The output is the weighted mixture of all positions, dominated by the relevant ones.
Nothing is discarded, since irrelevant positions simply receive a near-zero weight.
The weights are computed, not stored, so they change with every step and every input.
It was introduced in 2014 as an addition to translation models and improved them immediately.
The 2017 transformer paper removed everything else and kept only attention.
The title, attention is all you need, was a fair description of the result.
Attention weights are often shown as heatmaps and read as an explanation.
They show where the model looked, which is suggestive but not a full account of why.
5 questions on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.