Sequence-to-Sequence in NLP, explained in 5 lines

Module 04 · NLP · 20 concepts

In depth / 15 lines

Sequence-to-Sequence, in more detail

Sequence to sequence handles tasks where input and output are both sequences of different lengths.

Translation is the obvious case, since a sentence rarely has the same length in two languages.

Summarisation, question answering and speech transcription have the same shape.

The design splits the problem in two, an encoder and a decoder.

The encoder reads the whole input and compresses it into a context vector.

The decoder then generates the output one token at a time from that vector.

At each step it also sees what it has already produced, so the output stays coherent.

Generation continues until the model emits an end-of-sequence token.

Training feeds the true previous token rather than the model's own guess, which is called teacher forcing.

It trains much faster, at the cost of a model that has never seen its own mistakes.

The flaw in the original design is that one fixed vector must hold the entire input.

A forty-word sentence is squeezed through the same few hundred numbers as a four-word one.

Quality fell off sharply as sentences grew longer.

Attention was invented precisely to fix this, letting the decoder look back at every input position.

That addition mattered so much it became the whole architecture in the transformer.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Sequence-to-Sequence

Two networks and one vector between them

\mathbf{c} = \mathbf{h}_{T}^{\mathrm{enc}}

The encoder reads the whole input and its final state is handed to the decoder as a summary, traditionally called the context vector.

p(\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T'} p\big(y_t \mid y_{<t}, \; \mathbf{c}\big)

The decoder is a language model conditioned on that vector, generating one token at a time until it emits an end symbol.

This is what made translation, summarisation and dialogue the same problem, since none of them require the output to be the same length as the input.

The bottleneck, and the fix that became everything

\mathbf{c} \in \mathbb{R}^{d}

A fifty-word sentence and a five-word sentence are compressed into the same few hundred numbers, and translation quality fell off sharply with length for exactly that reason.

\mathbf{c}_t = \sum_{i=1}^{T} \alpha_{ti} \mathbf{h}_i^{\mathrm{enc}}

\alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{j} \exp(e_{tj})}

Attention replaced the single vector with a different weighted average at every output step, so the decoder can look back at the input word it currently needs.

Nothing is compressed away. The encoder's states are all kept, and the weights $\alpha$ say which ones matter now.

That change was introduced to patch the bottleneck in a recurrent model, and it turned out to be the only part worth keeping.

Teacher forcing, and the gap it leaves

\text{train} : p\big(y_t \mid y^{*}_{<t}\big), \qquad \text{infer} : p\big(y_t \mid \hat{y}_{<t}\big)

During training the decoder is fed the true previous token, which makes the steps independent and lets the whole sequence train in parallel.

At inference it is fed its own previous output, so one early mistake shifts the model into a state it never saw in training.

That mismatch is called exposure bias, and it is why generated text can start well and degrade, rather than being uniformly slightly wrong.

One translation, in order

Encode the source sequence, keeping every hidden state.
Start the decoder with a beginning-of-sequence token.
Compute attention weights over the encoder states from the current decoder state.
Predict a distribution over the vocabulary and pick a token.
Feed that token back in and repeat until the end symbol is produced.

Beam search, because greedy is not the objective

\hat{\mathbf{y}} = \arg\max_{\mathbf{y}} \; \log p(\mathbf{y} \mid \mathbf{x})

Taking the most likely token at each step does not give the most likely sequence, because a strong first token can lead into a weak continuation.

Beam search carries several candidates forward and expands each one, at $k$ times the cost and with a reliable gain in quality.

\frac{1}{|\mathbf{y}|^{\alpha}} \log p(\mathbf{y} \mid \mathbf{x})

Scores are normalised by length, because summing log probabilities penalises every extra token and an unnormalised beam search prefers short, unhelpful outputs.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

In a classic seq2seq model, what was the encoder-decoder bottleneck?

NLP

20 concepts in this module, then a graded test with every answer explained.

Study this module →