RNNs in NLP, explained in 5 lines

Module 04 · NLP · 20 concepts

In depth / 15 lines

RNNs, in more detail

A recurrent network reads a sequence one element at a time, in order.

After each step it updates a hidden state that it carries to the next one.

That state is its memory of everything seen so far, squeezed into a fixed-size vector.

The same weights are used at every step, so the network is small and length is unlimited.

This was the natural design for text, where meaning depends on what came before.

Training uses backpropagation through time, unrolling the loop into one deep network.

A sentence of 50 words becomes a 50-layer network sharing one set of weights.

Gradients then pass through 50 multiplications on the way back.

They vanish, and the early words receive essentially no learning signal.

So a plain RNN reliably remembers about five to ten steps and forgets the rest.

The pronoun at the end of a long sentence cannot be linked to the name at the start.

LSTMs and GRUs were built to hold information across longer gaps.

The deeper problem is that the steps must run in order, so nothing parallelises.

A GPU sits mostly idle waiting for step by step work.

Transformers replaced them by looking at all positions at once.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath RNNs

One state, carried forward

\mathbf{h}_t = \tanh\big( W_{hh} \mathbf{h}_{t-1} + W_{xh} \mathbf{x}_t + \mathbf{b} \big)

$\mathbf{h}_t$ : the hidden state after reading $t$ inputs.
$\mathbf{x}_t$ : the input at this step.
$W_{hh}$ : the recurrent weights, applied at every step.

The state is a summary of everything read so far, and it is the only thing carried between steps.

\mathbf{y}_t = W_{hy} \mathbf{h}_t + \mathbf{c}

The same weights are used at every timestep, which is weight sharing across time in the way a convolution shares across space.

That sharing is what lets one model read a sequence of any length, and it is also the source of everything that goes wrong.

Unrolling, and backpropagation through time

\frac{\partial \mathcal{L}}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial \mathcal{L}_t}{\partial W_{hh}}

Training treats the $T$ steps as a $T$ -layer network that happens to share weights, so the gradient at each step is summed into one update.

\frac{\partial \mathbf{h}_T}{\partial \mathbf{h}_t} = \prod_{k=t+1}^{T} \mathrm{diag}\big(\tanh'(\cdot)\big) W_{hh}^{\top}

The gradient reaching step $t$ from step $T$ is a product of $T - t$ copies of the same matrix.

\sigma_{\max}(W_{hh}) < 1 \implies \text{vanishing}, \qquad > 1 \implies \text{exploding}

Because it is the same matrix every time, the product is a power, and a power of anything other than exactly 1 goes to zero or to infinity.

This is why a plain RNN cannot learn a dependency more than about ten steps long, and why gated cells had to be invented.

One sequence, in order

Initialise the hidden state to zeros.
Read the first input, combine it with the state, and apply the nonlinearity.
Emit an output from the new state if the task needs one at this step.
Repeat for every timestep, carrying the state forward.
Sum the losses across steps, then backpropagate through the whole unrolled graph.

Long sequences are truncated, so the gradient is only carried back $k$ steps, which bounds the memory and formally limits what can be learned.

What it costs on modern hardware

\mathbf{h}_t \text{ requires } \mathbf{h}_{t-1}

The recurrence is sequential by definition, so the timesteps cannot be computed in parallel on a GPU no matter how many cores are available.

O(T) \text{ sequential steps}, \qquad O(1) \text{ for a transformer}

That is the decisive difference. A transformer sees the whole sequence at once and a recurrent network cannot, so training time scales with sequence length rather than with hardware.

Inference is the opposite case. An RNN carries a fixed-size state and costs the same per token however long the history is, while a transformer's cost grows with the context.

That advantage is why recurrent architectures keep returning, most recently as state space models, for settings where the sequence is very long and the state must stay small.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What limits how far back a plain RNN can remember?

NLP

20 concepts in this module, then a graded test with every answer explained.

Study this module →