Residual Connections in transformers

Module 05 · Transformers · 19 concepts

In depth / 15 lines

Residual Connections, in more detail

A residual connection adds a sublayer's input straight to its output.

So the block computes something and that something is added to what was already there.

The block therefore learns a correction rather than a replacement.

If no correction is needed it can output near zero and the input passes through untouched.

That sounds minor and it is what makes deep stacks trainable at all.

The gradient gets a direct route backwards that bypasses the block entirely.

It no longer has to survive multiplication by every layer's weights on the way back.

Without that route, a hundred-layer model would leave its early layers essentially untrained.

In a transformer every attention and every feed-forward sublayer is wrapped in one.

The result is often described as a residual stream running the height of the model.

Each layer reads from that stream and writes its contribution back into it.

Interpretability research leans on this picture, since contributions can be traced individually.

It also means every layer works in the same vector space, which is unusual and useful.

The idea came from ResNet in vision and transferred to language unchanged.

It is now in essentially every deep architecture, and its absence is the notable case.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Residual Connections

The equation

y = F(x) + x

$F$ : whatever the layer computes: an attention block, a convolution, a feed-forward network.
$x$ : the input, added back unchanged.

The layer no longer produces the output. It produces the change to the input, which is why $F$ is called the residual.

For that addition to be defined, the input and output must have the same width, which is why transformer blocks preserve $d_{\text{model}}$ throughout.

What it does to the gradient

\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I

The identity term is the whole point. Whatever the layer's own Jacobian does, the derivative can never be smaller than the identity contributes.

Through $L$ stacked layers the gradient reaching the bottom is a product of $L$ such terms.

\prod_{l=1}^{L} \left( \frac{\partial F_l}{\partial x} + I \right)

Without the $+I$ , that is a product of $L$ matrices with norms typically below one, and it decays exponentially.

That is the vanishing gradient, and it is what stopped networks deeper than about twenty layers from training at all.

With it, there is a path from the loss to every layer that is multiplied by one at each step.

Why depth stopped hurting

Before residuals, adding layers past a certain depth made training error worse, not just test error.

That was not overfitting. A deeper network can represent everything a shallower one can, by setting the extra layers to the identity. Plain layers find the identity hard to learn.

F(x) = 0 \implies y = x

With a residual, the identity is what the layer does when its weights are zero. It is the default rather than something to be discovered.

So a layer that has nothing useful to add can cheaply do nothing, and depth becomes safe.

One block, in order

Keep a copy of the input.
Run it through the sublayer.
Add the copy back to the result.
Normalise, either before step two or after step three depending on the convention.
Pass on, with the width unchanged.

The other reading

A residual network can be seen as an ensemble of many shallower paths: each residual connection offers a choice of going through the block or around it, so $L$ blocks give $2^L$ routes.

That explains why removing a single trained layer from a residual network usually barely changes the output, while doing the same to a plain network destroys it.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does a residual connection let each block learn?

Transformers

19 concepts in this module, then a graded test with every answer explained.

Study this module →