ResNet introduced the residual or skip connection.
A skip connection adds a layer's input directly to its output.
This gives gradients a clean path backwards through very deep networks.
It made networks of 50, 101 and even 152 layers trainable.
Residual connections are now standard in almost every deep architecture, transformers included.
Before ResNet, adding layers past about twenty made networks worse rather than better.
That was not overfitting, since the training error rose too.
The deeper network could not even learn to match the shallower one.
The gradient had to survive too many multiplications to reach the early layers.
ResNet solved it with a connection that skips over a block entirely.
The block's input is added straight to its output before moving on.
So the block only has to learn the difference, or residual, rather than the whole mapping.
If the best thing to do is nothing, the block can output zeros and the input passes through.
That makes a deeper network at least as good as a shallower one by construction.
The addition also gives gradients a clean route backwards that skips the multiplications.
That is the part that actually matters for training.
It made 50, 101 and 152-layer networks trainable, and they won the 2015 ImageNet competition.
ResNet-50 remains a standard backbone and a standard baseline a decade later.
The same skip connection appears in every transformer block.
It is one of the small ideas that changed what was buildable.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.