In deep networks gradients are multiplied together layer after layer.
If those numbers are small the gradient vanishes before reaching early layers.
If they are large it explodes and training becomes unstable.
Vanishing gradients mean the early layers barely learn anything.
ReLU, residual connections, normalization and gradient clipping are the standard defences.
Backpropagation multiplies a term for each layer it passes through.
Multiplying twenty numbers of about 0.5 leaves you with roughly one in a million.
The gradient reaching the first layer is then far too small to move anything.
That is a vanishing gradient, and the early layers simply stop learning.
The network still trains, but only its last few layers actually change.
The symptom is a loss that improves a little and then stops well short of good.
Sigmoid and tanh make it worse, since their slope is under one almost everywhere.
Multiplying numbers above one instead compounds upward and the gradient explodes.
Then one step throws the weights into nonsense and the loss becomes not-a-number.
Recurrent networks feel both most sharply, because depth equals sequence length.
ReLU helps the vanishing case, since its slope is exactly one for positive inputs.
Residual connections give the gradient a path that skips the multiplication entirely.
Normalisation layers keep the scale of activations steady from layer to layer.
Gradient clipping catches the exploding case after the fact.
Together these are why networks with a hundred layers can be trained at all.
The gradient reaching an early layer is the product of one Jacobian per layer above it.
Products of many numbers do not drift. They collapse or they blow up, and depth sets the exponent.
That is the whole phenomenon. Vanishing and exploding gradients are the same equation with the factor on either side of 1.
Two things multiply at every layer: the activation's derivative and the weight matrix.
A sigmoid can contribute at most a quarter, so its best case still divides the gradient by four per layer.
The weights contribute their largest singular value, which is why initialisation scale and weight decay both show up in this problem.
Recurrent networks are the extreme case: the same matrix is applied at every timestep, so the factor is raised to the sequence length rather than the depth.
A vanishing gradient is silent. The loss falls a little and then flattens, early layers stop changing, and the network behaves as though it were shallow.
An exploding gradient is loud. The loss jumps to a huge value or to NaN in a single step, and the weights are ruined immediately.
The loud one is easier to live with, because it is obvious and the fix is one line.
The residual connection is the deepest of these.
Adding the identity puts a 1 into every factor of the product, so the gradient cannot decay geometrically no matter how many layers there are.
That single change is what took networks from about 20 layers to more than a thousand.
Print the norm of the gradient at each layer after the first backward pass, not after an hour of training.
A healthy network shows norms within an order of magnitude of each other across depth. A tenfold drop per layer is the problem, visible in one step.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.