Weights start as small random numbers, and the size of those numbers matters more than it sounds.
Set them all to zero and every neuron in a layer computes the same thing forever.
Too large and the signal grows layer by layer until the activations saturate.
Too small and it shrinks until the deepest layers have almost nothing to learn from.
Xavier initialisation suits sigmoid and tanh, and He initialisation is the default for ReLU.
Before training starts every weight needs a value, and the choice is not free.
Set them all to zero and every neuron in a layer sees the same input and the same gradient.
They then stay identical forever, so a layer of 500 neurons learns one thing.
Small random values break that symmetry and let the neurons specialise.
The scale of that randomness is the part that matters.
Each layer multiplies the signal, so a scale slightly above one compounds layer by layer.
Twenty layers later the activations are enormous and the gradients explode.
A scale slightly below one compounds the other way and the signal fades to nothing.
The fix is to size the random numbers by how many inputs a neuron has.
Xavier initialisation does this for sigmoid and tanh, which are symmetric around zero.
He initialisation doubles the variance to account for ReLU discarding half its input.
ReLU networks with Xavier initialisation train visibly worse, which is how the split was found.
Biases are usually just set to zero, since the weights already broke the symmetry.
Every framework picks a sensible default, so you rarely set this by hand.
It matters when a deep network refuses to learn at all from the very first epoch.
If every weight starts identical, every unit in a layer receives the same gradient and stays identical forever. The layer has one unit's worth of capacity, whatever its width.
Randomness is what breaks that symmetry, which is why initialisation is random rather than clever.
The scale is the harder half. A layer's pre-activation is a sum of products, so its variance grows with the fan-in unless the weights shrink to match.
Setting the weight variance to the reciprocal of the fan-in makes the output variance equal to the input variance, so a signal neither explodes nor dies as it passes through the stack.
Xavier initialisation averages the two directions, because the forward pass is scaled by the fan-in and the backward pass by the fan-out.
It assumes the activation is roughly linear near zero, which is true of tanh and false of ReLU.
He initialisation corrects for that. ReLU zeroes half its inputs and so halves the variance, and the factor of 2 puts it back.
Use He with ReLU and its relatives, Xavier with tanh and sigmoid. Getting this one factor wrong is enough to stop a 30-layer network training at all.
Push one batch through an untrained network and print the standard deviation of each layer's activations.
Falling steadily towards zero means the initialisation is too small and the signal is dying. Growing without bound means it is too large.
A healthy start is roughly flat across depth, and it costs one forward pass to check.
Normalisation layers rescale activations at every step, so a network with batch norm or layer norm tolerates an initialisation that would otherwise be fatal.
Residual networks go further and scale the residual branches down by the depth, so the identity path dominates at the start and the network begins close to shallow.
Initialisation is now insurance rather than the main event, and it is still the first thing to check when the loss does not move at all.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.