Dropout randomly switches off a fraction of neurons during each training step.
This stops the network relying too heavily on any single neuron.
It effectively trains a huge ensemble of thinner networks at once.
A typical dropout rate is between 0.2 and 0.5.
It is turned off at inference time so the full network is used.
Dropout switches off a random selection of neurons on every training step.
A rate of 0.5 means each neuron has a coin-flip chance of being ignored this step.
The choice is remade for every batch, so a different network trains each time.
The point is to stop the network leaning on any single neuron.
A neuron that might vanish cannot be relied on, so the others must cover the same ground.
That spreads the representation out instead of concentrating it in a few units.
It also behaves like training an enormous ensemble of thinner networks that share weights.
At prediction time nothing is dropped, since you want the full network and a stable answer.
The activations are scaled to account for more neurons being active than during training.
Frameworks handle that scaling for you, which is why models have a train and an eval mode.
Forgetting to switch to eval mode is a classic bug that quietly degrades results.
Typical rates run from 0.2 to 0.5, with the higher end on wide fully connected layers.
Too much dropout starves the network and it underfits instead.
Convolutional layers use it sparingly, since their weights are already shared and constrained.
Transformers use it lightly, and the largest models often train with almost none.
During training, each unit's output is multiplied by a random zero or one, so a different subnetwork is trained on every batch.
At , half the units are gone from any given step, and which half changes constantly.
Dropping units changes the expected size of the output, and the next layer's weights were tuned for the undropped size.
Left alone, activations would shrink by a factor of at training time and jump back up at inference, which is a different function.
Dividing by during training restores the expectation, and this is called inverted dropout.
It is the version every framework implements, and the reason is efficiency rather than elegance: inference then needs no dropout code at all, which is the path that runs a billion times.
At inference, dropout is simply switched off and the layer is the identity.
Forgetting to switch it off is a classic bug, and it looks like a model that gives a different answer every time you ask it the same question.
A unit cannot rely on any particular other unit being present, because that one might be gone next step.
So the network cannot build long chains of co-adapted units that only work together, and has to spread the representation out redundantly.
It is also close to training an exponentially large ensemble of thinned networks and averaging them at test time.
That is the other standard explanation, and the one that connects dropout to the rest of regularisation.
Modern large models use it sparingly: with enough data the overfitting it fights is no longer the binding constraint, and the noise it adds costs more than it saves.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.