Batch Normalization in deep learning

Module 02 · Deep Learning · 25 concepts

In depth / 15 lines

Batch Normalization, in more detail

Batch normalization rescales the activations inside a layer while training.

For each feature it subtracts the mean and divides by the standard deviation of the batch.

Those statistics come from the other examples in the same batch, which is the key detail.

The result is then scaled and shifted by two parameters the network learns.

So the network can undo the normalisation if that turns out to be better.

Without it the scale of activations drifts as the weights change underneath.

Later layers then chase a moving target and training slows down.

Holding the scale steady allows much higher learning rates and far faster convergence.

It also injects a little noise, since each example is normalised by whichever rows it shared a batch with.

That noise gives a mild regularising effect for free.

At inference there is no batch, so running averages collected during training are used instead.

The gap between those averages and the training statistics is a common source of odd behaviour.

Small batches make the statistics unreliable, and a batch of one is meaningless.

That is the reason transformers use layer normalization instead.

In convolutional vision networks it remains close to standard.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Batch Normalization

Normalise, then undo it on purpose

Batch normalisation standardises each feature across the examples in the batch, then hands the network the means to reverse that if it wants to.

\mu_B = \frac{1}{m} \sum_{i=1}^{m} x_i \qquad \sigma_B^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu_B)^2

\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \qquad y_i = \gamma \hat{x}_i + \beta

$m$ : how many examples are in the batch.
$\mu_B, \sigma_B^2$ : the mean and variance of one feature over those examples.
$\epsilon$ : a small constant, about $10^{-5}$ , so a constant feature does not divide by zero.
$\gamma, \beta$ : learned scale and shift, one pair per feature.

Why add parameters that undo the normalisation

Forcing every activation to mean zero and variance one is a real constraint, and sometimes a harmful one.

A sigmoid fed only values near zero is operating in its linear region, which throws away the nonlinearity the layer exists to provide.

With $\gamma$ and $\beta$ the layer can learn any mean and any variance it likes, including the original ones.

\gamma = \sqrt{\sigma_B^2 + \epsilon}, \quad \beta = \mu_B \implies y_i = x_i

So the normalisation is a starting point the network can move away from, not a cage.

The axis, which is the whole difference from layer norm

The statistics are computed across the batch, separately for each feature.

For a batch of $m$ examples with $d$ features, that is $d$ means and $d$ variances, each averaged over $m$ numbers.

That is why the batch size matters: with $m = 2$ the estimates of mean and variance are almost noise, and the layer injects that noise into every forward pass.

Training and inference are different functions

At training time the statistics come from the batch. At inference there may be only one example, and its own mean is meaningless.

\mu_{\text{run}} \leftarrow \alpha \mu_{\text{run}} + (1 - \alpha) \mu_B

So training also keeps a running average of the statistics, and inference uses those fixed numbers instead.

This is the single most common source of "it works in training and not in production": the model was left in training mode, or the running statistics never converged.

One forward pass, in order

Compute the mean of each feature over the batch.
Compute the variance of each feature over the batch.
Subtract the mean and divide by the root of the variance plus epsilon.
Scale by the learned gamma and add the learned beta.
Update the running statistics for use at inference time.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why is batch normalization awkward with very small batches?

Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →