Backpropagation in deep learning

Module 02 · Deep Learning · 25 concepts

In depth / 15 lines

Backpropagation, in more detail

Backpropagation answers one question: how much did each weight contribute to the error?

Without an answer you would be guessing which of millions of numbers to move.

It works by applying the chain rule from calculus, which is how derivatives compose.

The chain rule says a small change here causes a change there, times a change further on.

So it starts at the loss and walks backwards through the network.

At each layer it computes how the loss would change if that layer output slightly more.

That is combined with what the layer stored on the forward pass to give the weight gradients.

The same quantity is passed further back so the previous layer can do the same.

One backward pass gives every weight in the network its own gradient.

It costs roughly twice a forward pass, no matter how many parameters there are.

That efficiency is the entire reason training large networks is possible.

The gradients are then handed to the optimizer, which decides the actual step.

Backpropagation computes direction, and the optimizer decides how far to go.

Frameworks build the graph of operations as you go and differentiate it automatically.

You almost never write it yourself, but knowing it explains vanishing gradients immediately.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Backpropagation

It is the chain rule

Backpropagation has no content beyond the chain rule from calculus, applied to a function that happens to be a hundred layers deep.

\frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}

A network is a composition of functions, so the derivative of the loss with respect to an early weight is a product of the derivatives of everything after it.

\frac{\partial \mathcal{L}}{\partial W^{(1)}} = \frac{\partial \mathcal{L}}{\partial a^{(L)}} \cdot \frac{\partial a^{(L)}}{\partial a^{(L-1)}} \cdots \frac{\partial a^{(1)}}{\partial W^{(1)}}

$a^{(l)}$ : the activations coming out of layer $l$ .
$W^{(l)}$ : that layer's weights.
$L$ : how many layers there are.

Why backwards and not forwards

The chain rule does not say which end to multiply from, and the two ends cost wildly different amounts.

A network has millions of parameters and one scalar loss, so the Jacobian is very tall and very thin.

Multiplying right to left means carrying a full matrix at every step. Multiplying left to right means carrying a single row, which is a vector.

\delta^{(l)} = \left( W^{(l+1)} \right)^{\top} \delta^{(l+1)} \odot f'(z^{(l)})

$\delta^{(l)}$ : the error signal at layer $l$ : how much the loss changes per unit change in that layer's pre-activation.
$\odot$ : elementwise multiplication.
$f'$ : the derivative of the activation function.

That recurrence is backpropagation. Every layer takes the error from the layer above, pushes it back through the weights, and scales it by the local slope.

From the error signal to the actual gradients

\frac{\partial \mathcal{L}}{\partial W^{(l)}} = \delta^{(l)} \left( a^{(l-1)} \right)^{\top} \qquad \frac{\partial \mathcal{L}}{\partial b^{(l)}} = \delta^{(l)}

The weight gradient is an outer product of the error coming back with the activation that went in.

This is why the forward pass has to be kept in memory: those activations are needed again on the way back, and that is where most of training's memory goes.

One step, in order

Forward: compute every layer's output and keep it.
At the top, compute the derivative of the loss with respect to the final output.
Push that error back through one layer: transpose-multiply by the weights, then scale by the activation's slope.
At each layer, form the weight gradient from that error and the stored input.
Repeat down to the first layer, then update everything at once.

What it costs

The backward pass touches every weight exactly as many times as the forward pass does, so it costs roughly twice as much arithmetic and needs the activations kept alive throughout.

That two-to-one ratio is the reason a training step is about three times the cost of an inference step, not a hundred times.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does backpropagation actually compute?

Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →