Forward Propagation in deep learning

Module 02 · Deep Learning · 25 concepts

In depth / 15 lines

Forward Propagation, in more detail

Forward propagation is the pass where an input travels through the network to an answer.

It starts with your features written as a row of numbers.

The first layer multiplies them by its weight matrix and adds its biases.

The activation function is applied to every value in that result.

The output becomes the input of the next layer, and the same thing happens again.

That repeats until the last layer, which produces the prediction.

In code this is a handful of matrix multiplications in a row.

Writing it as matrices is what lets a GPU do the whole batch at once.

A batch of 64 examples travels through together, not one after another.

During training every intermediate result is kept, because the backward pass needs them.

That storage is why training needs far more memory than prediction does.

At inference nothing needs keeping, so the intermediates are thrown away as they go.

The same pass is what runs when a trained model is actually used.

So the cost of one prediction is exactly the cost of one forward pass.

Training is this pass, then a backward pass, repeated a few hundred thousand times.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Forward Propagation

A layer is a matrix multiply and a bend

\mathbf{z}^{[l]} = W^{[l]} \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}

\mathbf{a}^{[l]} = g\big( \mathbf{z}^{[l]} \big)

$\mathbf{a}^{[l-1]}$ : the activations arriving from the layer below.
$W^{[l]}$ : the weight matrix of layer $l$ .
$g$ : the activation function, applied to each element on its own.

Those two lines are the entire forward pass. Every architecture is a choice of what $W$ looks like and where the bends go.

The input layer is $\mathbf{a}^{[0]} = \mathbf{x}$ , and the output of the last layer is the prediction.

The shapes have to agree

W^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}, \quad \mathbf{b}^{[l]} \in \mathbb{R}^{n_l}

A layer taking 784 inputs to 128 units holds a $128 \times 784$ matrix and a vector of 128 biases, which is 100,480 numbers.

Z^{[l]} = W^{[l]} A^{[l-1]} + \mathbf{b}^{[l]}

A batch changes nothing but a shape: stack $m$ examples as columns and the same multiply does all of them at once.

The bias is a vector added to every column, which is broadcasting, and it is the one step where the shapes do not match exactly on purpose.

One pass, in order

Put the input batch in $A^{[0]}$ .
Multiply by the layer's weights and add the bias.
Apply the activation elementwise.
Hand the result to the next layer as its input.
At the last layer, apply softmax or the identity instead, depending on the task.
Keep every $Z$ and $A$ , because the backward pass needs them.

Why the nonlinearity is not optional

W_2(W_1 \mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (W_2 W_1)\mathbf{x} + (W_2 \mathbf{b}_1 + \mathbf{b}_2)

Two linear layers with nothing between them collapse into one linear layer, because the product of two matrices is a matrix.

A hundred layers collapse the same way. Without $g$ , depth buys exactly nothing.

What it costs

\mathrm{FLOPs} \approx 2 \sum_{l} n_l \, n_{l-1}

Each weight is used once for a multiply and once for an add, so the forward cost is about twice the parameter count per example.

Memory is the other half of the bill: every activation is stored until the backward pass reads it, which is why training needs far more memory than inference.

Inference can free each layer's activations as soon as the next layer has consumed them, and that single difference is most of the gap.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

When does forward propagation happen?

Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →