As the forward pass runs, the framework records every operation and what it was applied to.
That record is a graph: each result remembers the inputs and the function that produced it.
Calling backward walks the graph in reverse and fills in a gradient for every parameter.
Nobody derives those gradients by hand any more, which is what made deep learning practical.
The graph is rebuilt on each forward pass, so loops and if statements in your model just work.
Backpropagation is the chain rule, and the chain rule needs to know what was computed from what.
Autograd is the bookkeeping that answers that question without anyone writing a derivative by hand.
As the forward pass runs, every operation on a tensor is recorded as a node in a graph.
Each node stores the inputs it consumed and the function it applied to them.
The result is a directed graph running from the input tensors down to a single scalar loss.
Calling backward on that loss walks the graph in reverse, node by node.
At each node the framework already knows the local derivative of that one operation.
It multiplies the gradient flowing in by that local derivative and passes the product further back.
Parameters are leaves of the graph, so what arrives at a leaf is that parameter's gradient.
PyTorch builds the graph as the code runs, which is what define-by-run means.
Because it is rebuilt every pass, a loop or a branch in your model needs no special treatment.
The cost is memory: every intermediate activation is held until the backward pass consumes it.
That is why batch size is limited by memory rather than by compute on most training runs.
Marking a tensor as not requiring gradients, or wrapping inference in no_grad, skips the recording entirely.
Understanding the graph is what turns a confusing out-of-memory error into an obvious one.
Every operation on a tensor records itself: what it did, what it was applied to, and how to differentiate itself.
The result is a directed acyclic graph whose nodes are tensors and whose edges are operations, built as a side effect of the forward pass.
Nothing is symbolic. The graph holds the actual values, which is why a derivative comes out as numbers rather than as an expression.
The chain rule can be applied from the inputs forward or from the loss backward, and the two cost wildly different amounts.
Forward mode computes the derivative of everything with respect to one input. Reverse mode computes the derivative of one output with respect to everything.
Training has millions of inputs and a single scalar loss, so reverse mode gets every gradient in one backward pass and forward mode would need millions.
Gradients accumulate because a tensor used in two places contributes through both paths, and the total derivative is their sum.
That accumulation is why frameworks require an explicit zeroing step between batches. Forgetting it silently sums the gradients of every batch so far.
The backward pass does roughly twice the arithmetic of the forward one, so a training step costs about three forward passes.
Memory is the sharper constraint. Every intermediate activation stays alive until its gradient is computed, so peak memory scales with depth times batch size.
Gradient checkpointing trades that back: store only some activations and recompute the rest during the backward pass, which is how large models fit at all.
Under a no-grad context no graph is recorded, which is the right mode for inference and saves both the memory and the bookkeeping.
Detaching a tensor cuts the edge behind it, so gradients flow no further back. That is a deliberate tool, and it is also the usual cause of a gradient that mysteriously arrives as zero.
An in-place operation on a tensor the graph still needs corrupts the backward pass, which is why frameworks raise an error rather than return a wrong number.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.