SGD updates the weights using one small batch at a time instead of all the data.
That makes each update far cheaper and much faster.
The noise from small batches can actually help escape poor local minima.
Momentum is usually added to smooth out the noisy path.
It is still competitive today, especially in computer vision.
Stochastic gradient descent computes the gradient on a small batch instead of all the data.
A batch of 32 or 128 rows stands in for the whole training set.
That estimate is wrong, but it is wrong cheaply and in a different way each time.
You get hundreds of updates in the time full batch descent would take for one.
More updates on rough directions beats one update on a perfect direction.
Stochastic means random, referring to which rows land in each batch.
The data is shuffled every epoch so the same grouping never repeats.
The noise turns out to help, since it can shake the weights out of a poor spot.
It also acts as a mild regulariser, because the model cannot settle on a razor-thin solution.
The path to the answer looks jagged rather than smooth, which is expected.
Momentum is nearly always added, carrying a fraction of the previous step into the next.
That damps the jitter across a narrow valley and speeds progress along it.
A typical momentum is 0.9, meaning most of the last step is retained.
SGD with momentum still wins on image models, where it often generalises better than Adam.
It needs a well-tuned learning rate, which is exactly what Adam saves you from.
The true gradient is the average over every training example, so one exact step costs a full pass through the dataset.
A batch gives an estimate that is correct on average and wrong on any particular step, which is the trade the method is named after.
A thousand noisy steps make far more progress than one exact step, because the loss surface does not require precision to descend.
Quadrupling the batch halves the gradient noise, and it quadruples the cost of a step. The returns are square root while the price is linear.
Because the noise falls with , the learning rate is usually scaled up with it. The linear scaling rule holds well up to a batch size in the thousands and breaks after that.
The noise is not purely a cost. It is also what lets SGD escape sharp minima, and models trained with very large batches generalise measurably worse for that reason.
The update becomes a running average of past gradients rather than the latest one, which cancels the noise in directions that keep changing sign.
That average spans roughly ten gradients at , so momentum builds speed along a consistent slope and damps oscillation across a narrow valley.
Plain SGD without momentum is rarely the right choice, and almost no published result uses it.
SGD applies one global learning rate, so a parameter with a small gradient takes a small step and the geometry of the loss is respected.
Adam rescales each parameter by its own gradient history, which converges faster and often lands in a sharper minimum.
On image models trained to convergence, SGD with momentum and a cosine schedule still produces the best test accuracy, and that result has survived a decade of attempts to overturn it.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.