Gradient Descent in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Gradient Descent, in more detail

Think of the error as a valley, with your model standing somewhere on the slope.

The goal is to reach the bottom, where the error is smallest.

You cannot see the whole valley, only the ground under your feet.

The gradient tells you which way is uphill.

So you take a step the other way.

Then you look again, and step again.

The learning rate is how big each step is.

This is the setting people get wrong most often.

Big steps make you bounce from one side of the valley to the other.

Tiny steps take forever, and can stop somewhere that is not the bottom.

You can work out the slope using all your data, which is slow but steady.

Or use a small batch of rows at a time, which is much faster.

Small batches give a rougher answer, and that roughness often helps.

It can knock the model out of a shallow dip that is not the real bottom.

Most training today changes the step size on its own as it goes.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Gradient Descent

The update

Every parameter moves a small step in the direction that reduces the loss fastest, which is the direction opposite the gradient.

\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t)

$\theta_t$ : every parameter in the model, as one vector, at step $t$ .
$\nabla_\theta \mathcal{L}$ : the gradient: how much the loss changes per unit change in each parameter.
$\eta$ : the learning rate, the one number you choose.

The minus sign is the whole idea. The gradient points uphill, so you go the other way.

Why the gradient is the steepest direction

For a small step $d$ , the change in loss is approximately the dot product of the gradient with that step.

\mathcal{L}(\theta + d) \approx \mathcal{L}(\theta) + \nabla \mathcal{L} \cdot d

A dot product is most negative when the two vectors point in exactly opposite directions.

So among all steps of a given length, the one that reduces the loss most is the one straight down the gradient.

This is a local statement and only a local one: it says nothing about where the minimum is, only which way is downhill right here.

What the learning rate does

\Delta \theta = -\eta \nabla \mathcal{L}

The gradient decides the direction and the relative size of each parameter's move; $\eta$ scales all of them together.

Too small and training crawls, taking thousands of steps to cross a region a handful would have covered.

Too large and the step overshoots the valley floor and lands higher up the opposite side, so the loss rises instead of falling.

The threshold between the two sits at roughly $\eta < 2 / \lambda_{\max}$ , where $\lambda_{\max}$ is the largest curvature of the loss surface.

So the safe learning rate is a property of the problem, not of your taste.

One step, in order

Run a forward pass and compute the loss.
Run backward and get the gradient for every parameter.
Multiply the whole gradient by the learning rate.
Subtract that from the parameters.
Throw the gradients away and start again.

The version nobody actually runs

\nabla \mathcal{L}(\theta) = \frac{1}{N} \sum_{n=1}^{N} \nabla \ell(x_n, \theta)

Written this way the gradient is an average over the entire training set, so one step needs a pass over all of it.

That is called batch gradient descent, and on a dataset of any size it is unusable. A single step could take an hour.

What is actually run estimates that average from a small random sample instead, which is stochastic gradient descent.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does the learning rate control in gradient descent?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →