Adam in deep learning, explained in 5 lines

Module 02 · Deep Learning · 25 concepts

In depth / 15 lines

Adam, in more detail

Adam is the optimizer most people reach for first, and usually never leave.

Plain gradient descent applies the same learning rate to every weight.

That is a poor fit when some weights need large moves and others tiny ones.

Adam keeps two running averages for each parameter as training goes along.

The first is an average of recent gradients, which is momentum by another name.

The second is an average of recent squared gradients, which measures how noisy that weight is.

The step is the first divided by the square root of the second.

So a weight with consistently large gradients ends up with a smaller effective rate.

A weight that has barely moved gets a relatively larger one.

That is what adaptive means here: every parameter gets its own step size.

Early in training both averages start at zero, so a correction is applied to unbias them.

The name comes from adaptive moment estimation, the two moments being those averages.

It converges quickly and tolerates a badly chosen learning rate far better than SGD.

It stores two extra numbers per parameter, so it needs roughly three times the memory.

The usual defaults of 0.9 and 0.999 are almost never worth changing.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Adam

The two averages

Adam carries two running averages for every single parameter, and both are updated on every step.

m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t

v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

$g_t$ : the gradient for this parameter at step $t$ .
$m_t$ : the average of recent gradients, which is momentum under another name.
$v_t$ : the average of recent squared gradients, which measures how noisy this parameter has been.
$\beta_1 = 0.9$ : how much of the old first average survives each step.
$\beta_2 = 0.999$ : the same for the second, and much slower on purpose.

Both are exponential moving averages: the weight on a gradient from $k$ steps ago is $\beta^k$ , so old news fades rather than being dropped.

Why they start wrong

Both averages are initialised at zero, so early on they are pulled towards zero by a starting value that means nothing.

\hat{m}_t = \frac{m_t}{1 - \beta_1^{\,t}} \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^{\,t}}

At $t = 1$ the denominator for $\hat{m}$ is $1 - 0.9 = 0.1$ , so the first average is multiplied by ten and the bias is gone in one step.

For $\hat{v}$ the denominator is $0.001$ , which is why the second average needs the correction for far longer than the first.

That is all bias correction is: dividing out exactly how much of the average is still the zero it was initialised to.

The step itself

\theta_t = \theta_{t-1} - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

$\eta$ : the learning rate, the one number you still choose.
$\epsilon = 10^{-8}$ : a floor that stops the division blowing up when a parameter's gradient has been zero.

Dividing by the root of the second average is the whole trick, and the root is what makes it a rescaling rather than a squashing.

A parameter whose gradients are consistently large has a large $\hat{v}$ , so it gets a smaller effective step.

A parameter whose gradients have been small has a small $\hat{v}$ , so the same $\eta$ moves it relatively further.

The units cancel: $\hat{m}$ is a gradient and $\sqrt{\hat{v}}$ is a gradient, so the ratio is dimensionless and $\eta$ alone sets the size of the move.

One step, in order

The backward pass hands you $g_t$ for this parameter.
Update $m_t$ from $m_{t-1}$ and $g_t$ .
Update $v_t$ from $v_{t-1}$ and $g_t^2$ .
Divide both by their bias correction to get $\hat{m}_t$ and $\hat{v}_t$ .
Take the step $\eta \, \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$ and subtract it.

Steps two and three are why Adam stores two extra numbers per parameter and needs roughly three times the memory of plain SGD.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What makes AdamW different from Adam?

Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →