TracksStudyAbout
Sign in
Start studying
Module 2 · Deep Learning Module contents
Module 2  ·  Card 25 / 25advanced

Mixed Precision Training

Training normally holds weights and activations as 32-bit floats, four bytes for every number.

Mixed precision does most of the arithmetic in 16 bits instead, halving memory and raising throughput.

Modern GPUs have dedicated units for 16-bit matrix multiplies, so the speedup is real rather than theoretical.

A master copy of the weights stays in 32 bits, because a tiny update would round away to nothing at 16.

Small gradients underflow to zero, so the loss is scaled up before the backward pass and scaled back after.

Back Finish module
Learn in depth ↓
Module 02 · Deep Learning · 25 concepts
In depth  /  15 lines

Mixed Precision Training, in more detail

A 32-bit float is the default numeric type in deep learning, and for most of the work it is overkill.

Mixed precision keeps the forward and backward arithmetic in a 16-bit format and the rest in 32.

Halving the width of activations halves the memory they occupy, so the batch can be twice as large.

It also halves the bytes moved between memory and the compute units, and training is usually memory-bound.

On top of that, recent GPUs have tensor cores that multiply 16-bit matrices several times faster.

Two numeric formats are in common use and they trade off differently.

Half precision has more mantissa bits and a narrow exponent range, so it is precise but overflows easily.

Bfloat16 keeps the exponent range of a 32-bit float and gives up mantissa bits instead.

That range is why bfloat16 usually trains without any extra machinery and half precision does not.

With half precision the danger is underflow: gradients smaller than about a millionth become exactly zero.

Loss scaling fixes this by multiplying the loss by a large constant before the backward pass.

Every gradient is scaled up with it, lands inside the representable range, then is divided back before the update.

The optimiser still keeps weights in 32 bits, because an update a thousand times smaller than the weight would disappear.

Modern frameworks handle all of this behind one autocast context and a gradient scaler.

The result is the same final accuracy in roughly half the time, which is why almost every large run uses it.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath Mixed Precision Training

Two number formats, used where each is safe

fp32:1+8+23 bits,fp16:1+5+10 bits\text{fp32} : 1 + 8 + 23 \text{ bits}, \qquad \text{fp16} : 1 + 5 + 10 \text{ bits}

The middle number is the exponent, which sets the range. The last is the mantissa, which sets the precision.

fp16 range≈6×10−8 to 65,504\text{fp16 range} \approx 6 \times 10^{-8} \text{ to } 65{,}504

Half precision halves the memory of every tensor and roughly doubles the arithmetic throughput on hardware with tensor cores.

What it loses is range, and gradients are exactly the quantity that lives near the bottom of that range.

Why a small gradient becomes zero

g<6×10−8  ⟹  g→0g < 6 \times 10^{-8} \implies g \to 0

Anything below the smallest representable value underflows to zero, and a zero gradient is indistinguishable from a parameter that had nothing to learn.

L′=S⋅L  ⟹  g′=S⋅g\mathcal{L}' = S \cdot \mathcal{L} \implies \mathbf{g}' = S \cdot \mathbf{g}

Loss scaling is the fix: multiply the loss by a large constant before the backward pass, which multiplies every gradient by the same constant and lifts them into range.

g=g′S\mathbf{g} = \frac{\mathbf{g}'}{S}

The gradients are divided by SS again before the optimiser step, so the update is unchanged and only the intermediate representation moved.

SS cannot be fixed in advance, because too small does not solve the underflow and too large overflows to infinity.

It is chosen dynamically: raise it while steps succeed, halve it and skip the step whenever a gradient comes back as NaN or infinity.

The master copy stays in fp32

θ←θ−ηg\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta \mathbf{g}

An update is typically a thousand times smaller than the parameter it modifies, and adding a tiny number to a large one in fp16 rounds straight back to the large one.

So the optimiser keeps a full precision copy of the weights and applies every update to that. The fp16 copy used in the forward pass is cast from it each step.

That costs memory rather than saving it, and it is the reason mixed precision reduces activation memory far more than parameter memory.

One step, in order

  1. Cast the weights to fp16 and run the forward pass in half precision.
  2. Multiply the loss by the current scale factor.
  3. Backward, producing scaled fp16 gradients.
  4. Unscale the gradients and check them for infinities or NaNs.
  5. If they are finite, step the fp32 master weights. If not, skip the step and halve the scale.

bf16, which removes most of this

bf16:1+8+7 bits\text{bf16} : 1 + 8 + 7 \text{ bits}

Brain float keeps fp32's full 8-bit exponent and gives up mantissa bits instead, so it has the same range as fp32 and less precision.

Because the range matches, gradients do not underflow and loss scaling is unnecessary. That is why every large model trained on recent hardware uses bf16 rather than fp16.

The lower precision is tolerable because gradient descent is an averaging process, and the noise from rounding is small next to the noise from sampling batches.

Some operations still run in fp32 whatever the mode: softmax, normalisation statistics, and any sum over a long axis.

Accumulating thousands of small values is where low precision actually hurts.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Why does mixed precision training keep a 32-bit master copy of the weights?
Concepts these five lines lean on
OptimizationPrecisionAccuracyLoss FunctionMatrices · Data ScienceRange · Data Science
Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close