Distributed training spreads one training run across many GPUs.
Data parallelism copies the model and splits the batch.
Tensor parallelism splits individual layers across devices.
Pipeline parallelism assigns different layers to different devices.
Frontier models combine all three, and communication becomes the bottleneck.
Distributed training spreads one training run across many GPUs.
It becomes necessary when a model or its batch will not fit on one device.
Data parallelism is the simplest form and the most common.
Every GPU holds a full copy of the model and processes a different slice of the batch.
Gradients are averaged across devices after each step so the copies stay in step.
That works until the model itself no longer fits on one GPU.
Tensor parallelism splits individual layers across devices.
A matrix multiplication is divided, with each device computing part of the result.
It needs fast interconnects, since devices exchange data within every layer.
Pipeline parallelism assigns different layers to different devices.
The batch is split into micro-batches so devices are not idle waiting for each other.
Frontier training combines all three, which is where the complexity lives.
Communication becomes the bottleneck rather than arithmetic.
ZeRO and FSDP shard optimizer state and gradients to cut memory further.
Failures are routine at this scale, so checkpointing is not optional.
Data parallelism runs the same model on different batches.
Tensor parallelism splits one matrix multiply across devices.
Pipeline parallelism puts different layers on different devices.
Which one to use is decided by what does not fit: the batch, a single layer, or the whole model.
A 70-billion-parameter model needs over a terabyte of optimiser state and gradients, so the question is never whether to distribute but how.
Every device holds a full copy, processes its own shard of the batch, and the gradients are averaged before the step.
The averaging moves roughly twice the parameter count per step regardless of the number of devices, which is why it overlaps with the backward pass rather than waiting for it.
ZeRO shards the optimiser state first, then the gradients, then the parameters.
Fully sharded data parallel keeps the data-parallel structure and splits the memory, gathering each layer's parameters just before it is needed and releasing them after.
That converts a memory problem into a bandwidth problem, and it is the default for large model training today.
A matrix multiply splits along the output dimension with no communication, and along the input dimension with a sum afterwards.
Attention splits naturally, one head per device.
Transformers split cleanly because heads are independent and the feed-forward block is two matrices in sequence.
That costs two all-reduces per layer.
The cost is communication inside every layer, which means tensor parallelism is used within a machine over fast interconnect and almost never across machines.
Splitting by layer means stage two waits for stage one on the first batch, and the idle time at the start and end is the bubble.
Many small micro-batches shrink it, and they also shrink the arithmetic efficiency of each one, which is the trade.
Using all three at once is called 3D parallelism.
That combination is what every frontier training run uses, and the ratios between the three are tuned to the specific cluster rather than chosen from a rule.
Model FLOPs utilisation is the number to watch. A well-tuned large run reaches 40 to 55 percent, and anything below 30 usually means communication is not overlapping.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.