GPUs have thousands of simple cores built for parallel arithmetic.
Neural networks are mostly matrix multiplications, which parallelise perfectly.
This makes GPU training often 10 to 100 times faster than CPU.
Model weights and data have to be explicitly moved onto the device.
GPU memory is usually the binding constraint on how large a model or batch can be.
A CPU has a few very fast cores, and a GPU has thousands of slow ones.
Neural networks are mostly large matrix multiplications, which split perfectly across many cores.
That mismatch is why the same training run can take days on a CPU and hours on a GPU.
The speedup is typically ten to a hundred times, depending on the model.
Weights and data have to be moved onto the device explicitly before they can be used.
Mixing a tensor on the GPU with one on the CPU is an immediate error.
Moving data back and forth is slow, so it is done as rarely as possible.
GPU memory is the real constraint, and it is much smaller than system memory.
It has to hold the weights, the gradients, the optimizer state and every stored activation.
Adam alone triples the memory the weights need.
Out-of-memory errors are answered by a smaller batch, mixed precision, or gradient accumulation.
Mixed precision keeps most numbers in 16 bits and roughly halves the memory.
Utilisation below full usually means the GPU is waiting on data loading, not on arithmetic.
More workers in the data loader is the usual fix for that.
Renting one by the hour is the normal way in, since owning the hardware rarely pays.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.