CUDA is NVIDIA's platform for general-purpose GPU programming.
Kernels are small programs executed by thousands of threads at once.
Memory bandwidth, not raw arithmetic, is usually the real limit.
Moving data between CPU and GPU is expensive and worth minimising.
Most deep learning frameworks sit on top of CUDA.
CUDA is NVIDIA's platform for running general computation on a GPU.
It is the reason NVIDIA dominates machine learning, more than the hardware itself.
A GPU has thousands of simple cores rather than a few complex ones.
A kernel is a small program run by many threads at once, one per data element.
Threads are grouped into blocks, and blocks into a grid, which shapes how work is scheduled.
The counter-intuitive part is that arithmetic is rarely the limit.
Memory bandwidth usually is, since reading data is slower than computing with it.
That is why fusing operations helps, as it avoids writing intermediate results to memory.
Flash Attention is exactly this idea applied to attention.
Moving data between CPU and GPU is far slower again and worth minimising.
Tensor cores are dedicated units for matrix multiplication at reduced precision.
Using them is most of why mixed precision training is so much faster.
Frameworks sit on top of CUDA, so most people never write a kernel.
Understanding the model still explains why your training is slow.
Low GPU utilisation almost always means the data pipeline is starving the device.
A CPU has tens of cores, deep caches and branch prediction.
A GPU has thousands of cores, shallow caches and no speculation.
A GPU is built for one instruction applied to enormous amounts of data, and every architectural choice follows from that.
Threads are scheduled in groups of 32 that must follow the same instruction. A branch where half the warp goes each way runs both sides in sequence, which is why divergent code is slow.
Latency is hidden rather than avoided: when a warp waits for memory, another warp runs, so occupancy matters more than any single thread's speed.
Arithmetic intensity decides which limit applies, and most operations in a neural network are on the wrong side of it.
An activation function reads and writes more bytes than it does arithmetic, so it runs at a tiny fraction of peak no matter how it is written.
That is the whole argument for kernel fusion: doing several elementwise operations in one pass over memory turns several bandwidth-bound kernels into one.
Data moves from registers to shared memory to L2 to HBM, slower at every step.
Shared memory is roughly an order of magnitude faster than the main memory on the card, and it is measured in kilobytes per block.
Tiling exists to exploit exactly that: load a block of the matrices into shared memory, reuse it many times, then move on.
FlashAttention is the same idea applied to attention, and it is why it is faster despite doing more arithmetic.
That overhead is why a model made of many tiny operations runs poorly, and why CUDA graphs, which replay a whole recorded sequence of launches, help small-batch inference so much.
A card reporting 100 percent utilisation may be doing bandwidth-bound work at 5 percent of peak FLOPs, so the useful number is achieved FLOPs against the roofline.
Tensor cores need aligned shapes.
Dimensions that are multiples of 8 or 16 let the matrix units engage, and a hidden size of 1,000 rather than 1,024 can cost a large fraction of the throughput.
Host and device run asynchronously, so timing a kernel without an explicit synchronisation measures the launch rather than the work, which is the most common benchmarking mistake.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.