Inference is using a trained model to generate output.
Prefill processes the whole prompt in one parallel pass.
Decode then produces output tokens one at a time.
A KV cache stores past key and value vectors so they are not recomputed.
Decoding is sequential, which is why long outputs feel slow.
Inference is running a trained model to produce output, and it is where the bill lives.
It happens in two phases with very different characteristics.
Prefill processes the entire prompt in one parallel pass.
It is compute-heavy and fast, since every position is handled at the same time.
Decode then produces the answer one token at a time.
Each token needs its own pass through the whole model.
Nothing about that can be parallelised, since each token depends on the one before it.
That is why the first word appears quickly and the rest arrives at a steady trickle.
The KV cache stores the key and value vectors of everything already processed.
Without it every new token would recompute the entire history.
The cache grows with the conversation and can consume more memory than the weights.
Decoding is limited by memory bandwidth rather than arithmetic, since the weights must be read per token.
That is why batching many users together is nearly free and why providers do it aggressively.
Time to first token and tokens per second are the two numbers users actually feel.
Most inference optimisation is an attack on one of those two.
Prefill computes the keys and values for the whole prompt at once, so it is a large matrix multiply and the hardware runs near its peak.
Decode produces one token per pass, so every weight in the model is read from memory to serve a single token.
That is the whole story of serving cost. Prefill is compute-bound, decode is memory-bound, and an optimisation that helps one usually does nothing for the other.
A 7-billion-parameter model in 16-bit weights reads about 14 GB per token. On an accelerator with 2 TB per second of bandwidth, that is roughly 7 ms, or 140 tokens per second.
The arithmetic in that step would take well under a millisecond. The chip spends most of its time waiting for weights.
Batching is therefore free throughput.
The weights are read once and serve every sequence in the batch, which is why a server's tokens per second rises almost linearly with batch size until the cache runs out of room.
Time to first token is set by prefill, so it grows with prompt length. Time per output token is set by decode and is roughly flat.
A long prompt with a short answer is a prefill problem. A short prompt with a long answer is a decode problem, and they are fixed by different things.
Streaming exists because that first term can be hidden behind the rest, and a user reads more slowly than the model writes.
The cache grows with every token and every concurrent request, and it is usually what limits batch size rather than the weights.
Sharing keys and values across query heads cuts that memory several-fold, which is why nearly every recent model uses it.
Quantising the weights to 8 or 4 bits attacks the bandwidth term directly, and it speeds up decode roughly in proportion.
That is the clearest example of why the bottleneck is worth knowing before optimising.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.