Inference Optimization

Inference optimisation makes serving a trained model cheaper and faster.

It matters more than training cost for most deployed systems, since it is paid on every request.

The KV cache is the first and most important technique.

Without it, every new token would recompute attention over the whole history.

With it, only the new token is processed and the rest is read from the cache.

That changes generation from quadratic work to linear.

Continuous batching keeps the GPU busy across many concurrent requests.

Requests are admitted and retired at each step rather than in fixed groups.

Speculative decoding drafts several tokens with a small model and verifies them in one pass.

Quantization reduces the bytes read per token, which is what decoding is limited by.

PagedAttention manages KV cache memory in pages, cutting waste from fragmentation.

Flash Attention reduces memory traffic within attention itself.

Together these can improve throughput by an order of magnitude.

vLLM, TensorRT-LLM and similar servers implement most of them for you.

Writing your own serving loop is rarely worth it when these exist.