Calling a model in a loop from a web handler wastes almost all of the GPU you paid for.
A serving framework batches requests continuously and manages the KV cache across them.
vLLM, TGI and TensorRT-LLM are the common choices, and they differ mostly in hardware and tuning.
The main win is throughput: many times more tokens per second from the same card.
They also handle streaming, quantized weights, and fair scheduling between concurrent users.
Running a language model behind a naive web handler is enormously wasteful.
Each request is processed alone, so the GPU sits mostly idle between short bursts of work.
Generation is bound by memory bandwidth, and one sequence at a time uses very little of it.
A serving framework exists to keep that hardware busy across many concurrent requests.
Continuous batching is the central technique and is what separates these systems from naive batching.
Static batching waits for a group of requests and runs them together until all have finished.
Because responses differ in length, the whole batch is held hostage by the longest one.
Continuous batching admits a new request the moment any sequence in the batch completes.
The result is that the batch stays full and the GPU stays saturated.
The second technique is careful management of the KV cache, which dominates memory during serving.
vLLM introduced paged attention, storing the cache in fixed blocks like virtual memory pages.
That removes the fragmentation that otherwise wastes a large fraction of GPU memory.
Prefix caching goes further by sharing the cache for a common prompt across many requests.
For a system with a long shared system prompt, that alone is a substantial saving.
The frameworks differ mainly in hardware support and tuning effort rather than in the ideas.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.