Continuous batching keeps a serving GPU busy by swapping requests in and out mid-flight.
Plain batching waits for every request in the group to finish before starting the next.
That wastes the GPU whenever one request is far longer than the rest.
Continuous batching drops finished requests and admits new ones at every step.
It is one of the largest throughput wins available when serving language models.
Continuous batching is what makes serving many users on one GPU economical.
Batching requests together is essential, since one request leaves the GPU mostly idle.
The weights must be read for every token regardless of how many requests share the pass.
So processing thirty-two requests together costs little more than processing one.
Plain static batching collects requests, runs them together, and waits for all to finish.
The problem is that requests have wildly different lengths.
One asking for two thousand tokens holds up thirty-one that finished after fifty.
The GPU spends most of the batch computing for a single remaining request.
Continuous batching manages the batch at every decoding step instead.
A finished request is removed immediately and a waiting one takes its slot.
The batch stays full, and new arrivals do not wait for the current group to end.
Throughput improvements of several times are typical on realistic traffic.
It interacts closely with KV cache memory, since each active request holds a cache.
PagedAttention manages that memory so slots can be reused without fragmentation.
vLLM popularised this combination and it is now standard in serious serving stacks.
A fixed batch runs until its longest sequence finishes. Every sequence that ended earlier leaves its slot idle for the remainder.
Output lengths in real traffic vary by an order of magnitude, so that fraction is often below 0.3, and the wasted capacity is paid for at full price.
Continuous batching makes the scheduling decision at each decoding step instead, so a completed sequence frees its slot immediately and a queued request takes it.
Decoding is a loop of identical steps, and the batch dimension is just the outer index of that step.
Nothing in the arithmetic requires the members of a batch to have started together, which is the observation the technique rests on.
That is the measured gain over static batching on realistic length distributions, and it is the single largest serving improvement available.
Each active sequence holds a growing cache, so the limit on concurrency is how many caches fit rather than how much arithmetic a step costs.
Reserving the maximum length for every request wastes most of the memory, because most requests are far shorter.
Paged attention is the fix.
Splitting the cache into small blocks allocated as the sequence grows removes that waste, the same way virtual memory removes it for processes.
It also allows sharing: two requests with the same prompt prefix can point at the same blocks, which is copy-on-write applied to attention.
Step three is where the latency trade lives. A long prompt arriving mid-flight delays everyone's next token unless prefill is chunked into pieces that interleave.
Tail latency rises with throughput.
A fuller batch means each step does more work, so every active sequence waits slightly longer per token.
Under memory pressure the scheduler may preempt a sequence entirely, dropping its cache and recomputing it later, which is invisible to the caller and expensive.
That calls for admission control.
The honest configuration exposes a target concurrency rather than accepting everything, because a server that admits every request degrades all of them equally instead of queueing some.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.