TracksStudyAbout
Sign in
Start studying
Module 14 · MLOps / LLMOps Module contents
Module 14  ·  Card 6 / 22beginner

Batch vs Real-Time Inference

Real-time inference answers one request while someone waits, so latency is the whole game.

Batch inference processes a large pile of inputs on a schedule, where only throughput matters.

Batch is far cheaper per item, because the hardware stays busy and requests are grouped efficiently.

Many features assumed to need real-time do not: recommendations can be computed nightly and looked up.

The question to ask is how stale an answer is allowed to be, and the cost follows from that.

Back Next
Learn in depth ↓
Module 14 · MLOps / LLMOps · 22 concepts
In depth  /  15 lines

Batch vs Real-Time Inference, in more detail

Every inference workload is one of two shapes, and choosing the wrong one is expensive.

Real-time inference serves a single request while a user or a caller waits for the answer.

The metrics that matter are latency percentiles, particularly the slow tail rather than the average.

Capacity has to be provisioned for the peak, which means paying for idle hardware most of the time.

Batch inference takes a large set of inputs and processes them on a schedule.

Nobody is waiting, so the only thing that matters is total throughput per unit of cost.

That freedom is what makes it cheap: requests can be grouped into large batches with no latency penalty.

Hardware runs at full utilisation, and interruptible or spot capacity becomes usable.

Providers offer batch APIs at a substantial discount for exactly this reason.

The design question is not technical but a product one: how fresh does this answer need to be?

A great deal of what is built as real-time does not need to be.

Recommendations, risk scores, summaries and enrichment can all be precomputed and looked up instantly.

A lookup is faster than any model call, so the user experience often improves as the cost falls.

Streaming sits between the two, processing events as they arrive with a delay of seconds.

Many mature systems run all three: batch for the bulk, streaming for updates, real-time for what genuinely cannot wait.

Deep dive

What is underneath Batch vs Real-Time Inference

One model, two economics

real time:minimise T per request\text{real time} : \text{minimise } T \text{ per request}
batch:maximise requests per hour\text{batch} : \text{maximise requests per hour}

The same model serves both, and almost every configuration choice that helps one hurts the other.

throughput=BTbatch\text{throughput} = \frac{B}{T_{\mathrm{batch}}}

Because the weights are read once per step regardless of batch size, adding requests to a batch costs almost nothing until the arithmetic or the cache runs out.

B=1→32:throughput×20,latency×1.5B = 1 \to 32 : \text{throughput} \times 20, \quad \text{latency} \times 1.5

That is the whole trade in one line, and it is why an idle server should still batch.

Static batching wastes most of it

Tbatch=max⁡ini×TPOTT_{\mathrm{batch}} = \max_i n_i \times \mathrm{TPOT}

A fixed batch finishes when its slowest member finishes, so a batch containing one 2,000-token answer and thirty 50-token answers holds thirty slots idle for the duration.

utilisation=∑iniBmax⁡ini\text{utilisation} = \frac{\sum_i n_i}{B \max_i n_i}

On realistic length distributions that is often under 30 percent, and the wasted capacity is paid for in full.

Continuous batching

The scheduler works per token rather than per request, so the batch is refilled continuously instead of being drained and refilled.

throughput×2 to 4 over static batching\text{throughput} \times 2 \text{ to } 4 \text{ over static batching}

That is the single largest serving improvement available, and it is why vLLM and similar servers exist rather than a plain loop over requests.

The constraint becomes memory: every active sequence holds a KV cache, so the batch size is limited by how many caches fit rather than by arithmetic.

One request path, in order

  1. The request arrives and joins the waiting queue.
  2. The scheduler admits it when there is KV cache room, not when the batch empties.
  3. Prefill runs for the new sequence, possibly interleaved with other sequences' decode steps.
  4. Decode proceeds one token per step across every active sequence.
  5. The sequence finishes, its cache is freed, and the next waiting request takes the space.

When to choose batch outright

If no user is waiting, choose batch.

Embedding a corpus, classifying a backlog, generating summaries overnight and running an evaluation set are all batch problems dressed as API calls.

batch API≈0.5×price\text{batch API} \approx 0.5 \times \text{price}

Providers price them at a discount because they can schedule the work into their own idle capacity, with a completion window of hours rather than seconds.

The engineering follows: batch work needs idempotent jobs, a result store and a retry policy, and does not need streaming, timeouts tuned to a human, or a warm autoscaled fleet.

Mixing the two on one deployment is the common mistake.

A batch job admitted alongside interactive traffic fills the KV cache and pushes every user's latency up, and the two belong on separate pools.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Why is batch inference cheaper per item than real-time inference?
Concepts these five lines lean on
Percentiles and Quartiles · Data ScienceEpochs, Batches and Iterations
MLOps / LLMOps

22 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close