Latency Optimization in MLOps, explained in 5 lines

Module 14 · MLOps / LLMOps · 22 concepts

In depth / 15 lines

Latency Optimization, in more detail

Latency is how long a user waits, and it shapes whether a feature feels usable.

For language models the largest single win is streaming.

Tokens appear as they are generated rather than after the whole answer is done.

The total time is unchanged and the experience is transformed.

Time to first token is what users actually feel, so measure it separately.

Output length matters far more than input length.

Prefill processes the prompt in parallel, and decoding is strictly sequential.

Halving the output roughly halves the wait, and halving the input barely moves it.

So ask for concision when speed matters.

Smaller models are faster, and routing simple work to them cuts the average sharply.

Caching removes the wait entirely on a hit.

Run independent calls in parallel rather than in sequence.

Network placement matters, since a distant region adds real milliseconds.

Measure p95 and p99, because the average hides the experiences people complain about.

Show progress for anything slow, since a visible wait is tolerated far better than a blank screen.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Latency Optimization

Decompose before optimising

T = T_{\mathrm{network}} + T_{\mathrm{queue}} + \mathrm{TTFT} + \mathrm{TPOT} \times (n - 1)

$\mathrm{TTFT}$ : time to first token, which is prefill.
$\mathrm{TPOT}$ : time per output token, which is decode.
$n$ : the number of tokens generated.

Each term has a different fix, and optimising the wrong one is the usual reason a week of work changes nothing.

n = 500, \; \mathrm{TPOT} = 20\,\mathrm{ms} \implies 10 \text{ seconds of decode}

For most model applications that last term dominates everything else, which means the shortest path to lower latency is generating fewer tokens.

Attack the output length first

T \propto n

Asking for a summary in three sentences rather than a page halves the latency directly, at no engineering cost.

Removing explanation the caller does not read is the same saving. So is stopping at a stop sequence rather than letting the model run to its limit.

Streaming does not reduce $T$ at all. It changes the perceived latency to $\mathrm{TTFT}$ , which is usually the more valuable number.

Then the queue, then the model

W = \frac{L}{\lambda}

Queue time is invisible in a single-request benchmark and is often the largest term under load, so it has to be measured in production rather than on a laptop.

\mathrm{TPOT} \approx \frac{\text{bytes of weights read}}{\text{memory bandwidth}}

Decode is memory-bound, so a smaller or quantised model is faster almost in proportion to its size, while a faster chip with the same bandwidth is not.

\text{4-bit} \implies \approx 3\times \text{ faster decode}

That is why quantisation is the most reliable single change for generation speed, and why it does far less for prefill.

One optimisation, in order

Measure the four terms separately, at the percentiles users feel.
Cut the output length and the prompt length, in that order.
Stream, so the first token arrives while the rest is produced.
Cache whatever repeats, including the system prompt through prompt caching.
Only then change the model, the hardware or the serving stack.

The structural wins

T_{\mathrm{parallel}} = \max_i T_i, \qquad T_{\mathrm{serial}} = \sum_i T_i

Independent calls should run concurrently. A pipeline that retrieves, reranks and classifies in sequence when two of those do not depend on each other is paying twice.

\text{speculative decoding} : \approx 2 \text{ to } 3\times

A small model drafts several tokens and the large one verifies them in a single pass, which raises the tokens produced per weight read.

Sending easy requests to a small model and hard ones to a large model lowers the average without lowering the ceiling, and the classifier that decides is usually cheap.

Percentiles are the last point and the most commonly missed. An average of 800 ms with a p99 of 12 seconds is a system that feels broken to one user in a hundred, every day.

Test yourself

2 questions on this card. Every option explains itself once you pick, right or wrong.

Question 1 of 2

Why measure p95 latency rather than average?

Question 2 of 2

Why does shortening outputs cut latency more than shortening inputs?

MLOps / LLMOps

22 concepts in this module, then a graded test with every answer explained.

Study this module →