Cost Optimization in MLOps, explained in 5 lines

Module 14 · MLOps / LLMOps · 22 concepts

In depth / 15 lines

Cost Optimization, in more detail

Cost optimisation keeps spend proportionate to the value delivered.

Model bills grow quietly, since each call is small and there are many.

The first lever is model choice, since capability and price differ by an order of magnitude.

Use the smallest model that clears the quality bar for each task.

Classification and extraction rarely need a frontier model.

Routing simple requests to a small model and hard ones to a large one is the standard shape.

The second lever is prompt length, since input tokens are billed on every call.

A bloated system prompt is a recurring charge on every request forever.

Trim examples that are no longer earning their place.

The third lever is caching, both exact and semantic, plus provider prompt caching.

The fourth is output length, since output tokens usually cost several times more than input.

Ask for shorter answers and set a maximum.

Batch requests where latency allows, since batch pricing is often substantially cheaper.

Track cost per request as a monitored metric so a regression is visible immediately.

Attribute cost per feature and per customer, or you cannot tell what is worth keeping.

Deep dive

What is underneath Cost Optimization

The unit is a token, and the two sides are priced differently

C = \frac{n_{\mathrm{in}}}{10^{6}} p_{\mathrm{in}} + \frac{n_{\mathrm{out}}}{10^{6}} p_{\mathrm{out}}

$p_{\mathrm{in}}, p_{\mathrm{out}}$ : the price per million input and output tokens.

Output is typically three to five times the price of input, because it is produced one sequential pass at a time while input is processed in one batch.

C_{\mathrm{monthly}} = C \times \text{requests per day} \times 30

The per-request cost is never the interesting number. A tenth of a cent at ten thousand requests a day is three hundred dollars a month, and that is the number to optimise against.

Where the tokens actually are

n_{\mathrm{in}} = n_{\mathrm{system}} + n_{\mathrm{context}} + n_{\mathrm{history}} + n_{\mathrm{query}}

In a retrieval application the retrieved context is usually most of the bill, and in an agent the accumulated history is.

\text{agent} : \sum_{t=1}^{T} |\mathbf{c}_t| \sim O(T^2)

Every agent step re-sends the whole conversation, so a twenty-step task costs on the order of four hundred single-step calls rather than twenty.

Truncating tool output before it enters the history is therefore the highest-value change in an agent, and it usually costs one line.

The four levers, in order of return

Prompt caching, which cuts repeated prefixes to about a tenth of the input price.
A smaller model for the requests that do not need the large one.
Fewer retrieved chunks and shorter tool outputs.
Batch processing for anything that is not interactive, often at half price.

\text{cache read} \approx 0.1 \times p_{\mathrm{in}}

The first is free in engineering terms as long as the prompt is ordered with the stable part first, and it is the one most systems leave on the table.

Routing, made concrete

C_{\mathrm{routed}} = f \, C_{\mathrm{small}} + (1 - f)\big(C_{\mathrm{small}} + C_{\mathrm{large}}\big)

$f$ : the fraction handled by the small model alone.

If the router itself is a model call, its cost is paid on every request including the ones that escalate, which is why a classifier or a rule is often better than a model.

f = 0.7, \; C_{\mathrm{large}} = 20 \, C_{\mathrm{small}} \implies \text{about } 30\% \text{ of the original bill}

The saving is real and it is bounded by the escalation rate, so measuring $f$ honestly on production traffic comes before building the router.

What to measure

Measure cost per successful request.

Cost per call is the wrong denominator. A cheaper model that fails a fifth of the time and triggers a retry with a larger one is not cheaper.

Track it per user, per feature and per day.

Attributing spend to a feature is what makes the decision possible, and it needs the token counts logged with the request rather than reconstructed from a monthly invoice.

The last lever is the one people reach for first: changing provider.

It moves the constant and leaves the structure alone, and a system sending ten thousand redundant tokens per request will be expensive on any provider.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What is usually the first lever for cutting model spend?

MLOps / LLMOps

22 concepts in this module, then a graded test with every answer explained.

Study this module →