Caching stores results so identical work is not repeated.
An exact-match cache keys on the precise input.
A semantic cache matches on embedding similarity instead.
Prompt caching reuses the processed prefix of a long prompt.
It is usually the cheapest large win in both latency and cost.
Caching stores a result so the same work is not done twice.
It is usually the cheapest large win available in both latency and cost.
An exact-match cache keys on the precise input and returns a stored response.
That works well when requests genuinely repeat, as in shared prompts or common questions.
A semantic cache matches on embedding similarity instead of exact text.
How do I reset my password and I forgot my password hit the same entry.
That covers far more traffic and risks returning an answer to a slightly different question.
Set the similarity threshold conservatively and log the near misses.
Prompt caching is offered by providers and reuses the processed prefix of a long prompt.
A long system prompt or a large document then costs a fraction on subsequent calls.
It needs the stable part first and the variable part last, which shapes how you build prompts.
Cache embeddings too, since re-embedding unchanged text is pure waste.
Every cache needs an invalidation story, or it will serve stale answers confidently.
Do not cache anything personalised across users, which is a genuine data leak.
Measure the hit rate, because a cache nobody hits is complexity for nothing.
A cache with a 40 percent hit rate on a request costing 2 seconds and 1 cent turns those into 1.2 seconds and 0.6 cents on average.
The saving is linear in the hit rate, so the whole engineering problem is raising without returning stale or wrong answers.
An exact cache is a dictionary. It is correct by construction, trivially invalidated, and it only fires on repeated identical requests.
A semantic cache embeds the query and returns a stored answer when a close enough one exists, which raises the hit rate and introduces a real risk.
That threshold is a correctness decision, not a tuning knob. Two questions can sit at 0.97 similarity and have different answers, and the cache will confidently return the wrong one.
The attention keys and values for a prefix are stored on the provider's side, so a long shared system prompt is processed once rather than per request.
Reading cached tokens typically costs about a tenth of processing them, and writing costs slightly more than normal input.
The prefix must match exactly, from the first token.
That is the constraint everything else follows from: anything that varies must go after everything that does not, or the cache never hits.
A timestamp at the top of a system prompt is the classic mistake, and it costs the entire discount.
Anything personalised, unless the key includes the user identity, or one person's answer is served to another.
A cached response for a creative request returns the same output every time, which may be exactly what was not wanted.
Time-sensitive answers need a time to live short enough to bound the staleness, and the honest default for anything derived from live data is not to cache it at all.
Memory is finite, so entries leave.
Least recently used is the standard policy and it is a poor fit for a long tail of rare but expensive queries, which is where a cost-aware policy earns its complexity.
One question on this card. Every option explains itself once you pick, right or wrong.
22 concepts in this module, then a graded test with every answer explained.