If a model can read a million tokens, why retrieve at all instead of pasting everything in?
Cost is the first answer: you pay for every token on every request, and retrieval sends far fewer.
Latency is the second, since a long prompt takes noticeably longer before the first word appears.
Accuracy is the third, because relevant facts buried in a huge context get missed.
Long context wins when the corpus is small and stable; retrieval wins when it is large or changing.
Context windows grew from a few thousand tokens to hundreds of thousands in a couple of years.
That prompted a fair question: is retrieval an obsolete workaround for a limit that has gone away?
The honest answer is that it depends on the size and the volatility of the corpus.
Start with what a long context actually costs, because it is charged on every single request.
Sending two hundred thousand tokens per question is orders of magnitude more expensive than sending two thousand.
Prompt caching reduces that considerably when the same context is reused, but does not remove it.
Latency follows the same shape, since the prefill pass over a long prompt is real work.
Accuracy is the least intuitive factor and often the deciding one.
Models attend unevenly across a long context, and material in the middle is recalled least reliably.
A single relevant sentence surrounded by a hundred thousand irrelevant ones is easy to miss.
Retrieval sidesteps that by putting only the relevant passages in front of the model.
It also gives you citations, since you know exactly which documents were supplied.
And it updates instantly: reindex one document and the next answer reflects it.
Long context wins when the corpus is small, self-contained and read whole, such as one contract.
The two combine well in practice: retrieve generously, then let a long window read all of it.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.