Chunking in RAG, explained in 5 lines

Module 09 · RAG · 19 concepts

In depth / 15 lines

Chunking, in more detail

Documents are too long to embed whole, so they are cut into chunks.

The chunk is the unit that gets retrieved, so the cut decides what the model can see.

Too large and the chunk covers several topics, diluting its vector and wasting context.

Too small and it loses the surrounding meaning that made it intelligible.

A sentence saying this reduces risk by 40 per cent is useless without knowing what this is.

Five hundred to a thousand tokens is a sensible starting range.

Overlap of ten to twenty per cent stops an answer being severed at a boundary.

Fixed-size splitting on characters is easy and cuts through the middle of sentences.

Recursive splitting tries paragraph, then sentence, then word, which is a better default.

Splitting on structure is better still, since headings mark real topic boundaries.

Semantic chunking cuts where the meaning shifts, measured by embedding distance.

Code, tables and legal clauses each need their own rules, since a function should not be split.

Small-to-big retrieval embeds a small precise chunk and returns its larger parent.

That gets precision in the search and context in the answer.

There is no universal best setting, so test a few against real questions and measure.

Deep dive

What is underneath Chunking

The unit of retrieval is the unit of failure

A retriever returns chunks, not documents, so the chunk is what the model gets to read and what it has to answer from.

n_{\mathrm{chunks}} = \frac{L_{\mathrm{doc}}}{S - O}

$L_{\mathrm{doc}}$ : the document length in tokens.
$S$ : the chunk size.
$O$ : the overlap between consecutive chunks.

Too small and a chunk arrives without the context that makes it mean anything. Too large and the embedding averages several topics into one vector that matches none of them well.

S \approx 200 \text{ to } 500 \text{ tokens}

That range is the usual compromise, and it exists because embedding models were trained on passages of roughly that length.

Why overlap is not optional

A fixed cut lands in the middle of a sentence, a table or an argument, and whichever half holds the answer may not hold the words the query matches.

O \approx 0.1 S \text{ to } 0.2 S

An overlap of ten to twenty percent means any span shorter than the overlap appears whole in at least one chunk.

\text{storage} \times \frac{S}{S - O}

The cost is duplication: 20 percent overlap stores 25 percent more vectors and returns near-duplicate results that then have to be deduplicated.

Splitting on structure rather than on length

Split on the largest natural boundary first, such as a section heading.
If a piece is still too long, split it on paragraphs.
If still too long, split on sentences.
Only then split on tokens, which is the last resort.
Merge any piece far below the target into its neighbour.

That cascade is what recursive splitting does, and it beats fixed-size cutting on every document that has structure.

Code, tables and transcripts each want their own rule. A function cut in half retrieves badly, and a table separated from its header is unreadable to the model.

The budget it has to fit

k \cdot S \leq B_{\mathrm{context}} - L_{\mathrm{prompt}} - L_{\mathrm{answer}}

$k$ : how many chunks are retrieved.
$B_{\mathrm{context}}$ : the model's context window.

Chunk size and the number of chunks trade directly against each other under a fixed budget, so the real decision is $k \cdot S$ rather than either alone.

Many small chunks give better recall and more duplication. Few large ones give more context per hit and waste tokens on irrelevant text around the answer.

The fix that removes most of the trade

Embed the small chunk and return the large one.

Parent document retrieval indexes fine-grained chunks for matching and hands the model the larger passage that contains the winner.

Contextual retrieval goes further and writes a sentence of document context onto each chunk before embedding it.

That repairs the dangling pronoun and the missing subject that a mid-document chunk always has.

Both accept extra work at indexing time to avoid a decision that cannot be made well at query time, and both are worth more than tuning $S$ .

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What goes wrong when chunks are too small?

RAG

19 concepts in this module, then a graded test with every answer explained.

Study this module →