TL;DR — Most RAG evaluation harnesses define 'correct retrieval' as matching the chunk that produced the gold answer — but that gold chunk was carved out by the same chunking scheme you're supposed to be testing. This makes chunking failures invisible: the eval can only detect wrong retrieval, never a right retrieval of a broken chunk. The fix is decoupling gold answer spans from chunk boundaries and stress-testing the chunker directly, before embeddings or re-rankers ever enter the picture.
Every RAG postmortem eventually blames the same three suspects: the embedding model, the re-ranker, or the prompt. Almost none of them blame the chunker. That's not because chunking is fine. It's because most eval setups are structurally incapable of catching chunking failures — they're built in a way that makes chunking invisible by construction.
Here's the loop nobody notices they're in. You build a gold evaluation set by taking a document, chunking it, and asking a human or a model to write a question whose answer lives in one of those chunks. Then you evaluate retrieval by checking whether the retriever surfaces that same chunk. Context precision, context recall, hit rate at k — all of it measures whether you got back the chunk the gold set says you should get back. But the gold set was defined after chunking, using the chunk boundaries as the unit of truth. You are testing whether your retriever agrees with your chunker. You are not testing whether your chunker preserved the information the retriever needed to find.
Chunking Is a Modeling Decision Wearing a Preprocessing Costume
Teams treat chunking as plumbing — pick a size, pick an overlap, ship it. But a chunk boundary is a claim about semantic independence. Every time you split text, you're asserting "everything needed to answer a question about this passage is contained within these N tokens." That assertion is usually false. Definitions get separated from the terms they define. A caveat in paragraph four gets orphaned from the claim it qualifies in paragraph one. A table's header row ends up in a different chunk than its data rows.
None of this shows up in a standard eval because the eval's gold answer was extracted from a chunk that, by definition, already contained the answer. You never generated a gold question whose answer was split across two chunks — because the workflow that builds gold sets naturally selects for chunks that already look "complete." The eval set is a survivorship-biased sample of your chunking scheme's successes.
Why Better Embeddings and Re-Rankers Can't Rescue This
This is the part that trips up senior engineers, because it looks like a retrieval-quality problem and gets treated as one. If context precision is low, the instinct is to swap embedding models, add a re-ranker, tune the retrieval k. Sometimes that helps — a re-ranker can push a marginally relevant chunk higher, and a better embedding model can find semantically related chunks a keyword search would miss. But neither can put back information that a chunk boundary already deleted.
If the answer to "what are the exceptions to this policy" requires the policy statement in chunk 12 and the exceptions list in chunk 14, no re-ranker fixes that by ranking chunk 14 highly. You needed both chunks retrieved and stitched with enough context that the model understands they're connected. Re-ranking optimizes ordering. It does nothing about completeness. And completeness is exactly the property that chunking either preserves or destroys, upstream of every other component in the stack.
The Diagnostic Your Eval Suite Is Missing
If you want to know whether your chunker is the problem, you have to break the circularity deliberately. That means building a small, separate evaluation set where gold answers are written from the source document, before chunking happens, by someone who has no idea how the text will be split. Then run the chunker, embed, retrieve, and check something more specific than "did we get the right chunk": did the union of retrieved chunks contain every fact required to answer the question, and was each fact retrievable independently of the others being adjacent in the original text.
This surfaces a failure category that standard RAG metrics don't have a name for: partial-answer retrieval. The retriever did its job — it found something relevant — but what it found is a fragment, and the generator either hallucinates the missing piece or, worse, confidently answers with only half the truth and no acknowledgment that anything is missing. Context recall metrics that operate at the chunk level will happily score this as a hit if the fragment happens to overlap with the gold chunk. They were never designed to ask whether the fragment was sufficient.
Stress-Testing Boundaries Instead of Trusting Them
A more useful diagnostic than any aggregate retrieval score is a boundary stress test: take your actual production chunking scheme and run it against documents with known internal dependencies — numbered lists with cross-references, tables with separated headers, contracts with defined terms used pages later, code with function definitions far from their call sites. Measure, mechanically, how often a dependency pair ends up split across chunks with no shared retrieval anchor. This number will usually be uncomfortably high, and it will not correlate cleanly with your existing eval scores, because your existing eval scores were never measuring this.
Once you have that number, the fix set is narrow and well understood: semantic or structure-aware chunking that respects document hierarchy instead of fixed token windows, contextual chunk augmentation where each chunk carries a summary of its parent section, or retrieval strategies that fetch chunk neighborhoods rather than isolated chunks. None of these are exotic. What's missing in most stacks isn't the technique — it's the measurement that tells you which technique you actually need.
Treat the Gold Set as a Liability, Not an Asset
The uncomfortable conclusion is that a RAG eval set built on top of your own chunking scheme has a shelf life measured in chunker changes. Every time you adjust chunk size, overlap, or splitting logic, the gold set that was "aligned" to the old chunks may no longer represent the failure modes of the new ones — and worse, it will still report a healthy score because it was never independent of the chunker in the first place. Evals that are entangled with the artifact they're supposed to evaluate don't decay gracefully; they decay silently, reporting green while the system underneath them changes shape.
Decoupling gold answers from chunk boundaries costs more upfront — you need source-level annotation instead of chunk-level annotation, and it's slower to build. But it's the only way to get a signal that actually distinguishes "the retriever is weak" from "the chunker already threw away what the retriever needed." Those are different bugs, with different fixes, and right now most RAG teams are debugging the wrong one because their eval can't tell the two apart.
Top comments (0)