A customer cancels a TechNova order after it ships, because TechNova sent the wrong item. Does the 15% restocking fee apply? The policy is clear enough for a human: Section 7 charges a 15% restocking fee on post-shipment cancellations, and Section 3 waives it when TechNova shipped the wrong item. The correct answer is no.
I put that question to a small retrieval experiment three ways: fixed-size chunks, fixed-size with overlap, and section-aware splitting. Every configuration ranked the Section 7 fee rule highly. None returned the Section 3 waiver condition. The retrieved evidence looked relevant and could not support the correct answer.
That is the failure this article is about. Retrieval relevance is not the same as answer completeness: a chunk can rank highly because it shares keywords and topic with the query while the condition that reverses the answer sits in a chunk that was never returned.
What is chunking?
In many RAG systems, larger documents are split into smaller pieces called chunks before indexing and retrieval. The simplest way is fixed-size chunking: cut the text every N characters or tokens (say every 500), sometimes overlapping a small part of one chunk with the next so some context is preserved. It is cheap and fast, but blind to document structure. With character-based splitting, the cut can fall wherever the count runs out, even in the middle of a word. Fixed-size chunking splits by counting, not by meaning.
Structure-aware chunking follows the shape of the document instead. It tries to keep natural units such as sections, clauses, tables, or code blocks together instead of cutting through them at a fixed count.
Experiment setup
To see how much these boundaries matter, I created a synthetic TechNova billing and cancellation policy with numbered sections, cross-references, lists, and a table. I made the policy myself, so I knew the correct answers and exactly what text was available to the retriever. I froze five questions and the sufficiency criteria before running retrieval.
I compared three configurations:
- A: fixed-size, 500 characters, no overlap
- B: fixed-size, 500 characters, 100-character overlap
- C: section-aware, one chunk per Markdown section (
##heading plus its content), with no 500-character limit
All three used the same embedding model (all-MiniLM-L6-v2), cosine similarity, and a top-k of 3 (the system keeps only the three highest-scoring chunks per question). I did not add BM25, reranking, contextual retrieval, or LLM answer generation, because I wanted to isolate the effect of chunking.
| Strategy | Passed |
|---|---|
| A: fixed-size, no overlap | 4/5 |
| B: fixed-size, 100-char overlap | 2/5 |
| C: section-aware | 4/5 |
| Question | A: Fixed | B: Overlap | C: Section-aware |
|---|---|---|---|
| Q1 — Refund timing | Pass | Pass | Pass |
| Q2 — Fee calculation | Pass | Pass | Pass |
| Q3 — Wrong item / waiver | Fail | Fail | Fail |
| Q4 — Waiver conditions | Pass | Fail | Pass |
| Q5 — Table/header dependency | Pass* | Fail | Pass |
* Fixed-size passed Q5 only because both the table row and its header reached the top three separately; section-aware carried them in one chunk.
This is one synthetic policy, one embedding model, and top-k of 3, so the results are illustrative rather than a benchmark.
On the two straightforward questions (refund timing and fee calculation), every strategy passed. The harder cases appeared when the answer depended on structure: a rule with an exception, a complete list, or a table row that needs its header. And all three strategies failed the same question, Q3. The per-question breakdown is where the real story is.
The frozen run is inspectable in the explorer and reproducible on GitHub.
The most surprising result was Q3
"A customer cancels after shipment because TechNova sent the wrong item. Does the 15% restocking fee apply?"
Every configuration (fixed-size, overlap, and section-aware) failed this question. The failure showed two different problems. Under fixed-size splitting, the boundary damaged the useful text itself: the cut ended one chunk at "However, the restoc", severing the clause that reverses the answer. Under overlap and section-aware splitting, a retrieved chunk even stated that the fee is waived if the order qualifies under Section 3, yet the actual qualifying condition never reached the top-k to complete the answer.
In each case, the retriever returned the Section 7 rule that a post-shipment cancellation incurs a 15% restocking fee, but it did not return the relevant Section 3 waiver condition showing that the fee is waived when TechNova ships the wrong item.
What surprised me most
I expected section-aware splitting to clearly outperform fixed-size chunking, and overlap to perform better than fixed-size. The results were more mixed. Fixed-size and section-aware both passed 4 of 5 questions, while overlap passed 2 of 5.
That made the experiment more useful, because it showed that better chunk boundaries do not automatically mean the retriever will return better results. In the overlap configuration, overlap created more similar chunks, and those chunks competed for the same three retrieval slots. That does not mean overlap is harmful; in this run, it changed the ranking competition enough that the chunk with enough information to answer correctly was still in the index, but it did not make the top three results.
What overlap taught me
Q4 asked: "What conditions make a post-shipment cancellation eligible for a restocking-fee waiver?"
Overlap preserved more text around boundaries, but it also created more near-duplicate chunks. Two of B's top three results were neighboring overlapping chunks with very similar Section 7 text, while the chunk containing the complete waiver list remained outside the top three. Sufficient evidence existed in the index but did not make the top three.
Fixed-size passed the same question, but only by luck: the 500-character cut happened to land inside a section heading, leaving the four-condition list intact. Section-aware passed it by construction. The totals tie at 4 of 5, but the failure modes differ. In this run, fixed-size failed by lottery. Structure-aware failed predictably.
I do not want to conclude that overlap hurts retrieval; only that overlap is not a guaranteed fix.
Production takeaway
My current takeaway is to use fixed-size chunking as a baseline, test it with real questions, and check whether the retrieved chunks contain enough information to answer correctly, not just whether they look relevant.
If arbitrary boundaries repeatedly split rules, exceptions, tables, or other meaningful units, move toward structure-aware splitting. That is not a claim that it scores higher. In this run it did not. It is a claim that it removes one class of failure, the arbitrary cut, while Q3 shows the other classes remain. But if the missing evidence lives in another section and never reaches top-k, that is a retrieval problem, not just a chunking problem.
How do you catch this in production?
You cannot see this failure in the similarity scores. In this run, the highest-ranked chunks for Q3 looked relevant but did not contain enough evidence to support the correct answer. Cosine similarity measures how close a chunk is to the query, not whether the evidence is complete.
The most reliable way to catch this before production is the one this experiment used: a small evaluation set with known expected evidence, frozen before you test. For the queries that matter to your system, write down which text the retriever must return, then check whether the top-k actually contains it. Test for sufficiency, not relevance.
When a query fails, the failure type points to the class of fix:
| Failure type | What happened | Where to look for fixes |
|---|---|---|
| Boundary | Required text was physically split across chunks | Structure-aware splitting, overlap, larger chunks, parent or chunk expansion |
| Ranking | Evidence was in the index but lost the top-k competition | Reranking, larger top-k, hybrid retrieval (exact terms and identifiers), contextual retrieval, fewer redundant chunks |
| Cross-reference | Retrieved text points to a section that never arrives | Linked or metadata retrieval, parent-child retrieval, query expansion, multi-step or graph-style retrieval |
These fixes stack, and none of them guarantees a correct answer. They narrow the ways retrieval can fail silently.
Chunk by structure, not character count.
Originally published at aiinpracticehub.com.
Top comments (0)