This article is Part 4 of my RAG in Practice series, where I explain retrieval-augmented generation in practical, production-oriented terms.
In this part, we continue the RAG in Practice series.
TechNova is a fictional company used as a running example throughout this series.
Two Decisions Before the Model Sees Anything
Two decisions shape every answer before the model sees the question. They are how you split your documents, and how you search the pieces. Part 3 walked through the whole pipeline. This article zooms in on those two decisions, chunking and retrieval, and on how to tell whether a change made them better or worse. That check is what this article calls evaluation. You keep a set of real test questions, each paired with the chunk or chunks needed to answer it, and rerun them after every change to see whether the right information still comes back. The last section shows how.
Most tutorials pick a chunk size, often 512 tokens, and move on. That works when every document looks the same. TechNova's don't. Different document structures need different boundaries.
| Document | What it looks like | What must stay together |
|---|---|---|
firmware-changelog.md |
Short entries, one per version | Each version entry |
troubleshooting-guide.md |
Numbered steps under a heading | The whole procedure |
product-specs.html |
A comparison table | Each row with its column headers |
Those links open the source files on GitHub. There is nothing you need to download.
A fixed-size rule ignores those boundaries. It can mix two firmware versions in one chunk or split a procedure in two. Tables need care even earlier, at parsing (Part 3).
We'll follow one customer question about TechNova's WH-1000 wireless headphones through the whole article.
"My WH-1000 doesn't show up in Bluetooth. What do I do?"
The answer is in the pairing section of TechNova's troubleshooting guide, and it needs step 4, the reset.
Chunking: Split by Structure, Not by Size
Chunk boundaries should follow the document's structure, not a token count. Part 3 showed the pairing section cut between step 3 and step 4. Here it is again.
Pairing the WH-1000 with a phone or laptop
1. Charge the headphones for at least 10 minutes.
2. With the headphones off, hold the power button for 5 seconds,
until the light flashes blue.
3. Open Bluetooth settings on your device and select WH-1000.
---------------------- chunk boundary ----------------------
4. Not listed? Reset the headphones. Hold the power and
noise-cancellation buttons together for 10 seconds, then
repeat step 2.
5. When you hear "Connected", the light turns solid blue.
That cut comes from fixed-size chunking. A fixed-size chunker counts tokens from the top of the guide and cuts every 512. If the guide's first 512 tokens happen to end just after step 3, the cut goes there. Nothing in the content decides it. The customer's question needs step 4, and step 4 is now in a different chunk from the heading and the first three steps.
Three Ways to Split
| Strategy | How it splits | What happens to the pairing section | Use it for |
|---|---|---|---|
| Fixed-size | Every N tokens, wherever the count lands | Cut wherever the 512th token falls, here between step 3 and step 4 | Quick prototypes, and plain text with no structure |
| Recursive | By headings first, then paragraphs, then sentences. It only goes smaller when a piece is still too long. | The whole section stays in one chunk, because it's about 90 tokens | The default for most documents |
| Semantic | Compares each sentence with the next and cuts where the meaning shifts | Usually keeps the steps together, but a step that changes topic, like the reset, can start a new chunk | Long text with no headings, such as call transcripts |
Fixed-size chunking is still worth knowing. Many tutorials use it, it's simple and predictable, and it's a useful baseline to measure smarter splitting against. Recursive chunking uses a size limit too. It just follows the document's structure first and only cuts by size when a piece is still too long.
Recursive chunking is the default because most documents already mark their own boundaries. Headings, paragraphs and numbered lists tell the chunker where one idea ends. The pairing section is short enough to stay whole, so the customer's question finds one chunk with all five steps. In this article that chunk is troubleshooting#pairing. The same rule splits the firmware changelog into one chunk per version entry. A fixed-size chunker would cut through entries and mix two versions in one chunk.
Semantic chunking is something to try later, not where to start. It costs more and is harder to control. To find where the meaning shifts, it compares nearby pieces of text using embeddings, which adds work during ingestion. Its cut points come from the embedding model, so if you switch models, your chunks can change too. And when it cuts in the wrong place, there is no heading or numbered step to show you why. Reach for it when a document has no headings or lists to follow, such as a call transcript, and your test questions show recursive chunking is still separating information that belongs together.
Here is the turning point of a long support call. Like most transcripts, it has no headings.
Customer: My WH-1000 keeps disconnecting after it goes to sleep.
Agent: Hold the power and noise-cancellation buttons for 10 seconds.
Customer: That fixed it.
----------- semantic cut: the topic changes -----------
Customer: If it happens again, can I still return them?
Agent: New products can be returned within 15 days of delivery.
Recursive chunking finds no headings or numbered steps here, so it falls back to sentences or size. Its cut can land at an arbitrary sentence or size boundary, even between the return question and its answer. Semantic chunking cuts where the call moves from troubleshooting to returns, so each topic gets its own chunk.
None of this helps if parsing already destroyed the structure. A table flattened into one line of text can't be rescued by any chunking strategy (Part 3).
What Overlap Fixes, and What It Doesn't
Overlap repeats a sentence or two from the end of one chunk at the start of the next. Here is what it does to the fixed-size cut.
Chunk 1 heading, step 1, step 2, step 3
Chunk 2 step 3 (repeated), step 4, step 5
Chunk 2 now starts with some context. Chunk 1 still ends at step 3, with no reset. If the question retrieves chunk 1, the answer is just as incomplete as before.
Overlap helps prose, where a sentence leans on the one before it. It doesn't put a procedure back together. For procedures, the fix is recursive splitting. Overlap also has costs. It takes more storage, and two overlapping chunks can both be retrieved, filling the top results with near-duplicates.
Start Simple, Then Climb
Chunking is a ladder, not a menu. Parse well, default to recursive, add overlap for prose, and try semantic chunking only when evaluation shows you need it. Stop as soon as retrieval is good enough.
A vague answer like "The return policy may vary depending on the product" can start with chunks that are too broad, or split so that the specific fact was diluted. The symptom shows up in the answer, but the cause may be upstream in chunking. Part 7 shows how to trace the failure to the layer where it first appears.
Two more patterns help when chunks lose the meaning of their section. Contextual enrichment adds a line of context to each chunk before it's embedded (Part 3). Parent-child chunking searches small chunks but sends the model the larger section they came from (Patterns).
Late Chunking: Same Chunks, More Context
Late chunking doesn't replace recursive chunking. It works on top of it. Recursive chunking decides where the cuts go. Late chunking keeps those cuts and changes how each chunk is turned into a vector.
Late chunking is another way to give chunks context, and it isn't automatically better than standard chunking.
When it's worth trying. Some chunks only make sense with what came before them. A paragraph that says "these items" or "this window", a section whose topic was set by a heading much earlier in the document, or an answer deep in a long FAQ. If many of your chunks look like that, and asking an LLM to write context for each one would be too slow or too costly at your volume, late chunking can be a more efficient way to get some of that context into the vectors. If adding the document title and section heading to each chunk already fixes the problem, as it does for TechNova's Outlet chunk in Part 3, you don't need late chunking.
Take a long section of TechNova's headphones FAQ, under the heading "Battery and charging". An earlier paragraph introduces the WH-500, and recursive chunking leaves a later paragraph as its own chunk.
Headphones FAQ > Battery and charging
The WH-500 is TechNova's sport model, with a 30mm driver
and support for SBC and AAC.
...
---------------------- chunk boundary ----------------------
Its battery lasts 8 hours on a single charge.
The last chunk never says "WH-500". Embedded on its own, its vector means "some headphones have an 8-hour battery". A question about the WH-500's battery can match the WH-1000's battery chunk just as well, the same confusion you'll see in Vector Search below. Adding the title and heading wouldn't help here, because they say "Headphones FAQ" and "Battery and charging", not "WH-500".
Late chunking fixes the vector, not the text. To see how, look at how an embedding model makes a vector.
- One vector per token. The model reads its input and produces a vector for every token. Each token's vector is shaped by the tokens around it.
- Combined into one. The model then combines those token vectors, often by averaging, into one vector for the whole input.
- The input sets the context. With standard chunking, each chunk is a separate input. The battery chunk goes in alone, so its tokens, including "Its", never see the WH-500 paragraph.
Late chunking changes the input. The chunk boundaries are decided first, as usual. Then the embedding model reads the whole FAQ section in one pass, so the tokens of "Its battery lasts 8 hours" are shaped by the WH-500 paragraph above them. Finally, the token vectors inside each chunk's boundaries are averaged into that chunk's vector. The chunk's text hasn't changed. Its vector was made with the WH-500 paragraph in view.
| What the chunk's vector reflects | What the LLM reads at answer time | |
|---|---|---|
| Standard chunking | Only "Its battery lasts 8 hours". Nothing says WH-500. | The bare chunk |
| Late chunking | The chunk, read together with the WH-500 text before it | The bare chunk |
| Enrichment written by an LLM (Part 3) | The chunk plus an added line naming the WH-500 | The chunk plus the added line |
So why not use late chunking everywhere? It sounds strictly better, but it isn't.
- Extra context can blur a chunk. Each vector now carries some of the whole document, so it can be less specific to what this chunk actually says.
- The document has to fit the model's maximum input length. Longer documents are processed in windows, and context doesn't cross from one window to the next.
- You need the token vectors. The embedding model and library have to return a vector per token, not only the final combined one.
- The LLM doesn't see that extra context directly. As the table shows, only the vector gets the context. Enrichment changes the text, so it helps search and the model reading the chunk.
A 2025 comparison by Merola and Singh found the same trade-off. Contextual retrieval, which is enrichment written by an LLM, kept meaning together better but cost more compute. Late chunking was cheaper but lost some relevance and completeness. Treat late chunking as one more technique to test when chunks lose their document's context, not as the next step up the chunking ladder. Try it once your baseline works and you have an evaluation set to show whether it helps.
Retrieval: Keyword, Semantic, or Hybrid
Keyword search matches words. Vector search matches meaning. Each one misses what the other catches. Chunking decides what can be found. Retrieval decides how it's searched.
Keyword Search (BM25)
BM25 scores each chunk by the question's words it contains, and gives more weight to words that are rare across all your chunks. It needs no embedding model and it's fast. "WH-1000 return policy" finds the return policy right away.
It fails when the customer uses different words from the document. "Can I send my headphones back?" shares almost no words with "Return window for new products", so BM25 ranks the policy low.
Vector Search
Vector search compares meaning, so "Can I send my headphones back?" lands next to the return policy even with no shared words (Part 3).
Its weakness is the opposite one. "WH-1000 battery life" and "WH-500 battery life", for TechNova's cheaper headphones, can embed almost on top of each other, because the model reads both as "headphone battery life". If it doesn't separate the two product codes, it can return the wrong product's chunk.
Hybrid Search and Reciprocal Rank Fusion
Hybrid search runs both searches and merges the two ranked lists. The usual merge is Reciprocal Rank Fusion (RRF). It ignores the raw scores, which can't be compared between the two methods, and uses only rank positions. Each chunk gets 1 / (60 + rank) from each list, and the two are added. The 60 is a common default that keeps any single first place from dominating.
Here is the pairing question.
| Chunk | BM25 rank | Vector rank | RRF score |
|---|---|---|---|
wh1000#overview (specs, mentions Bluetooth 5.3) |
1 | 2 | 1/61 + 1/62 = 0.0325 |
troubleshooting#pairing |
2 | 4 | 1/62 + 1/64 = 0.0318 |
firmware#v3.1 (faster reconnection) |
6 | 1 | 1/66 + 1/61 = 0.0315 |
warranty#claims |
9 | 3 | 1/69 + 1/63 = 0.0304 |
Vector search alone put the pairing chunk 4th, just below a top-3 cut. The firmware note about faster reconnection sounds close in meaning, but it never says "Bluetooth". Keyword search put the pairing chunk 2nd, because it contains both "WH-1000" and "Bluetooth". Combined, the pairing chunk moves up to 2nd and makes the top 3.
Hybrid search costs a keyword index alongside the vector index, and two searches per question. Many vector databases now support keyword and vector search together. It's a common production choice when you need both exact-term and semantic matching.
One Question, Three Configurations
The same model gives three different answers depending on how the documents were split and searched. This table puts the earlier examples together. Each configuration keeps the top 3 chunks. Row A uses the fixed-size split from the pairing example and illustrates what happens when retrieval keeps the first half of the procedure but not the second. Rows B and C reuse the illustrative ranking table above. B takes the vector-search ranking. C takes the combined RRF ranking. These are teaching examples, not results from a benchmark run.
| Chunking | Search | What the model gets | Answer | |
|---|---|---|---|---|
| A | Fixed-size, 512 tokens | Vector | The chunk with steps 1 to 3. The chunk containing the reset is not in the top 3. | Repeats the steps the customer already tried. The reset is missing. |
| B | Recursive | Vector |
firmware#v3.1, wh1000#overview, warranty#claims. The pairing chunk is 4th and not kept. |
"Update to firmware 3.1, which improves reconnection." The wrong fix. |
| C | Recursive | Hybrid with RRF |
wh1000#overview, troubleshooting#pairing, firmware#v3.1
|
The full procedure, including the reset |
The model was the same every time. What changed was what reached it.
Reranking: Fix the Order, Not the Search
A reranker reorders the candidates that search already found. It doesn't find new ones. In configuration C, with hybrid search, the spec chunk wh1000#overview is first and the procedure is second. The spec chunk mentions Bluetooth, but it doesn't answer the question.
A reranker is a model that reads the question and one candidate together and scores how well that candidate answers it. Reading them together lets it judge whether the chunk actually answers the question, not just whether they are similar in meaning. Vector search compares a query vector with chunk vectors made separately. Scoring each pair at question time is slower, so a reranker usually runs only on the shortlist that search returns. For the pairing question, it puts troubleshooting#pairing first, because only that chunk tells the customer what to do when the headphones don't appear.
The order matters even when the right chunk is already kept. In many systems, the lowest-ranked chunk is the first one cut at assembly when the prompt runs out of room (Part 3).
Add a reranker when the right chunk is found but ranked too low. In Part 7's terms, that's recall@10 high and recall@3 low.
You decide for the system, not one question at a time. The simplest setup runs the reranker on every question. The decision to add it comes from your evaluation set. Look at where each needed chunk lands. If many needed chunks appear in the top 10 but fall below the top 3, ranking is likely the problem, and a reranker is worth testing. Add it, run the same questions again, and keep it if recall@3 improves without breaking questions that used to pass, and the extra latency is acceptable. Later, if cost or latency matters, you can decide whether to rerank only some queries. That's an optimization, not where to start.
A reranker isn't the only fix when the right chunk is found but ranked too low. Each option has a cost.
| Fix | Cost |
|---|---|
| Add a reranker | An additional ranking model and extra time on every question |
| Keep more chunks | More tokens in every prompt, and more room for noise |
| Add hybrid search | A keyword index to build and keep in sync |
Measure Before You Change Anything
If you can't measure retrieval, you can't tell whether a change helped.
A team swaps its general-purpose embedding model for a domain-specific one, expecting retrieval to improve. They redeploy, and customer satisfaction drops. It takes two weeks to trace the problem. The new model embeds TechNova's product codes differently, and WH-1000 questions now sometimes retrieve WH-500 content. Nobody measured before or after.
Every decision in this article is an experiment. Start with two retrieval metrics, which Part 7 defines in full.
-
Recall@k. Of the chunks a question needs, how many are in the top k retrieved chunks? The pairing question needs one chunk,
troubleshooting#pairing. With vector search alone, it ranked 4th, so recall@3 is 0%. With hybrid search, it ranked 2nd, so recall@3 is 100%. - Precision@k. Of the top k retrieved chunks, how many were needed? With hybrid search, the top 3 held the pairing chunk and two chunks the question didn't need, so precision@3 is 1 of 3, or 33%. Low precision with full recall may be acceptable, but extra chunks add noise and consume prompt space.
Start with about 20 real questions, each with the chunks it needs. Run them before and after every change, and change one thing at a time. If you change chunking and search together and the numbers move, you can't tell which change did it. Look at which questions changed, not only the average. A change can raise the average while breaking a question that used to pass.
The same goes for metadata filters from Part 3, such as restricting results to one product or policy version. A filter that wrongly removes the needed chunk shows up here as a drop in recall, so test a new filter like any other change.
Three Takeaways
- Split by structure, not by size. Recursive chunking keeps sections and procedures whole when they fit the size limit. Add overlap or semantic chunking only when evaluation shows you need to.
- Keyword and vector search miss different things. Hybrid search catches both kinds of match. Add a reranker when the right chunk is found but ranked too low.
- Measure before and after every change, one change at a time. Part 7 shows how.
Next, Part 5 builds a working RAG system against real documents, using the decisions from this article.
References & artifacts
- RAG in Practice examples repository
- Carlo Merola and Jaspinder Singh, Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation (arXiv, 2025)
Part of the RAG in Practice series, originally published on AI in Practice Hub alongside the full series and companion code.
Top comments (2)
This is one of the clearest treatments of chunking trade-offs I've seen. The three-configuration example comparing the same question against different chunking/retrieval setups is exactly the kind of concrete demonstration that abstract explanations miss.
A few things worth expanding on for practitioners:
On late chunking: The compute cost tradeoff deserves more attention. Late chunking requires running the full-document embedding pass before splitting, which for long documents means holding the full KV cache during the embedding forward pass. For a 50-page PDF that's tractable; for document collections at scale it changes your ingestion architecture significantly. The coherence gains are real — pronoun resolution and cross-reference preservation genuinely improve retrieval on procedural documents — but batching strategy matters a lot for throughput.
On rerankers: Cross-encoders like BGE-Reranker or Cohere Rerank are powerful, but there's an underused middle path: ColBERT-style late interaction models. Instead of a single similarity score, ColBERT computes token-level MaxSim interactions, which preserves more fine-grained relevance signal than a bi-encoder while being faster than a full cross-encoder. For technical documentation where precise term matching matters (product codes, version numbers, procedure steps), it often outperforms both.
On the evaluation point: The 20-50 query evaluation set you recommend is right, but the construction of that set is where teams go wrong. If you build the evaluation set from the same documents you're chunking, you're measuring retrieval against a distribution the system was already optimized for. Ideally, you want adversarial queries — paraphrases, synonym-based queries, cross-section queries that require combining two chunks — alongside the straightforward ones. The BM25 precision / embedding recall trade-off shows up most dramatically on adversarial queries.
On metadata filtering: Worth mentioning that metadata filters interact with chunk overlap in a subtle way. If you filter by product_id before retrieval, overlapping chunks that span a product boundary (e.g., a comparison section) may be excluded entirely even though they contain the answer. Document-level filtering is usually safer than chunk-level filtering for documents with cross-cutting structure.
Thank you, Vikrant. Really appreciate the thoughtful points.
I agree, especially on late chunking cost and metadata filtering. I kept those at a high level in this article, but your comment adds very useful real-world detail.
The evaluation-set point is also important. If the test queries are too friendly, they can give false confidence. Adversarial and cross-section questions make the evaluation much more meaningful.
Thanks again for reading and adding this context.