Learn how document boundaries, passage size, overlap and metadata affect retrieval, with a runnable local experiment.
Originally published on Stack & Craft.
What you will learn: Choose meaningful passage boundaries, explain the trade-offs of size and overlap, and measure whether chunking preserves the evidence a question needs.
Prerequisites: Start with the Building and Evaluating a RAG System guide. Worked exercises use Python’s standard library and need no paid model.
What is a chunk?
A chunk is a piece of source content prepared for retrieval. It may be a paragraph, a section or a bounded passage. It should retain enough context to interpret its facts. Token counts are model-dependent; characters, words and tokens are different units.
Small passages
Focused passages can make matching more precise, but a sentence may lose its subject, exception or qualification when separated from surrounding text.
“Within 14 days” is ambiguous without knowing which plan and event it refers to.
Large passages
Longer passages retain more context but may mix unrelated rules and consume more of the generation request. More text is not automatically more useful evidence.
A whole handbook may mention refunds, payroll and security in one retrieval unit.
Overlap
Overlap repeats some boundary text in adjacent chunks. It can help preserve a split idea, while also creating duplicated evidence and extra indexing cost.
Two retrieved passages may look like independent support but repeat the same paragraph.
A boundary can change the answer
Consider a fictional policy: “Annual plans: refund requests must be submitted within 14 days. Monthly plans: payments are non-refundable.” Splitting after the first colon could detach the plan type from its deadline. Flattening a table can similarly associate a number with the wrong row.
Preserve the relationship
Keep the annual-plan label with its 14-day rule. Preserve the monthly-plan exception in a separately traceable passage. If a table is converted to text, repeat the relevant row and column headings so each value has meaning.
Build a traceable passage
Boundary choice is only one part of ingestion. Extraction must preserve readable content before chunking begins. An OCR error or missing table column cannot be repaired just by changing the chunk size.
From document to usable evidence
- Extract structure — Retain headings, paragraphs, lists and table relationships. Validate scans and unusual layouts.
- Choose natural boundaries — Prefer sections and paragraphs before imposing a size limit. Handle long paragraphs explicitly.
- Attach identity — Keep document ID, section or page, version, and access metadata beside the passage.
- Retrieve and inspect — Ask representative questions. Check that each selected passage preserves the subject, fact and relevant exception.
- Measure the change — Hold the question set and retrieval configuration fixed while comparing chunking variants.
Select a starting strategy
There is no universal best chunk size. Start with boundaries that fit your documents and the question types users ask, then compare variants using evidence-level labels.
| Situation | What it means | Next check |
|---|---|---|
| Short FAQ answers | One answer may form a useful retrieval unit. | Keep its question or heading with the answer. |
| Long policy sections | One section may contain several rules. | Split by subheading and inspect exceptions at boundaries. |
| Tables | Cells lose meaning when flattened carelessly. | Preserve row/column labels or use a table-aware representation. |
| Repeated overlap | Many near-duplicate results can crowd out other facts. | Track unique evidence coverage, not only passage count. |
Run the chunk-boundary experiment
Run exercises/chunking_demo.py in the RAG starter. It compares tiny character windows with a complete policy paragraph and prints the actual passages. Its checks verify that the full paragraph preserves both the plan label and deadline. The small example demonstrates a boundary issue; it is not a benchmark for a real corpus.
Record your observation
Inspect the program’s printed chunks. Which passage contains both “Annual plans” and “14 days”? Does the smallest passage contain the full rule? Compare two sizes on your own source and keep the question set unchanged.
Check freshness and deletions
When a source changes, the old passages should not remain silently searchable alongside the replacement unless versioned history is intentional. Record the source version and index build configuration.
A larger chunk is not a freshness fix
If a query returns a superseded policy, fix source/version selection and reindexing. Changing passage size does not make an old fact current.
A practical workflow
- Inspect extracted text and table relationships.
- Write questions and the exact evidence each needs.
- Choose structure-aware boundaries and a size limit.
- Compare chunk variants while holding other settings fixed.
Practice
Take a source with a rule and an exception. Make three chunk versions. Mark the smallest passage that preserves the complete rule, then explain when an adjacent passage would also be needed.
Check your understanding
Is 500 characters equal to 500 tokens?
No. Tokenization depends on the model and text. Character limits are a separate engineering choice.
Does overlap always improve retrieval?
No. It can preserve boundary context but also adds duplicates and cost. Measure unique evidence coverage.
Can good chunking recover a missing table column?
No. First fix extraction so the needed information is present.
Sources
- Original RAG research — Lewis et al.: Research background on combining retrieval and generation. Examples and exercises in this guide are original teaching material.
- LangChain: text splitter reference: Primary documentation for further reading. The worked examples here are original teaching illustrations.
Continue learning
Read the original guide on Stack & Craft for the illustrated version and related lessons.
The Local RAG Document Assistant includes the companion exercises.
Top comments (0)