Hey friends! Welcome back to our ongoing series on building real-world, production-grade Retrieval-Augmented Generation (RAG).
In Part 3, we had an honest conversation about the “infinite context window” myth. We saw why simply giving an LLM hundreds of pages of company documentation isn’t the magical solution it sounds like:
- It drastically increases API costs.
- It drives up query latency.
- It dilutes the model’s attention.
- When too much information is placed into the prompt, models struggle with the “Lost in the Middle” phenomenon — missing facts buried among irrelevant content.
So we arrived at a much more practical conclusion:
Don’t give the LLM everything. Give it only the specific information it actually needs.
Sounds simple, right?
Well… this is where things get interesting. Because before we can retrieve the right information, we first have to decide: what exactly are we retrieving?
A 500-page employee handbook isn’t stored as one giant text block. We have to break it down into smaller pieces. Those pieces are called chunks. And suddenly, a seemingly simple question appears: Where exactly should we cut the document?
Cut in the wrong place, and you destroy the exact context you were trying to preserve. Cut too large, and your search becomes blurry and imprecise.
That is the Chunking Dilemma.
A Quick Road Ahead:
A full production RAG pipeline consists of:
Documents → Chunking → Embeddings → Vector / Document DB → Retrieval → LLM PromptIf terms like “embeddings” or “vector databases” sound new, don’t worry! We have dedicated deep-dive articles coming up for each of those topics. In fact, because chunking is so foundational, we are splitting it into two digestible reads:
Today (Part 4): The Ingestion Pipeline, the Goldilocks chunk-sizing problem, and baseline splitting strategies.
Part 5: Advanced chunking architectures (Bounded Semantic Chunking, Parent-Document Retrieval, Breadcrumbs, and Anthropic’s Contextual Retrieval).
Let’s dive in!
Before Retrieval, There Is an Entire Pipeline
Let’s zoom out for a moment. A production RAG system is split into two fundamentally different journeys:
┌────────────────────────────────────────────────────────────────────────┐
│ OFFLINE INGESTION PIPELINE │
│ │
│ Raw Sources (PDFs, Wikis, Codebases, Confluence, DBs, Tickets) │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Parse & Clean: Extract text, tables, code, headers │ │
│ └──────────────────────────────┬─────────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ CHUNKING: Break documents into meaningful, retrievable units │ │
│ └──────────────────────────────┬─────────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Metadata Enrichment: Breadcrumbs, page, source, parent_id... │ │
│ └──────────────────────────────┬─────────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Embedding Generation: Text → Vector coordinates │ │
│ └──────────────────────────────┬─────────────────────────────────┘ │
│ ▼ │
│ Vector Database & Document Store (Indexed for Search) │
└────────────────────────────────────────────────────────────────────────┘
Notice something critical: Chunking isn’t a minor preprocessing detail. It defines the physical units of memory your search engine will later evaluate.
Now compare that with what happens when a user asks a question:
┌────────────────────────────────────────────────────────────────────────┐
│ ONLINE QUERY PIPELINE │
│ │
│ User Question │
│ │ │
│ ▼ │
│ Query Embedding │
│ │ │
│ ▼ │
│ Vector Search / Keyword Retrieval │
│ │ │
│ ▼ │
│ Top-K Matching Chunks Retrieved │
│ │ │
│ ▼ │
│ Reranking & Metadata Filtering │
│ │ │
│ ▼ │
│ LLM Prompt + Retrieved Context │
│ │ │
│ ▼ │
│ Accurate Answer │
└────────────────────────────────────────────────────────────────────────┘
The online query pipeline gets all the attention because it touches the user interface. But the offline ingestion pipeline quietly sets the hard ceiling on what your system is capable of finding. And this is where our story begins.
Storytime: The Day the Staging Bot Lied
Imagine you’re building your first enterprise RAG prototype. You follow a quickstart tutorial, load a few sample PDFs, and write those famous three lines of code:
# The tutorial dream
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
chunks = text_splitter.split_documents(docs)
You test it with five sample questions. Five out of five answers look brilliant! High fives all around. You demo it to your team, everyone is impressed, and you connect the bot to your company’s actual knowledge base: 50,000 files spanning employee policies, SEC financial filings, API documentation, and architecture wikis.
Two days later, Slack starts blowing up:
- “The bot says our Q3 revenue dropped by 24%!” (Yet the official earnings release says global revenue grew by 24%).
- “The bot told an employee they’re not eligible for parental leave!”
- “The API endpoint the bot gave me doesn’t exist!”
Your first instinct is to blame the LLM. You assume the model is hallucinating, so you upgrade from GPT-4o-mini to Claude 3.5 Sonnet, lower the temperature to 0.0, and spend four hours rewriting the system prompt. Nothing fixes it.
Eventually, you stop looking at the LLM. You open your database and inspect the exact chunk that was retrieved for that financial query:
_"...decreased by 24% due to unexpected supply chain disruptions in the European sector."_
It looks like a reasonable sentence. Except it isn’t.
The chunk started right in the middle of a sentence! The text immediately preceding it had said:
“While overall global revenue grew by 24%, regional European shipping volume decreased by 24%…”
The retrieval system didn’t retrieve a complete thought. It retrieved a severed fragment of a thought. And the LLM had no choice but to answer using that mutilated fragment.
Here is the vital takeaway:
A retrieval system can only retrieve the pieces you created during ingestion. If those pieces are poorly designed, no multi-billion-parameter LLM or reranker can magically reconstruct the truth.
The “Goldilocks Dilemma” of Chunk Sizing
So, what should your chunk size be? 100 tokens? 500? 1,000? 2,000?
There is no universal magic number. We face the classic Goldilocks Problem: chunks can be too small, or they can be too large.
┌───────────────────────────────┐ ┌───────────────────────────────┐
│ CHUNKS TOO SMALL │ │ CHUNKS TOO LARGE │
│ (e.g., 50 tokens) │ │ (e.g., 2,000 tokens) │
├───────────────────────────────┤ ├───────────────────────────────┤
│ ✗ Missing critical context │ │ ✗ Multiple mixed topics │
│ ✗ Severed rules & exceptions │ │ ✗ Mean pooling dilution │
│ ✗ The "Half-Truth" catalyst │ │ ✗ Semantic blur in search │
│ ✗ Orphaned pronouns │ │ ✗ Wasted prompt token budget │
└───────────────────────────────┘ └───────────────────────────────┘
1. The Small Chunk Trap (The “Half-Truth” Catalyst)
It’s tempting to think: “If I make my chunks tiny (like 1 or 2 sentences), search precision will be laser-sharp because the chunk will match the user’s exact keywords!”
While small chunks match search queries easily, they act as a direct catalyst for hallucinations and dangerous half-truths.
Consider this employee handbook policy:
“Employees are entitled to 20 days of paid vacation per year. However, contractors and interns are not eligible for paid vacation.”
If your chunker cuts too aggressively, it produces:
- Chunk A: “Employees are entitled to 20 days of paid vacation per year.”
- Chunk B: “However, contractors and interns are not eligible for paid vacation.”
Now an intern asks your company bot:
“Am I eligible for paid vacation?”
The retrieval system searches for “eligible paid vacation”. Chunk A looks like a 99% match. It retrieves Chunk A and sends it to the LLM.
The LLM reads: “Employees are entitled to 20 days of paid vacation per year.”
And what does it conclude?
“Yes! You are entitled to 20 days of paid vacation!”
Notice what happened here:
- The LLM did not make up fake information from thin air.
- What it said was technically written in the document.
- But it produced a catastrophic half-truth because chunking separated the rule from its exception.
This happens whenever chunking splits:
- A rule from its exception
- A condition from its prerequisite
- A definition from the term it defines
- A pronoun (“it”, “they”) from its antecedent noun
Rule of Thumb: The goal of chunking is not “make every chunk exactly 400 tokens.” The goal is to create chunks that are independently meaningful units of thought.
2. The Large Chunk Trap (Semantic Blur & Mean Pooling Dilution)
So you might think: “If small chunks lose context, let’s make them huge! Say, 2,000 tokens. Modern embedding models can read thousands of tokens, so why not give it the whole section?”
Because you run into the opposite problem: Loss of Search Precision (Semantic Blur).
An embedding model does not store 2,000 separate concepts. It must compress that entire 2,000-token chunk into a single mathematical vector (e.g., 768 or 1,536 numbers).
Under the hood, most transformer embedding models do this through a mathematical operation called Mean Pooling: they calculate the mathematical average of all token vectors across the chunk!
Now imagine what that means in practice. Suppose a 2,000-token chunk covers four disparate topics:
- Tokens 1–500: Office parking garage permits
- Tokens 501–1000: Healthcare and dental coverage
- Tokens 1001–1500: Kubernetes cluster access for engineers
- Tokens 1501–2000: Client dinner expense policies and alcohol limits
When the embedding model computes the mean vector for this chunk, the vector’s direction in mathematical space is pulled in four completely different directions:
- 25% Parking
- 25% Health
- 25% Kubernetes
- 25% Dinner Expenses
Now, an employee asks:
“What is the maximum Food expense allowed for a client dinner?”
The user’s query is only 12 tokens. Its embedding vector is 100% focused on client dinners and expenses.
When your vector database calculates the Cosine Similarity between:
- Query Vector (100% Dinner Expenses)
- Giant Chunk Vector (25% Dinner Expenses + 25% Parking + 25% Health + 25% Kubernetes)
What happens? The similarity score drops significantly (e.g., to 0.48) because 75% of the giant chunk’s vector is pulling toward parking garages and server clusters!
Meanwhile, a focused 120-token chunk that only discussed client dinner expenses will score 0.89!
(Note: Those similarity numbers are illustrative, not universal laws. The key takeaway is the difference in semantic focus).
The information hasn’t disappeared from the 2,000-token chunk. But the representation used for retrieval has become an “averaged-out soup.” That is Semantic Blur.
The Chunking Sweet Spot
So we have two competing requirements:
- Small chunks: High retrieval precision, but risk missing surrounding context.
- Large chunks: Broad context, but risk semantic dilution and prompt bloat.
Instead of asking: “What is the best chunk size?”, the better engineering question is:
“What is the smallest unit of text that preserves enough self-contained meaning for my retrieval task?”
At this point, your natural instinct might be: “Okay, I get it. I shouldn’t make my chunks 50 tokens, and I shouldn’t make them 2,000 tokens. Let’s aim for that sweet spot! I’ll just open my Python script, configure _chunk_size = 500_, and start slicing.”
Hold on.
Before you write that line of code, there is a subtle, deceptive trap waiting for you.
The Silent Trap: Characters Are Not Tokens
When you configure chunk_size = 500 in a text splitter, ask yourself: 500 of what? In Python, length is measured in raw characters (letters, spaces, and punctuation):
text = "RAG pipeline"
len(text) # Returns 12 characters
If you don’t supply a custom tokenizer, many standard splitters count 500 characters. In English prose, an average word is about 5 characters plus a space. That means a 500-character chunk is barely 80 to 100 words! Without realizing it, you just walked right back into the Small Chunk Trap.
And it gets more dangerous: embedding models and LLMs do not process characters or words. They process tokens.
While clean English prose follows a rough rule of thumb:
General English Rule of Thumb:
1 Token ≈ 0.75 words ≈ 4 characters
The real world is rarely that clean. Consider what happens when your enterprise document contains:
- Indented Python or JavaScript code
- Markdown tables with pipes, dashes, and whitespace
- URLs, JSON payloads, and technical identifiers
- Text in non-English languages (like Hindi, Japanese, or German)
In these scenarios, character-to-token ratios explode! A chunk that looks like an innocent 1,500 characters can easily expand into 700 tokens.
Why does this matter so much? Because of a number that trips up even experienced AI engineers: the embedding model’s context ceiling.
The 512-Token Truncation Trap
We often hear about frontier LLMs boasting massive 128k or 1-million-token context windows. But don’t confuse your Generation LLM with your Embedding Model:
- Generation LLM Context Window (e.g., 128k for GPT-4o, 1M for Gemini): Determines how much total retrieved context you can feed to the prompt at query time.
- Embedding Model Sequence Limit (e.g., 512 tokens for
**e5-base**,**bge-base**, BERT): Determines how much text can be converted into a vector during the offline ingestion pipeline.
Many of the most popular, cost-effective open-source embedding models deployed across the industry have a hard sequence limit of just 512 tokens.
Now, picture what happens when that 1,500-character chunk (which expanded to 700 tokens) arrives at a 512-token embedding model:
The tokenizer silently applies **truncation=True**. It embeds the first 512 tokens, chops off the remaining 188 tokens, and tosses them in the trash!
There is no error thrown in your console. Your ingestion script happily prints 100% Ingested. Your vector database stores a vector. But the final 188 tokens never participated in the embedding. If the exact answer to your user's question lived in those truncated lines, your vector search will never find it.
Production Rule: Never design your chunking pipeline using raw character counts alone. Always measure and enforce chunk boundaries using the actual tokenizer and token limit of your target embedding model.
Now that we know we must measure in tokens and stay within our model’s physical limits, how do we actually slice the text?
Let’s explore the three foundational strategies.
Three Baseline Production Chunking Strategies
Now let’s look at the first three foundational chunking strategies used in real-world systems.
Strategy 1: Fixed-Size Chunking (The Naive Baseline)
This is the simplest approach: slice text every N characters or tokens unconditionally, like a cookie cutter.
Original sentence:
"Under clause 14, the total aggregate liability of the Vendor shall not exceed $1,000,000."
Fixed cut at character 60:
[Chunk 1]: "Under clause 14, the total aggregate liability of the Vendor"
[Chunk 2]: " shall not exceed $1,000,000."
If a user asks: “What is the vendor’s liability cap?”, vector search pulls Chunk 1 because it matches “liability of the Vendor”. But Chunk 1 has no numbers! The LLM reads Chunk 1 and answers: “The agreement mentions liability under clause 14, but does not state any dollar amount.”
Verdict: 🛑 Do not use this in production. It cuts words, severs numbers from clauses, and destroys grammar.
Strategy 2: Recursive Character Splitting (The Enterprise Workhorse)
For general unstructured text — articles, reports, memos, documentation — a recursive splitter is the standard baseline.
Instead of slicing text at an arbitrary position, it tries progressively more granular separators:
Paragraph (\n\n) ──► Line (\n) ──► Sentence (. ) ──► Word ( ) ──► Character ("")
It acts like a thoughtful editor. It first asks: “Can I keep entire paragraphs together within my chunk size limit?” If a paragraph is too long, it steps down and tries line breaks. If a line is still too long, it splits on sentences. It only ever splits words as an absolute last resort.
Raw Document
│
├── Can this paragraph fit within chunk_size?
│ ├── YES ──► Keep it together!
│ └── NO ──► Step down to line breaks (\n)
│
├── Can this line fit?
│ ├── YES ──► Keep it together!
│ └── NO ──► Step down to sentence endings (. )
│
└── Step down until pieces fit comfortably within token bounds.
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=400, # Target tokens
chunk_overlap=80, # 20% safety net
separators=["\n\n", "\n", ". ", " ", ""]
)
Chunk Overlap: The Safety Net
Whenever you use recursive splitting, you should configure chunk overlap (typically 10% to 20% of your chunk size):
[ Chunk 1: Tokens 0 ──────────────► 400 ]
[ Chunk 2: Tokens 320 ──────────────► 720 ] (80-token overlap)
[ Chunk 3: Tokens 640 ──────────────► 1040 ]
Why overlap matters:
Sometimes, a critical clause spans across a chunk boundary:
Without Overlap:
- Chunk 1:
"...employees can claim reimbursement after..." - Chunk 2:
"...submitting Form 104-B within 30 days."
The relationship between the requirement and the form is severed.
With Overlap:
The sliding window captures the complete boundary clause unbroken in Chunk 2!
The Trade-Off: More overlap means slightly more storage and duplicate text in your vector index. A starting point of 10%–20% (50 to 80 tokens = roughly 2–3 sentences) provides a solid safety margin.
Strategy 3: Structure-Aware Chunking (Markdown, Code, and Tables)
Humans don’t write in random blocks of 400 words. We write using explicit structures: headings, sections, code blocks, and tables.
When you run a naive text splitter on structured documents, you throw that structure away:
- A sub-bullet point is separated from its parent
# Heading. - A Python function gets cut across its loop body.
- A table row is disconnected from its column headers.
1. Markdown Header Splitting
In technical manuals and wikis, headings (#, ##, ###) define logical units of thought.
Structure-aware splitters divide text based on these headings and automatically inject the heading path into each chunk’s metadata:
from langchain_text_splitters import MarkdownHeaderTextSplitter
headers_to_split_on = [ ("#", "Section"),
("##", "Subsection"),
("###", "Topic")
]
markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=headers_to_split_on
)
Now, each chunk isn’t just floating text. It is an indexed object carrying its exact structural hierarchy in its metadata payload:
{
"metadata": {
"Section": "Benefits",
"Subsection": "Health Insurance",
"Topic": "Out-of-Network Coverage"
},
"page_content": "Employees can claim out-of-network reimbursement after..."
}
This structural metadata solves two major production problems:
- Prevents Orphan Text: Even if the chunk itself never mentions the words “Health Insurance”, the metadata permanently binds the text to its parent topic.
- Enables Metadata Filtering: At query time, you can pre-filter searches directly in your database (e.g., only search where
"Section": "Benefits"), eliminating irrelevant company documents before vector search even begins!
2. Code-Aware Splitting
If you are indexing a software repository, chunking must respect the programming language’s Abstract Syntax Tree (AST):
def process_payment():
try:
charge_customer()
update_order()
except PaymentError:
rollback()
Cutting this code block arbitrarily in the middle of the try block destroys its syntax and meaning. Language-aware splitters keep entire functions, classes, or imports together as single logical blocks.
3. The Table Problem (Financial 10-K & 10-Q Filings)
Tables are one of the fastest ways to break a naive RAG pipeline. Consider this balance sheet:
| Product Segment | Q1 2024 ($M) | Q2 2024 ($M) | Q3 2024 ($M) |
| --------------------- | -----------: | -----------: | -----------: |
| Cloud Subscriptions | $4,200 | $4,600 | $5,100 |
| Professional Services | $320 | $310 | $305 |
Now imagine a standard text splitter cuts this table horizontally when it hits a character limit:
- Chunk 1:
Product Segment | Q1 2024 | Q2 2024 | Q3 2024Cloud Subscriptions | $4,200 | $4,600 | $5,100 - Chunk 2:
Professional Services | $320 | $310 | $305
Now an analyst asks:
“What was Professional Services revenue in Q2 2024?”
The vector search retrieves Chunk 2. But Chunk 2 contains three bare numbers without the header Q2 2024! The retrieval succeeded, but answer generation fails because the structural context was lost.
The Production Fix for Tables:
- Use Table-Aware Parsers that convert rows into structured formats (like Markdown or JSON) that repeat the column headers on every slice:
{
"Company": "ACME Corp",
"Report": "Q3 2024",
"Segment": "Professional Services",
"Period": "Q2 2024",
"Revenue": "$310M"
}
- Now the number has complete meaning:
Professional Services → Q2 2024 → Revenue → $310M.
Where Do We Go From Here?
We’ve now established the fundamentals:
- The Ingestion Pipeline is the kitchen prep that determines your RAG system’s accuracy ceiling.
- Tiny chunks cause “half-truth” hallucinations; giant chunks cause “semantic blur” via mean pooling.
- Token limits like 512 tokens must be strictly respected to avoid silent data truncation.
- Recursive and structure-aware splitting protect paragraphs, code blocks, and tables.
But what happens when documents don’t have neat headings?
What happens when you have messy meeting transcripts where people switch topics unpredictably?
And what happens when your vector search needs small chunks for search accuracy, but your LLM needs an entire 2-page policy for context?
In Part 5, we take chunking to the advanced level:
- Bounded Semantic Chunking: Detecting topic shifts dynamically with mathematical distance thresholds and size guardrails.
- Parent-Document Retrieval (Small-to-Big): How to search tiny child chunks while feeding large parent sections to the LLM (and how this prevents an $8,400 medical bill surprise!).
- Breadcrumbs vs. Anthropic’s Contextual Retrieval: The complete architectural face-off.
- The Production Decision Matrix: Choosing the exact strategy for your document types.
👉 Continue to Part 5: Advanced RAG Chunking: Hierarchical, Semantic, and Contextual Retrieval
What’s the trickiest chunking bug you’ve encountered in your own pipelines? Have you seen tables or severed sentences trip up your models? Let’s chat in the comments!
Top comments (0)