Imagine I give you a whole pizza and say:
“Eat it.”
You look at it and think, Sure, I can eat it.
But now imagine I give you the same pizza without cutting it.
Can you eat it comfortably?
Not really.
You would probably cut it into smaller pieces first.
And that is exactly what we do with large documents before giving them to an AI system.
A 200-page PDF, a 50,000-line documentation file, or a huge company knowledge base is useful to us as a complete document.
But for a Retrieval-Augmented Generation (RAG) system, giving the entire document to the retrieval system is usually not the best idea.
So we break it down.
Document
↓
Chunks
↓
Embeddings
↓
Vector Database
↓
Retrieval
↓
LLM
↓
Answer
That process of breaking a large document into smaller, meaningful pieces is called Chunking.
And here is the interesting part:
Chunking is not simply “splitting text into 500 characters.”
It is a retrieval design decision.
The wrong chunk can make the right information difficult to retrieve.
Index
- Why Does Chunking Matter?
- How Chunking Fits Into RAG
- The Chunk Size Trap
- Chunking Strategies
- Chunk Size and Chunk Overlap
- Implementing Chunking with LangChain
- How to Choose the Right Chunking Strategy
- How to Evaluate Chunking
- A Practical Chunking Evaluation Experiment
- Common Chunking Mistakes
- My Practical Chunking Workflow
- Final Takeaway
1. Why Does Chunking Matter?
Let's say we have this document:
Company Employee Handbook
Chapter 1: Leave Policy
Employees receive 20 paid leaves every year...
Chapter 2: Work From Home
Employees can work remotely two days per week...
Chapter 3: Insurance
Employees are eligible for health insurance...
Now a user asks:
“How many days can I work from home?”
We don't need the entire employee handbook.
We need the small section containing the Work From Home Policy.
So ideally:
DOCUMENT
│
┌────────────┼────────────┐
↓ ↓ ↓
Chunk 1 Chunk 2 Chunk 3
│ │ │
Leave Policy WFH Policy Insurance
│
↓
Relevant Chunk
The goal is simple:
Retrieve the smallest useful piece of information without destroying its meaning.
The Two Ways Chunking Goes Wrong
There are two opposite mistakes.
Chunks are too large
Imagine:
┌──────────────────────────────┐
│ Leave Policy │
│ Work From Home │
│ Insurance │
│ Salary │
│ Performance Review │
│ Travel Policy │
│ ... │
└──────────────────────────────┘
You ask:
“How many leaves do I get?”
The answer exists somewhere inside the chunk.
But so does a lot of unrelated information.
The retriever has brought the answer plus noise.
Chunks are too small
Now imagine:
Chunk 1:
Employees are eligible for
Chunk 2:
20 days of paid leave
Chunk 3:
every financial year.
The original document contained a complete thought.
Our chunking destroyed it.
The information exists.
But the meaning is fragmented.
This gives us the first important principle:
Chunking is a balance between context and precision.
Too large:
More context
↓
More noise
Too small:
More precision
↓
Less context
The goal is somewhere in between.
2. How Chunking Fits Into RAG
A typical RAG pipeline looks like this:
ORIGINAL DOCUMENT
│
▼
┌─────────────┐
│ Chunking │
└──────┬──────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
Chunk 1 Chunk 2 Chunk 3
│ │ │
▼ ▼ ▼
Embedding Embedding Embedding
│ │ │
└────────────┼────────────┘
▼
Vector Database
│
│ User Query
▼
Query Embedding
│
▼
Retrieval
│
▼
Relevant Chunks
│
▼
LLM
│
▼
Answer
There is one important thing to notice:
Chunking decides what the embedding model actually gets to understand.
If you change the chunks, you change the embeddings.
If you change the embeddings, you can change retrieval.
And if retrieval changes, the context given to the LLM changes.
So chunking is not just preprocessing.
It is part of your retrieval architecture.
3. The Chunk Size Trap
This is probably one of the most misunderstood parts of chunking.
Someone might ask:
“My LLM supports 128K tokens. Should I create 8K-token chunks?”
Not necessarily.
Your LLM's context window is only one part of the equation.
Your chunk size depends on several things:
CHUNK SIZE
│
┌────────────────┼────────────────┐
↓ ↓ ↓
Embedding Model Retrieval LLM
Input Limit Requirements Context Budget
│ │ │
└────────────────┼────────────────┘
│
▼
Practical Chunk Size
And there are two more factors that matter:
- Nature of your documents
- Nature of your questions
So the decision becomes:
Document Structure
+
Embedding Model Limit
+
Retrieval Requirements
+
LLM Context Budget
+
Question Type
↓
Candidate Chunk Sizes
↓
Evaluation
↓
Final Chunk Size
This is why there is no universal:
chunk_size = 500
number.
Don't start with “What chunk size does everyone use?” Start with “What information am I trying to retrieve?”
4. Chunking Strategies
There is no universally best chunking strategy.
A legal contract, a Markdown documentation page, and a Python codebase don't contain information in the same structure.
So why would we split them in exactly the same way?
LangChain provides different text splitters, and its documentation recommends RecursiveCharacterTextSplitter as a strong starting point for generic text.
Let's group the important approaches.
4.1 Fixed-Size Chunking
The simplest approach:
Every 1000 characters → new chunk
Example:
from langchain_text_splitters import CharacterTextSplitter
splitter = CharacterTextSplitter(
chunk_size=1000,
chunk_overlap=100
)
chunks = splitter.split_text(text)
The advantage?
Simple and predictable.
The problem?
It doesn't necessarily care about meaning.
It can split:
at the end of a paragraph
or:
in the middle of a sentence.
So fixed-size splitting is useful for simple, predictable data, but it is rarely the only strategy worth considering.
4.2 Recursive Character Chunking
This is usually the best starting point for general text.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
chunks = splitter.split_text(text)
Conceptually, the splitter tries to preserve larger natural boundaries before breaking the text further:
Paragraph
↓
Newline
↓
Space
↓
Character
So instead of immediately cutting through a sentence, it tries to preserve meaningful blocks first.
Document
│
├── Paragraph
│ │
│ └── fits? → keep together
│
├── Paragraph
│ │
│ └── too large?
│ ↓
│ split further
│
└── ...
That's why it is such a useful generic starting point.
When you don't know where to start, start simple. Then measure.
4.3 Token-Based Chunking
Characters and tokens are not the same thing.
For example:
1000 characters ≠ 1000 tokens
The exact relationship depends on the tokenizer, language, and text.
If your downstream system has strict token constraints, token-aware splitting becomes useful.
from langchain_text_splitters import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
Token-based splitting becomes particularly useful when you need precise control over how much tokenized content enters the embedding or downstream pipeline.
4.4 Structure-Aware Chunking
Sometimes the document already tells us how it should be chunked.
Consider Markdown:
# Authentication
## Login
Information about login...
## Password Reset
Information about resetting passwords...
# Payments
## Refunds
Information about refunds...
Why destroy that structure?
Instead, preserve it.
The same idea applies to:
- Markdown
- HTML
- JSON
- XML
- source code
- legal documents
- technical documentation
For example, a chunk can retain:
Document: API Documentation
Section: Authentication
Subsection: Password Reset
Content:
To reset your password...
Now the chunk contains both:
information + context
That's much more useful for retrieval.
4.5 Code Chunking
Code is not prose.
Consider:
class PaymentService:
def process_payment(self):
...
Randomly splitting every 500 characters can easily separate:
class
↓
method
↓
implementation
which destroys useful relationships.
For code, you want to preserve programming structure where possible:
Repository
↓
Class
↓
Method
↓
Logical Block
LangChain provides language-aware splitting for multiple programming languages.
So for code:
Respect the syntax before respecting the character count.
4.6 Semantic Chunking
Now we move from:
“Where should I split the characters?”
to:
“Where does the meaning change?”
Consider:
The company was founded in 1998.
It started with five employees.
The company launched its first product in 2001.
Revenue crossed $10M in 2005.
There is a natural semantic transition between these ideas.
Semantic chunking attempts to identify those changes rather than blindly following fixed character boundaries.
This can be useful for highly topic-driven documents.
But there is a trade-off.
It can introduce:
- additional computation
- additional complexity
- additional tuning
So don't automatically assume:
Semantic = Better
Sometimes:
Simple + predictable + evaluated > sophisticated + unevaluated.
5. Chunk Size and Chunk Overlap
Now we reach the famous:
chunk_size=1000
chunk_overlap=200
But what do these numbers actually mean?
Chunk Size
Chunk size defines how much content goes into one chunk.
For example:
chunk_size = 1000
means the splitter attempts to create chunks around that size according to the unit it uses.
That unit could be:
- characters
- tokens
- another model-specific measure
For example, RecursiveCharacterTextSplitter measures characters by default.
Chunk Overlap
Suppose the document says:
Machine learning models require training data.
The quality of this data directly affects model performance.
Good data leads to better generalization.
If we split aggressively:
Chunk 1:
Machine learning models require training data.
Chunk 2:
The quality of this data directly affects model performance.
The relationship between the two chunks can become weaker.
Overlap creates a bridge.
Chunk 1
┌───────────────────────────────┐
│ Machine learning models... │
│ Training data affects... │
└───────────────┬───────────────┘
│
overlap
│
▼
┌───────────────────────────────┐
Chunk 2 │ Training data affects... │
│ Good data leads to... │
└───────────────────────────────┘
So:
Overlap protects context at the boundary.
But more overlap is not automatically better.
Too much overlap means:
More overlap
↓
More chunks
↓
More embeddings
↓
More storage
↓
More duplicate retrieval
↓
More context repetition
↓
Higher cost
Use overlap when it helps preserve meaning across boundaries.
6. Implementing Chunking with LangChain
Let's build a simple example.
Install the splitter package:
pip install -U langchain-text-splitters
Then:
from langchain_text_splitters import RecursiveCharacterTextSplitter
text = """
Chunking is important for Retrieval Augmented Generation.
Large documents are difficult to retrieve efficiently.
By breaking documents into smaller meaningful pieces,
we can retrieve only the information required to answer
a user's question.
"""
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
print(f"Chunk {i}")
print(chunk)
print("-" * 50)
The flow is:
Original Text
│
▼
RecursiveCharacterTextSplitter
│
┌────┼────┐
↓ ↓ ↓
C1 C2 C3
│ │ │
└────┼────┘
↓
Embeddings
For token-controlled pipelines:
from langchain_text_splitters import TokenTextSplitter
splitter = TokenTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = splitter.split_text(text)
And for structured documents, choose the splitter that understands that structure rather than flattening everything into plain text.
7. How to Choose the Right Chunking Strategy
Here's the question that actually matters:
Which chunking method should I use?
Don't choose based on popularity.
Choose based on your data.
| Document Type | Good Starting Strategy |
|---|---|
| General text | Recursive |
| Markdown | Header / structure-aware |
| HTML | HTML-aware |
| Source code | Language-aware |
| JSON | Structure-aware |
| Highly topic-driven text | Semantic |
| Strict token constraints | Token-based |
| Legal / structured documents | Structure-aware |
| Tables | Preserve table structure |
Think about it like this:
What is my data?
│
┌────────────────┼────────────────┐
↓ ↓ ↓
Plain Text Structured Code
│ │ │
↓ ↓ ↓
Recursive Structure- Language-
aware aware
│ │ │
└────────────────┼────────────────┘
↓
Check Embedding Model
↓
Choose Candidates
↓
Evaluate
And that last step is important.
Choosing a chunking strategy is a hypothesis.
Evaluation tells you whether the hypothesis was correct.
8. How to Evaluate Chunking
This is where many chunking tutorials stop.
They shouldn't.
Because the real question isn't:
“Did my document split successfully?”
It is:
“Did splitting improve retrieval?”
A chunk can look perfectly reasonable to a human and still perform badly in a retrieval system.
So we need an evaluation dataset.
Step 1: Create Real Questions
Suppose we're building an employee-policy chatbot.
Create questions such as:
Q1: How many annual leaves does an employee get?
Q2: How many days can employees work remotely?
Q3: What is the maternity leave duration?
Q4: What happens to unused leaves?
Q5: What is the resignation notice period?
For each question, know where the answer actually exists.
For example:
Question:
How many annual leaves does an employee get?
Expected Source:
Employee Handbook
→ Leave Policy
→ Section 2.1
Now we have something measurable.
9. Retrieval Metrics
Suppose the correct chunk is:
Chunk 17
Our retriever returns:
Top 5:
Chunk 91
Chunk 43
Chunk 17 ← Correct
Chunk 52
Chunk 8
The correct chunk appeared in the top five.
That's useful.
Now we can measure different aspects of retrieval quality.
9.1 Recall@K
Recall@K asks:
Did the relevant chunk appear anywhere in the top K results?
For example:
Top 5:
Chunk 91
Chunk 43
Chunk 17 ← Relevant
Chunk 52
Chunk 8
The relevant chunk is present.
Therefore:
Recall@5 = 1
or 100% for this query.
Across many queries:
Recall@5 =
Queries where relevant information
appeared in top 5
───────────────────────────────────
Total queries
So if:
90 / 100
queries retrieved the correct information within the top 5:
Recall@5 = 90%
In simple terms:
Recall asks: “Did I find it?”
9.2 Precision@K
Precision@K asks a different question:
“How many of the retrieved results were actually relevant?”
Suppose:
Top 5:
Chunk 17 → Relevant
Chunk 21 → Relevant
Chunk 42 → Irrelevant
Chunk 63 → Irrelevant
Chunk 91 → Irrelevant
Then:
Precision@5 = 2 / 5
= 40%
So:
Recall → Did I find the answer?
Precision → How much irrelevant information did I retrieve?
This distinction becomes especially important when large chunks contain multiple unrelated topics.
9.3 MRR — Mean Reciprocal Rank
Now imagine two systems.
System A
Top 5:
1. Correct ✓
2. Wrong
3. Wrong
4. Wrong
5. Wrong
System B
Top 5:
1. Wrong
2. Wrong
3. Correct ✓
4. Wrong
5. Wrong
Both systems found the correct answer within the top 5.
So their Recall@5 is the same.
But are they equally good?
Not really.
System A put the correct result first.
That's where MRR — Mean Reciprocal Rank becomes useful.
For one query:
MRR = 1 / rank of first relevant result
So:
Correct at rank 1
→ 1 / 1
→ 1.0
Correct at rank 2
→ 1 / 2
→ 0.5
Correct at rank 3
→ 1 / 3
→ 0.33
Correct at rank 5
→ 1 / 5
→ 0.20
For multiple queries, we take the average of these reciprocal ranks.
For example:
Query 1 → Correct at #1 → 1.00
Query 2 → Correct at #2 → 0.50
Query 3 → Correct at #3 → 0.33
MRR = (1.00 + 0.50 + 0.33) / 3
≈ 0.61
So:
Recall tells you whether you found the answer. MRR tells you how high you ranked the first correct answer.
This is particularly useful when the order of retrieved results matters.
9.4 Don't Evaluate Only Retrieval
A RAG system has multiple layers:
User Question
│
▼
Retrieval
│
▼
Retrieved Chunks
│
▼
LLM
│
▼
Answer
Therefore, evaluate both retrieval and final answers.
Retrieval Quality
Did we retrieve the right information?
Useful metrics:
- Recall@K
- Precision@K
- MRR
Answer Quality
Did the model actually use that information correctly?
Useful dimensions:
Answer Correctness
Did the model produce the correct answer?
Answer Relevance
Did the answer actually address the user's question?
Faithfulness / Groundedness
Is the answer supported by the retrieved context?
This distinction matters.
Good retrieval does not automatically mean a good answer.
And:
A good-looking answer does not automatically mean good retrieval.
10. A Practical Chunking Evaluation Experiment
Let's say we have:
1,000 documents
200 evaluation questions
We test three configurations.
Experiment 1
Recursive
chunk_size = 256
overlap = 32
Experiment 2
Recursive
chunk_size = 512
overlap = 64
Experiment 3
Recursive
chunk_size = 1024
overlap = 128
Now measure:
| Configuration | Recall@5 | Precision@5 | MRR | Answer Correctness |
|---|---|---|---|---|
| 256 / 32 | 82% | 76% | 0.71 | 84% |
| 512 / 64 | 91% | 83% | 0.86 | 92% |
| 1024 / 128 | 93% | 61% | 0.78 | 87% |
Which one wins?
Probably:
512 / 64
Why?
The 1024 configuration has slightly higher Recall@5.
But it retrieves significantly more irrelevant information and ranks the relevant information less effectively.
So the final answer quality also drops.
This gives us another important principle:
The largest chunk is not necessarily the best chunk.
And:
The goal isn't to maximize one metric. It's to find the best trade-off for your application.
11. Metadata: The Missing Piece
Let's say our chunk is:
Employees can work remotely two days per week.
That's useful.
But this is better:
Document: Employee Handbook
Section: Work From Home
Department: HR
Year: 2026
Content:
Employees can work remotely two days per week.
Why?
Because the content tells us what the information says.
Metadata tells us where it came from.
Metadata can also enable filtering:
department = HR
document = Employee Handbook
year = 2026
Then semantic retrieval can happen over a much more relevant subset.
So:
Good chunking tells you what the text says. Good metadata tells you where it belongs.
12. Common Chunking Mistakes
❌ 1. Using an arbitrary chunk size
Tutorial says 500
↓
I use 500
That's not a strategy.
Test multiple values against your actual data.
❌ 2. Ignoring the embedding model
Your LLM may support 128K tokens.
Your embedding model has its own input constraints.
Always check:
Embedding model
↓
Input/token limit
↓
Tokenizer
↓
Actual chunk size
❌ 3. Using the same strategy for every document
A legal contract and a Python codebase are not the same thing.
Use the structure already present in the data.
❌ 4. Using too much overlap
More overlap doesn't automatically mean more context.
It can mean:
More duplication
↓
More embeddings
↓
More storage
↓
More retrieval noise
↓
Higher cost
❌ 5. Destroying tables and structure
Imagine:
| Product | Price | Discount |
|---------|-------|----------|
| A | 100 | 10% |
| B | 200 | 15% |
If you blindly split this structure, you can lose relationships between columns and values.
For structured data:
Preserve structure before splitting aggressively.
13. My Practical Chunking Workflow
If I were starting a new RAG system tomorrow, I wouldn't immediately build a complicated semantic chunking pipeline.
I'd start simple.
Step 1 — Understand the Data
PDF?
Markdown?
HTML?
Code?
Legal documents?
Tables?
Step 2 — Preserve Natural Structure
Document
↓
Headers
↓
Sections
↓
Paragraphs
↓
Sentences
Don't destroy structure that already exists.
Step 3 — Check the Embedding Model
Ask:
What is the model's input limit?
What tokenizer does it use?
How does it behave with longer inputs?
Remember:
The embedding model is part of the chunk-size decision.
Step 4 — Choose an Initial Strategy
For example:
Plain text
↓
Recursive
Markdown
↓
Header-aware
Code
↓
Language-aware
Highly topic-driven
↓
Semantic
Step 5 — Test Multiple Chunk Sizes
For example:
256
512
768
1024
with sensible overlap values.
Don't treat these as universal recommendations.
They are simply candidate configurations to test.
Step 6 — Build a Real Evaluation Dataset
Create questions based on what your users actually ask.
Question
↓
Expected Source
↓
Expected Information
Step 7 — Measure Retrieval
Track:
Recall@K
Precision@K
MRR
Step 8 — Measure Final Answers
Track:
Answer Correctness
Answer Relevance
Faithfulness / Groundedness
And where useful, also track:
Latency
Token Usage
Embedding Cost
Storage Cost
Step 9 — Pick the Simplest Winner
Suppose:
Recursive Chunking
performs almost as well as:
Semantic Chunking
but is cheaper, faster, and easier to maintain.
Then:
Take the simpler solution.
The best chunking strategy isn't the most complicated one. It's the one that performs well on your data.
14. The Complete Chunking Decision Framework
Put everything together:
START
│
▼
Understand Data
│
▼
Preserve Data Structure
│
▼
Check Embedding Model
│
▼
Understand User Questions
│
▼
Choose Initial Strategy
│
▼
Choose Chunk Size Range
│
▼
Choose Overlap
│
▼
Build Evaluation Dataset
│
▼
Run Retrieval Tests
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Recall@K Precision@K MRR
│ │ │
└─────────────┼─────────────┘
↓
Evaluate RAG Answers
│
▼
Correctness / Relevance /
Groundedness
│
▼
Compare Results
│
▼
Tune and Repeat
│
▼
Production
The important thing is that chunking doesn't end when the document is split.
It ends when you know those chunks are helping your retrieval system.
15. Final Takeaway
Let's go back to our pizza.
You don't eat a whole pizza in one bite.
You cut it.
But you also don't cut it into 1000 tiny pieces.
Because then you have created another problem.
Chunking works the same way.
Too Large
↓
Too Much Noise
Too Small
↓
Lost Context
Just Right
↓
Better Retrieval
↓
Better Context
↓
Better Answers
And the "just right" size is not a universal number.
It depends on:
your document structure + your questions + your embedding model + your retrieval strategy + your LLM context budget.
So don't ask:
“What is the best chunk size?”
Ask:
“What is the best chunk size for my data, my embedding model, and my retrieval problem?”
Because ultimately:
Chunking isn't about making documents smaller.
It's about making knowledge retrievable.
And that's the real job of chunking.
Cut the document enough to retrieve what matters — but not so much that you lose why it matters.
Top comments (0)