Imagine you're building a RAG system for a developer documentation platform. You decide every document should be split into 500-token chunks. It sounds reasonable. The chunks are small enough for retrieval, large enough to contain some context, and easy to process consistently.
Then a developer asks: “How do I configure authentication for the API?” Your retriever finds a chunk containing the configuration parameter, but the previous chunk contains the authentication prerequisites and the next chunk contains the actual example.
The system retrieved technically relevant text. It just didn't retrieve enough of the right context. This is one of the reasons I don't think chunk size should start with a number. 500 tokens isn't a chunking strategy. It's a constraint. A good chunk should represent a meaningful unit of information that can stand on its own when retrieved.
Different Documents Have Different Boundaries
Consider source code. If you split a Python file every 500 tokens, you might end up with half a class in one chunk and the rest in another.
That's usually a terrible retrieval boundary. A better approach is to preserve natural code structures:
File
├── Class
│ ├── Method
│ ├── Method
│ └── Method
└── Function
A function or class often represents a much better semantic unit than an arbitrary token range. The same principle applies to documentation. A product guide might have:
Installation
↓
Configuration
↓
Authentication
↓
Usage
↓
Troubleshooting
If someone asks how to configure authentication, I want the chunk to preserve the relevant section rather than splitting it because an arbitrary token limit was reached.
Legal documents make this even more obvious. A clause can depend heavily on its surrounding definitions, exceptions, and conditions. Splitting a clause in the middle can produce chunks that are individually readable but legally incomplete.
And structured business documents have their own natural units: tables, policies, sections, procedures, and records. There isn't one universal chunk size that works well for all of them.
Start With the Questions, Not the Tokens
I'd first ask: What kinds of questions will users ask? Suppose users are searching a technical knowledge base. They might ask: “How do I reset my API key?”
That's likely a procedure-oriented query. Another user might ask: “Which plans support SSO?” That's a product-policy query. Another might ask: “Why am I getting error 4017?” That's an error-diagnosis query.
These questions may require very different pieces of context. A chunking strategy that works for one may perform poorly for another. The objective isn't to create chunks that are mathematically uniform. It's to create chunks that are retrievable units of meaning.
Semantic Boundaries Matter More
Think about a simple document:
- Refund Policy
- Customers can request a refund within 30 days of purchase.
- Enterprise customers must contact their account manager.
- Refunds are not available for annual contracts after renewal.
If the system splits this into arbitrary pieces, a query about enterprise refunds could retrieve the first sentence without the exception that follows it. The retrieved text is technically relevant. But it's missing the condition that changes the answer.
That's a dangerous retrieval failure. A better chunking strategy recognizes that these statements belong to the same semantic section. This is why I prefer thinking in terms of information boundaries rather than token boundaries.
Metadata Is Part of the Chunk
The content itself isn't always enough. I'd also carry metadata that helps the retrieval system understand where the chunk came from.
For example:
Now retrieval can use more than the text. A query such as: “How does authentication work in Payments API v3?” can benefit from product, version, document type, and section metadata.
Metadata can also help prevent an older version of a document from competing equally with the current one. That's particularly important for technical documentation where APIs and configuration requirements change frequently.
Overlap Isn't Automatically Good
Chunk overlap is another setting that often gets treated as a default.
For example:
- Chunk 1: tokens 1-500
- Chunk 2: tokens 400-900
- Chunk 3: tokens 800-1300
Overlap can help preserve information that sits across boundaries. But more overlap also means more duplicated content, more storage, and potentially more noisy retrieval results.
I'd use overlap when the structure of the document actually requires it. If you're splitting a long narrative where meaning crosses boundaries, some overlap can be useful. If you're splitting well-defined functions, headings, or independent records, aggressive overlap may add little value.
The question shouldn't be: “How much overlap should every chunk have?” It should be: “Where does context actually cross the boundary?”
Token Count Still Matters
This doesn't mean token count is irrelevant. It matters for context limits, latency, embedding costs, retrieval efficiency, and how much information you eventually send to the model. But I'd treat token count as a constraint, not the definition of a good chunk.
You might end up with a 200-token chunk because that's the natural semantic unit. You might have a 1,200-token chunk because a complete procedure or code class needs that much context. The important question is whether the chunk can meaningfully answer the kinds of questions you're expecting.
If a 200-token chunk contains the exact answer, making it 500 tokens doesn't automatically make it better. And if a 500-token chunk cuts a critical explanation in half, forcing it into that limit doesn't make it useful.
Evaluate Chunking Through Retrieval
The best way to know whether your chunking strategy works isn't to look at the chunks and decide they "feel right." Test them. Build representative questions and inspect what gets retrieved.
For each query, ask:
- Did the correct document appear?
- Did the relevant section appear?
- Does the chunk contain enough context?
- Was an important condition separated from the answer?
- Are irrelevant chunks ranking above useful ones?
- Does changing the chunking strategy improve retrieval?
This is where RAG evaluation becomes important. You can experiment with different chunking strategies and measure retrieval quality rather than relying on intuition.
For example:
- Strategy A → Fixed 500 tokens
- Strategy B → Fixed 1,000 tokens
- Strategy C → Section-based
- Strategy D → Structure-aware
The winner shouldn't be the one with the nicest-looking chunks. It should be the one that consistently retrieves the evidence required to answer real questions.
Good Chunking Is About Context
I don't think there's a universal "best" chunk size for RAG. There are good chunking strategies for particular data, query patterns, and retrieval systems.
Source code should respect code structure. Documentation should respect headings and procedures. Legal content should preserve clauses and their conditions. Business documents should preserve meaningful sections and records. Metadata should travel with the content. Overlap should be intentional. And token count should constrain the system without dictating its structure.
Because the goal of chunking isn't to create pieces of a certain size. It's to create pieces that contain the right information in the right context. A chunk is good when retrieving it gives the model enough evidence to answer the question correctly. Good chunking isn't about making chunks the right size. It's about making them contain the right context.
Top comments (1)
This matches what we see on structured docs: fixed token windows regularly split a procedure from its prerequisites (or a section from the example that makes it actionable), so the retriever returns a locally relevant fragment that can’t stand alone.
What worked better than picking 500 vs 1000: define chunk boundaries from document structure first (section / heading path / resource / clause), then apply a soft token ceiling inside those units. Keep a stable section_path (or parent id) on every chunk so sibling fetches stay deterministic when the answer needs the neighboring prerequisites chunk.
Token count is a budget, not a semantic unit.