DEV Community

Cover image for RAG Chunking Done Right: Split on Structure, Attach Metadata, and Measure Recall Before You Touch chunk_size
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

RAG Chunking Done Right: Split on Structure, Attach Metadata, and Measure Recall Before You Touch chunk_size

#llm #ai #python #rag

When a RAG system gives a wrong answer, most people go straight for the model or the prompt. In my experience, the cause is often upstream: the right passage was never retrieved, because the chunk that held it was cut in the wrong place, was too big to rank well, or had lost the context that made it findable.

Chunking is the first step of almost every RAG pipeline, and it's the one people think about least. This post is the practical version: the trade-off you're actually tuning, what overlap does and doesn't buy you, why I split on document structure instead of fixed character counts, why every chunk should carry metadata, and how to measure recall before changing anything.

(Earlier posts covered hybrid search, lost in the middle, and long context vs RAG. This one sits one step before all of them: what goes into the index in the first place.)

1. Chunk size is a precision vs. context trade-off

There's no universally "right" chunk size, because two forces pull in opposite directions:

  • Small chunks are precise. Each embedding represents one idea, so a focused query matches it cleanly and you send fewer irrelevant tokens to the model. But a small chunk can lose the context it needs to make sense on its own.
  • Large chunks keep context together, but they dilute the embedding with unrelated content and fill your context window with tokens the model doesn't need.

Chroma's research team measured this directly in their report Evaluating Chunking Strategies for Retrieval. They evaluated retrieval at the token level (recall, precision and IoU) across five corpora and found that some chunking strategies outperformed others by up to 9% in recall. Some results worth knowing:

  • With text-embedding-3-large and 5 retrieved chunks, RecursiveCharacterTextSplitter at 200 tokens with no overlap scored 88.1% recall and 7.0% precision. At 800 tokens with 400 overlap, recall was 85.4% and precision 1.5%.
  • They wrote that the heuristic recursive splitter at chunk size 200 with no overlap "performs well" and was consistently high across all their metrics.
  • They also noted that default settings of popular chunkers can perform relatively poorly. OpenAI's file search default (800-token chunks, 400-token overlap) gave slightly below-average recall and the lowest scores on their other metrics in that setup.

The speculation in that report matches what I see in practice: recall peaks somewhere in the middle. Too large and the relevant sentence is diluted; too small and the chunk doesn't carry enough context to be retrieved as one unit.

Know your defaults, because you're probably using one:

Tool Default / documented setting
OpenAI vector stores (file search) 800-token chunks, 400-token overlap; configurable from 100 to 4096 tokens, overlap at most half the chunk size (docs)
LlamaIndex SentenceSplitter Docs example uses chunk_size=1024, chunk_overlap=20 (docs)
LangChain RecursiveCharacterTextSplitter Measures size in characters by default, splits on ["\n\n", "\n", " ", ""] (docs)

Note that last one: a LangChain chunk_size=1000 is 1,000 characters, not 1,000 tokens, unless you pass a token-based length_function. I've seen teams compare "chunk sizes" across tools without noticing they were using different units.

2. Overlap is insurance, not a free upgrade

Overlap repeats the end of one chunk at the start of the next, so a sentence that falls on a boundary isn't cut in half. LangChain's docs describe it as helping "to mitigate loss of information when context is divided between chunks."

The cost is redundancy: the same tokens get stored, retrieved and sent to the model more than once. In Chroma's results, reducing overlap improved IoU because that metric penalizes repeated tokens. But overlap isn't useless: with the smaller all-MiniLM-L6-v2 embedding model, their best recall (82.4%) came from 250-token chunks with 125-token overlap, and it dropped to 77.1% with zero overlap.

My takeaway: overlap helps most when your boundaries are arbitrary. If you split on real structure (next section), you need much less of it.

3. Split on structure, not character counts

Fixed-size splitting ignores where ideas begin and end. A 500-character window will happily cut a table in half, or glue the last paragraph of "Refund policy" to the first paragraph of "Shipping."

Most real documents already tell you where the boundaries are: Markdown headings, HTML sections, PDF titles, code functions. Use them.

  • LangChain's MarkdownHeaderTextSplitter splits on the headers you specify and puts each header into the chunk's metadata (docs).
  • LlamaIndex has MarkdownNodeParser for Markdown and CodeSplitter for source code (which splits by language, with chunk_lines and max_chars limits).
  • LangChain's recursive splitter is already a mild version of this idea: it tries paragraphs first, then lines, then words.

The pattern I use is two-stage: split by section first, then apply a size limit inside each section only when a section is too long. LangChain's docs note that when you do this with split_documents, overlap is applied within a section and does not cross section boundaries, which is exactly what you want.

4. Attach metadata to every chunk

A chunk that says "The limit is 30 days" is nearly useless in isolation. Thirty days for what? In which product? Which version of the policy?

Anthropic's Contextual Retrieval post describes this exact problem with an SEC-filing example: a chunk saying "The company's revenue grew by 3% over the previous quarter" doesn't say which company or which quarter. Their fix is to prepend a short, chunk-specific context (usually 50-100 tokens, generated by Claude) before embedding and before building the BM25 index. In their tests, contextual embeddings alone reduced the top-20 retrieval failure rate by 35% (5.7% → 3.7%).

You can get a cheap version of this without an LLM call: store the document title and section path as metadata, and prepend them to the text you embed. Metadata also lets you filter (by product, version, tenant) before ranking, and it gives you citations for free.

5. Measure recall on real queries before changing chunk size

This is the step most teams skip. Changing chunk size changes every embedding in your index, so you should know whether it helped.

  1. Collect 50-100 real user questions (from logs, support tickets, or your team).
  2. For each, label the document section that actually answers it.
  3. Measure recall@k: the share of questions where a correct section appears in the top k results.
  4. Change one variable at a time (size, overlap, splitter), re-index, re-measure.

Anthropic's evaluation used exactly this kind of metric (1 minus recall@20). Chroma's report also showed you can generate an evaluation set from your own corpus with an LLM if you don't have labeled queries yet, though real queries are better.

Code: structure-aware chunks with metadata, plus a recall check 

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

def chunk_markdown(doc_title: str, markdown: str):
    # Stage 1: split on real structure; headers go into metadata
    by_section = MarkdownHeaderTextSplitter(
        headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
    ).split_text(markdown)

    # Stage 2: cap size only inside long sections (chunk_size is in characters here)
    sized = RecursiveCharacterTextSplitter(
        chunk_size=1200, chunk_overlap=100
    ).split_documents(by_section)

    for d in sized:
        section = " > ".join(d.metadata[k] for k in ("h1", "h2", "h3") if k in d.metadata)
        d.metadata.update({"doc_title": doc_title, "section": section})
        # Embed this text: title + section path + chunk body
        d.metadata["embed_text"] = f"{doc_title} | {section}\n{d.page_content}"
    return sized

def recall_at_k(eval_set, search, k=10):
    """eval_set: [(question, expected_section)], search(q, k) -> list of chunks."""
    hits = sum(
        any(c.metadata["section"] == expected for c in search(q, k))
        for q, expected in eval_set
    )
    return hits / len(eval_set)
Enter fullscreen mode Exit fullscreen mode

The chunk_size=1200 and chunk_overlap=100 values are a hypothetical starting point, not a recommendation. Run recall_at_k with your current settings first, then change one value at a time.

A hypothetical example

Say an internal HR bot keeps answering "How many days do I have to submit expenses?" with the travel booking policy. The fixed 1,000-character chunk that holds the answer starts mid-sentence, and its only identifying text is "Submit within 30 days." After switching to section-based splitting with HR Handbook | Expenses > Deadlines prepended, the chunk carries the words the question uses. You would confirm that with recall@10 on your labeled questions, not by eyeballing one answer.

Takeaways

  • Chunk size trades precision for context. There's no universal best value; measure yours.
  • Know your tool's defaults and units (tokens vs. characters).
  • Overlap is insurance for arbitrary boundaries. Splitting on structure reduces the need for it.
  • Split on headings and sections first, then cap size within sections.
  • Put doc title and section path on every chunk, and embed them with the text.
  • Measure recall@k on real queries before and after every change.

References

Top comments (0)