Hu Jin

Posted on Nov 13

Chunking Strategies for RAG: Matching Split Rules to Document Types

#ai #rag #chunking #embeddings

Chunking is the quiet bottleneck in most RAG systems. Embeddings and rerankers cannot recover meaning you already destroyed by splitting in the wrong place. A fixed 500-character window is fine for a demo; production corpora need rules that follow document structure.

This article maps common document types to practical split strategies, then covers size, overlap, and metadata that make those chunks usable at query time.

Start from structure, not character count

Humans write in headings, paragraphs, functions, and messages. Retrievers work best when each chunk is roughly one coherent idea plus enough surrounding context to stand alone. Prefer semantic boundaries over arbitrary byte offsets.

Strategies by document type

Markdown and HTML. Split on heading hierarchy (h1 → h2 → h3). Keep the heading path in the chunk text or metadata (section: Billing / Refunds). Merge tiny sections; split huge ones at paragraph boundaries under the same heading.

PDF reports and Word docs. Extract text carefully (layout matters). Prefer paragraph or section splits after cleaning headers/footers. For slide decks, one slide (or one slide + speaker notes) is often the right unit.

Source code. Split by function, method, or class—not by lines alone. Include the signature and a short file/module path in every chunk. For docs-in-code (README, docstrings), treat them as Markdown.

Chat logs, tickets, and email threads. Chunk by message or by a small window of turns (for example, question + next two replies). Preserve speaker, timestamp, and ticket ID in metadata so the model can cite them.

Tables and CSV-heavy pages. Do not flatten a 200-row table into one giant string if queries target individual rows. Chunk by logical row groups, and repeat column headers in each chunk so "Q3 revenue" still makes sense without the table start.

Mixed docs (wiki pages). Detect blocks: prose vs code fence vs table. Apply the matching rule per block, then attach shared page metadata.

Size and overlap

A useful default for many English knowledge bases:

Target size: about 300–800 tokens per chunk.
Overlap: about 10–20% so sentences that straddle a boundary appear in both neighbors.
Hard max: stay under your embedding model's limit with room for the query and instruction tokens later.

Too small → lost context and noisy retrieval. Too large → diluted similarity scores and wasted prompt space. Tune with recall@k on a labeled set rather than gut feel alone.

Metadata worth storing

Every chunk should carry more than raw text:

source / URL / doc ID
title and section path
page or line_start / line_end
doc_type (policy, API, ticket, code)
acl / tenant / sensitivity labels for filtered retrieval

At query time, filter on metadata first when you can (product, region, permissions), then rank the remaining chunks.

A simple implementation pattern

def chunk_markdown(md: str, max_tokens=600, overlap_ratio=0.15):
    sections = split_on_headings(md)  # keep heading path with each section
    chunks = []
    for section in sections:
        if token_len(section) <= max_tokens:
            chunks.append(section)
        else:
            chunks.extend(window_split(section, max_tokens, overlap_ratio))
    return chunks

Use a tokenizer close to your embedding model when measuring size. Character length is a rough proxy only.

Pitfalls to avoid

Splitting mid-sentence or mid-table row without overlap.
Stripping headings so every chunk looks like anonymous prose.
One strategy for every file type in a mixed corpus.
Re-chunking without re-embedding after you change rules—old vectors no longer match the new text units.
Ignoring permissions at chunk level; security filters belong in the index metadata.

Good chunking is domain-specific engineering, not a single magic number. Match the split to the document type, keep overlap and metadata honest, and measure retrieval before you blame the model.

About the author

I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.

DEV Community