RAG Chunking Strategy: How to Preserve Context Instead of Destroying It

Every retrieval-augmented generation system eventually comes down to one uncomfortable truth: the model can only answer as well as the chunks it retrieves. Teams spend weeks comparing embedding models, tuning vector databases, and A/B testing prompts, then feed everything through a chunking step they wrote in twenty minutes using a default splitter with a fixed size and a fixed overlap. The result is predictable — retrieval that looks fine in a demo and falls apart the moment a real user asks a question that spans two paragraphs, references a table, or needs the heading three levels up to make sense.

A good RAG chunking strategy is not a preprocessing afterthought. It is the layer that determines what "context" even means to your retriever and your language model. Get it wrong, and you can have the best embedding model on the market and still return answers that are technically retrieved but semantically incoherent. Get it right, and even a modest embedding model paired with a well-tuned reranker will outperform a fancier stack sitting on top of sloppy chunks.

This post walks through how to think about a chunking strategy for RAG from first principles: why naive splitting breaks context, how structure-aware and semantic approaches fix it, how to handle special content like tables and code, how to tag chunks with metadata that actually helps retrieval, and how to evaluate whether your chunking is working at all. We'll include working Python examples you can adapt directly.

Why Naive Fixed-Size Chunking Breaks Semantic Context

The simplest possible approach to chunking is to slice a document every N characters or tokens, sometimes with a sliding overlap. It's fast, deterministic, and trivial to implement — which is exactly why it's the default in nearly every RAG tutorial. It is also, in most production settings, the wrong default.

Fixed-size chunking treats a document as an undifferentiated stream of characters. It has no concept of a sentence boundary, a paragraph, a heading, a list item, or a code block. The consequences show up in predictable ways:

• Mid-sentence cuts. A 512-character window can, and regularly does, end in the middle of a clause. The embedding for that chunk represents half an idea, which produces a noisy vector that doesn't cleanly match either the question that led into it or the question that would have matched the completed sentence.
• Orphaned references. A paragraph that says "as shown in the table below" is useless without the table. A step that says "repeat step 3 with the updated config" is meaningless without step 3 nearby. Fixed windows split these relationships apart at random.
• Lost hierarchy. A sentence buried three heading levels deep — say, under "Authentication > OAuth > Refresh Tokens" — carries very different meaning than the same sentence with no heading context. Naive chunking discards the heading trail entirely, so the retriever has no way to disambiguate.
• Inconsistent information density. A fixed character count treats a dense code block and a paragraph of prose identically, even though a hundred characters of code often encodes far more meaning than a hundred characters of narrative text.

None of this means fixed-size or recursive splitting is useless — it's a fine fallback for unstructured prose. The problem is applying it uniformly to every document type, which is what most default pipelines do. A serious RAG chunking strategy starts by acknowledging that different documents need different treatment, and that the goal of chunking is not "split text into pieces" but "preserve the smallest self-contained unit of meaning that a retriever can act on independently."

Structure-Aware Chunking: Respecting Headings, Paragraphs, Code, and Lists

Structure-aware chunking uses the document's own formatting signals — Markdown headings, HTML tags, PDF layout metadata, list markers, code fences — to decide where chunk boundaries should fall. Instead of asking "have we hit 500 characters yet," it asks "have we reached the end of a logical section." This is the single highest-leverage change most teams can make to their RAG chunking strategy, because it directly addresses the orphaned-reference and lost-hierarchy problems above.

The general algorithm looks like this:

1. Parse the document into a tree of structural elements (headings, paragraphs, list items, tables, code blocks).
2. Walk the tree and accumulate elements into a chunk until adding the next element would exceed a target size.
3. Never split inside an atomic element — a code block, a table row, a list item — unless it is itself larger than the maximum chunk size.
4. Attach the active heading path to every chunk so retrieval-time context is preserved even after the chunk is pulled out of the document.

Here is a working example of a structure-aware Markdown chunker that respects headings, keeps code blocks intact, and carries the heading trail forward as metadata:

import re
from dataclasses import dataclass, field

@dataclass
class MarkdownChunk:
    text: str
    heading_path: list = field(default_factory=list)
    char_count: int = 0

def chunk_markdown(document: str, max_chars: int = 1200) -> list[MarkdownChunk]:
    heading_re = re.compile(r'^(#{1,6})\s+(.*)$')
    lines = document.splitlines()

    chunks: list[MarkdownChunk] = []
    heading_stack: list[str] = []
    buffer: list[str] = []

    def flush():
        text = "\n".join(buffer).strip()
        if text:
            chunks.append(MarkdownChunk(
                text=text,
                heading_path=heading_stack.copy(),
                char_count=len(text),
            ))
        buffer.clear()

    i = 0
    while i < len(lines):
        line = lines[i]

        if line.strip().startswith("```

"):
            code_lines = [line]
            i += 1
            while i < len(lines) and not lines[i].strip().startswith("

```"):
                code_lines.append(lines[i])
                i += 1
            if i < len(lines):
                code_lines.append(lines[i])
            code_block = "\n".join(code_lines)

            if sum(len(b) for b in buffer) + len(code_block) > max_chars:
                flush()
            buffer.append(code_block)
            i += 1
            continue

        heading_match = heading_re.match(line)
        if heading_match:
            level = len(heading_match.group(1))
            title = heading_match.group(2).strip()
            flush()
            heading_stack = heading_stack[: level - 1]
            heading_stack.append(title)
            buffer.append(line)
            i += 1
            continue

        prospective_len = sum(len(b) for b in buffer) + len(line)
        if prospective_len > max_chars and line.strip() == "":
            flush()
        else:
            buffer.append(line)
        i += 1

    flush()
    return chunks

Notice what this buys you over naive splitting: every chunk knows exactly which section it came from, code blocks are never truncated mid-function, and section boundaries are respected even when a heading has very little content underneath it. When you embed the chunk, you can prepend the heading path directly into the text sent to the embedding model, which meaningfully improves retrieval for section-specific queries.

For HTML or PDF-derived documents, the same principle applies — you just need a different structural parser. The point is the same across formats: chunk on the document's actual structure, not on an arbitrary character count.

Chunk Size and Overlap Tradeoffs

Even after you adopt structure-aware chunking, you still need to pick a target size and decide whether to overlap adjacent chunks. There's no universally correct number, but the tradeoffs are well understood:

• Smaller chunks (100–300 tokens) produce more precise embeddings because each vector represents a narrower, more specific idea. This improves retrieval precision — you're less likely to retrieve a chunk that's "sort of" relevant because it contains one relevant sentence buried in four irrelevant ones. The cost is that small chunks often lack enough context to be useful on their own.
• Larger chunks (500–1000+ tokens) preserve more context and reduce the chance that an answer's supporting evidence is split across multiple chunks. But the embedding for a large chunk is an average over many different ideas, which dilutes similarity scores and hurts retrieval recall for narrow, specific queries.
• Overlap (commonly 10–20% of chunk size) reduces the chance that a fact gets fully severed at a boundary. The cost is redundant storage, redundant embeddings, and a higher chance of returning near-duplicate chunks in your top-k results.

A practical starting point for a RAG chunking strategy handling technical documentation is 300–500 tokens per chunk with roughly 10–15% overlap, then tuning based on evaluation results. Q&A-style or FAQ content often works better with smaller, near-atomic chunks and little to no overlap. Long-form narrative content — reports, contracts, transcripts — tends to benefit from slightly larger chunks because meaning accumulates over several sentences.

The mistake to avoid is picking one size and overlap value for an entire heterogeneous corpus. A single global setting is a symptom of not having a real chunking strategy for RAG at all.

Semantic Chunking vs. Recursive Character Splitting

Recursive character splitting — as implemented by libraries like LangChain's RecursiveCharacterTextSplitter — tries to split on a prioritized list of separators, falling back to a coarser separator only when a finer one isn't available within the size budget. It's a meaningful improvement over pure fixed-size slicing because it at least tries to respect natural language boundaries, but it still doesn't understand meaning — it's purely syntactic.

Semantic chunking goes a step further: it embeds sentences or small groups of sentences, then looks for points where the embedding similarity between consecutive segments drops sharply, and treats those drop points as chunk boundaries.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunk(sentences, threshold=0.30, min_sentences=2, max_sentences=12):
    embeddings = model.encode(sentences, normalize_embeddings=True)

    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = float(np.dot(embeddings[i - 1], embeddings[i]))
        should_break = (
            sim < threshold and len(current) >= min_sentences
        ) or len(current) >= max_sentences

        if should_break:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])

    if current:
        chunks.append(" ".join(current))
    return chunks

Semantic chunking tends to outperform recursive splitting on prose-heavy content where topic shifts don't line up neatly with paragraph breaks — think transcripts, long-form articles, or legal text. It costs more to compute and it's less predictable in chunk size.

In practice, most production-grade RAG chunking strategies are hybrids: structure-aware splitting handles the macro level, and semantic or recursive splitting handles the micro level. Treating semantic chunking as a wholesale replacement for structure awareness is usually a mistake — a heading boundary is a stronger and cheaper signal than an embedding similarity dip.

Metadata Tagging: Source, Section, and Page

A chunk without metadata is a chunk you can't reason about, debug, or filter. At minimum, every chunk in a well-designed RAG chunking strategy should carry enough metadata to answer three questions after retrieval: where did this come from, what part of the document is it, and can I cite it back to the user?

from dataclasses import dataclass, field
from datetime import datetime, timezone
from hashlib import sha256

@dataclass
class ChunkMetadata:
    chunk_id: str
    source_id: str
    source_uri: str
    document_title: str
    section_path: list
    page_number: int
    chunk_index: int
    content_type: str
    language: str
    token_count: int
    created_at: str
    checksum: str

def build_chunk_metadata(text, source_id, source_uri, document_title,
                          section_path, chunk_index, content_type="prose",
                          page_number=None, language=None, token_count=0):
    checksum = sha256(text.encode("utf-8")).hexdigest()[:16]
    return ChunkMetadata(
        chunk_id=f"{source_id}-{chunk_index:04d}-{checksum}",
        source_id=source_id,
        source_uri=source_uri,
        document_title=document_title,
        section_path=section_path,
        page_number=page_number,
        chunk_index=chunk_index,
        content_type=content_type,
        language=language,
        token_count=token_count,
        created_at=datetime.now(timezone.utc).isoformat(),
        checksum=checksum,
    )

This pays off in several concrete ways. section_path lets you re-inject heading context into the prompt or filter retrieval to a specific section. page_number lets you generate accurate citations for PDF-derived answers. content_type lets you apply different reranking or formatting logic to code versus prose versus tables. checksum lets you detect duplicate chunks across re-ingestions.

Metadata is also what makes hybrid retrieval possible — combining vector similarity with metadata filters is usually cheaper and more reliable than trying to encode all of that nuance into the embedding itself.

Embedding Model Context Window Considerations

Your chunk size decision is not independent of your embedding model. Most embedding models have a maximum input length — often 512 tokens for older sentence-transformer models, up to 8,000+ tokens for newer long-context embedding models. Two failure modes show up when this isn't accounted for.

First, if your chunks exceed the model's max input length, the model silently truncates the input, and everything after the cutoff is simply ignored when computing the embedding. Second, even when a model technically accepts longer input, embedding quality for semantic search often degrades on very long chunks because the single output vector has to summarize a wider spread of content.

A safe rule of thumb: target chunk sizes at 50-70% of your embedding model's max context window, not the full window. This leaves headroom for the heading-path prefix and any metadata you prepend to the chunk text before embedding, and it avoids operating at the truncation boundary where behavior can be inconsistent across providers.

Evaluating Chunking Quality: Retrieval Precision and Recall

You cannot tune a chunking strategy for RAG by intuition alone — you need a way to measure whether a given configuration actually improves retrieval. The standard approach is to build a small evaluation set of representative queries, each paired with the ground-truth chunk(s) that should be retrieved to answer it.

def evaluate_retrieval(eval_set, retriever, k=5):
    precisions, recalls, mrrs = [], [], []

    for item in eval_set:
        retrieved = retriever(item["query"], k=k)
        relevant = item["relevant_chunk_ids"]

        hits = [c for c in retrieved if c in relevant]
        precisions.append(len(hits) / k)
        recalls.append(len(hits) / len(relevant) if relevant else 0.0)

        rr = 0.0
        for rank, chunk_id in enumerate(retrieved, start=1):
            if chunk_id in relevant:
                rr = 1.0 / rank
                break
        mrrs.append(rr)

    return {
        "precision_at_k": sum(precisions) / len(precisions),
        "recall_at_k": sum(recalls) / len(recalls),
        "mrr": sum(mrrs) / len(mrrs),
    }

Precision@k tells you how much of what you retrieve is actually useful. Recall@k tells you whether the relevant information is being found at all. Mean reciprocal rank (MRR) tells you how close the best relevant result is to the top of the list.

Run this evaluation across a matrix of chunk sizes, overlap ratios, and splitting strategies on the same document set, and you'll usually find the "best" configuration is document-type specific rather than universal.

Handling Tables and Code Specially

Tables and code blocks are the two content types most consistently destroyed by generic chunking, and they deserve dedicated handling in any serious chunking strategy for RAG.

For tables, splitting a table by character count almost always separates header rows from data rows, which makes each resulting chunk unreadable in isolation. The fix is to treat each table as an atomic unit if it's small enough to fit within a chunk, and if it's too large, split by row while repeating the header row at the top of every resulting chunk.

For code, chunking should never split inside a function, class, or fenced code block — a truncated function is both semantically meaningless and often syntactically invalid. Where possible, use a language-aware parser to chunk at function or class boundaries, and always retain the surrounding prose that explains what the code does.

A useful general pattern is content-type-aware routing: detect whether a document segment is prose, table, code, or list before deciding how to chunk it, rather than applying one splitting function to the whole document uniformly.

Chunking for Different Document Types

The right RAG chunking strategy varies meaningfully by source format:

• Markdown / structured docs. Use the heading-and-fence-aware approach shown earlier. Headings are a strong, cheap signal.
• PDFs. PDFs are the hardest common case because layout information is often only recoverable through a dedicated parsing library. Always preserve page numbers in metadata for citation purposes.
• Meeting or call transcripts. Transcripts lack headings almost entirely, so structure-aware chunking has little to work with. Semantic chunking or speaker-turn-aware chunking tends to outperform fixed-size splitting significantly.
• HTML / web content. Strip navigation, ads, and boilerplate before chunking — this "content extraction" step matters as much as the chunking algorithm itself.
• Structured records (JSON, database exports, tickets, emails). Often best chunked at the record level rather than by character count.

A Practical Decision Framework

When you're designing a chunking strategy for RAG from scratch, it helps to work through a short sequence of questions rather than reaching for a default splitter:

1. What structure does the source format actually offer? If there are headings, use them as primary boundaries.
2. What content types are embedded in the document? Identify tables and code up front and route them through dedicated atomic-chunking logic.
3. What's the typical query pattern? Narrow, fact-lookup queries favor smaller chunks and higher precision.
4. What are the embedding and generation model's context limits? Size chunks to leave headroom below the embedding model's max input.
5. What metadata do you need for citation and filtering? Decide this before ingestion, not after.
6. How will you measure whether it's working? Build the evaluation set before you optimize.

None of these steps are exotic engineering — they're deliberate design decisions that most default RAG tutorials skip in favor of getting a demo running quickly. The gap between a demo-quality RAG chunking strategy and a production-quality one is almost entirely in this layer.

For background on the underlying vector search mechanics that make any RAG chunking strategy effective at retrieval time, see Pinecone's overview of vector search fundamentals. Combined with the structure-aware and semantic chunking approach covered above, a solid grasp of vector search helps you reason about why a given chunking choice moves retrieval quality up or down.

Getting your RAG chunking strategy right is not optional if you want reliable retrieval. A well-tuned chunking strategy reduces hallucinations, a poor one multiplies them, and every production RAG pipeline should have its chunking benchmarked before it ships.

Getting Chunking Right the First Time

Chunking sits at the intersection of document understanding, information retrieval, and prompt engineering, which is exactly why it's easy to underinvest in and expensive to fix later. Teams that treat their RAG chunking strategy as a first-class design decision, evaluated with real precision and recall metrics rather than eyeballed on a handful of test queries, consistently ship retrieval systems that hold up under real user questions instead of just demo scripts.

If you're building or hardening a retrieval-augmented system and want an experienced team to help design the ingestion pipeline, chunking strategy, metadata schema, and evaluation harness end to end, my AI integration services work covers exactly this kind of problem for engineering organizations that need RAG to work reliably in production, not just in a demo.