Your RAG Chunks Are Cut Mid-Sentence: A Small Script to Find Them

Here's a question a docs assistant should get right: "How long does the old API key keep working after I rotate it?" The docs page says it plainly: 24 hours. The assistant still answers vaguely, or not at all. The retriever looks fine, the prompt looks fine, and the page is definitely indexed.

Then you print the chunks, and the sentence about 24 hours starts with "y keeps working". The word "Rotate" is in a different chunk. So is the heading. The answer is in your index, but no single chunk contains both the question's topic and the fact.

In an earlier article I covered the debug order for bad RAG answers: check retrieval before you touch the prompt. This one zooms in on the first stage of that pipeline, chunking, and gives you a small script that shows where your cuts land before you spend time on embeddings, rerankers or prompts.

Why chunking deserves its own check

A retrieval-augmented generation (RAG) system splits documents into chunks, embeds each one, and later retrieves the chunks closest to a question. Everything downstream works on those chunks. A retriever can only return text that exists as a chunk, and the model can only answer from the chunks it receives.

That makes chunk boundaries a quiet, upstream source of errors. Three problems show up again and again:

• Cut mid-sentence. A fixed character count doesn't care about sentences, so a fact gets split across two chunks and neither half is complete.
• Lost heading. The text says "click Rotate, then confirm", but the heading that said which key and which screen landed in the previous chunk. Anthropic's write-up on Contextual Retrieval describes this problem directly: individual chunks can lack the context of the document they came from.
• Too small or too large. A chunk of four words carries no meaning; a chunk covering five sections matches everything a little and nothing well.

None of these produce an error message. You only see them if you look at the chunks.

The sample document

To make this concrete, here's a short, made-up docs page. It's an example written for this article, not real product documentation:

# API keys

## Rotate a key

Open Settings, then API keys. Find the key you want to replace and click Rotate. The old key keeps working for 24 hours so you can update your services.

## Revoke a key

Click Revoke next to the key. Revoking is immediate: every request that uses the key fails from that moment. You cannot undo it.

## Limits

Each project can have up to 5 active keys. Rotating a key does not count as a new key.

A small chunk check in Python

The script below splits the same text two ways and flags each chunk. It's a sketch for inspection, not a production splitter, and it uses only the Python standard library.

import re
import sys

def fixed_size(text, size):
    """Cut every `size` characters, no overlap."""
    return [text[i:i + size] for i in range(0, len(text), size)]

def by_heading(text):
    """Split on markdown headings; keep the heading path with each chunk."""
    chunks, path, body = [], {}, []

    def flush():
        content = " ".join(body).strip()
        if content:
            heading = " > ".join(path[k] for k in sorted(path))
            chunks.append(f"[{heading}] {content}")
        body.clear()

    for line in text.splitlines():
        m = re.match(r"^(#+)\s+(.*)", line)
        if m:
            flush()
            level = len(m.group(1))
            path = {k: v for k, v in path.items() if k < level}
            path[level] = m.group(2)
        elif line.strip():
            body.append(line.strip())
    flush()
    return chunks

def flags(chunk, is_last):
    text = chunk.strip()
    out = []
    if not text.startswith(("#", "[")):
        out.append("NO HEADING")
    if not is_last and not text.endswith((".", "!", "?", ":")):
        out.append("CUT MID-SENTENCE")
    if len(text.split()) < 8:
        out.append("TOO SMALL")
    return out

def report(name, chunks):
    print(f"== {name}: {len(chunks)} chunks")
    for i, c in enumerate(chunks, 1):
        f = flags(c, i == len(chunks))
        one_line = " ".join(c.split())
        print(f"C{i} ({len(c.split())} words) {f or ['ok']}")
        print(f"   starts: {one_line[:38]!r}")
        print(f"   ends:   {one_line[-38:]!r}")

text = open(sys.argv[1]).read()
report("fixed-size, 120 chars", fixed_size(text, 120))
print()
report("by heading", by_heading(text))

The flags are deliberately simple heuristics:

• NO HEADING: the chunk doesn't start with a heading or a heading path, so it may not make sense on its own.
• CUT MID-SENTENCE: the chunk doesn't end with sentence punctuation (the last chunk is exempt).
• TOO SMALL: fewer than 8 words. Pick a threshold that fits your content.

What it printed

I ran it with python3 chunk_check.py sample.md (Python 3.13.5). This is the actual output:

== fixed-size, 120 chars: 4 chunks
C1 (25 words) ['CUT MID-SENTENCE']
   starts: '# API keys ## Rotate a key Open Settin'
   ends:   'o replace and click Rotate. The old ke'
C2 (24 words) ['NO HEADING', 'CUT MID-SENTENCE']
   starts: 'y keeps working for 24 hours so you ca'
   ends:   'ick Revoke next to the key. Revoking i'
C3 (22 words) ['NO HEADING', 'CUT MID-SENTENCE']
   starts: 's immediate: every request that uses t'
   ends:   'undo it. ## Limits Each project can ha'
C4 (16 words) ['NO HEADING']
   starts: 've up to 5 active keys. Rotating a key'
   ends:   'ing a key does not count as a new key.'

== by heading: 3 chunks
C1 (35 words) ['ok']
   starts: '[API keys > Rotate a key] Open Setting'
   ends:   'hours so you can update your services.'
C2 (29 words) ['ok']
   starts: '[API keys > Revoke a key] Click Revoke'
   ends:   ' from that moment. You cannot undo it.'
C3 (23 words) ['ok']
   starts: '[API keys > Limits] Each project can h'
   ends:   'ing a key does not count as a new key.'

Read the fixed-size result the way your retriever sees it:

• The "24 hours" fact sits in C2, which starts with "y keeps working" and goes on to talk about revoking. It has no "Rotate a key" heading. A question about rotation now has to match a chunk that is mostly about revocation.
• C3 mixes the end of the revoke section with the start of the limits section.
• C4 begins with "ve up to 5 active keys", and the "Limits" heading is at the end of C3.

The heading-based split gives three chunks, one per section, each starting with its heading path. The rotation question and the 24-hour fact are now in the same chunk, with the section title attached.

This doesn't mean fixed-size chunking is always wrong, or that splitting on headings is always right. A section can be far too long to embed as one chunk, and plenty of documents have no headings at all. The point is narrower: look at where the cuts land on your own documents before you tune anything downstream.

Turn it into a habit

A one-off look is useful. A repeatable check is better. A few ways to fold this into your pipeline:

1. Run the check on a real sample. Take 10 to 20 representative documents, including the messy ones: long tables, FAQs, changelogs, PDFs converted to text.
2. Count the flags, then read the flagged chunks. A high share of mid-sentence cuts tells you to change strategy; a handful of flagged chunks tells you which documents need special handling.
3. Keep the heading path with every chunk. Even if you keep a size limit, prepending the section title gives each chunk a fighting chance to stand on its own.
4. Split long sections on paragraphs, not characters. Only fall back to a hard size cut when a single paragraph is too long.
5. Re-run your retrieval test after any chunking change. Chunking changes what retrieval can find. A small recall@k test, like the one in the earlier article, tells you whether the change actually helped on your questions.

A checklist before you touch embeddings or prompts

• Print a sample of chunks and read them like a stranger would.
• Check for mid-sentence cuts, missing headings, and chunks that are too small or too large.
• Make sure the facts users ask about sit in the same chunk as the topic they ask about.
• Attach the section title (or heading path) to each chunk.
• Change one chunking setting at a time and re-run your retrieval test after each change.

Where to go next

If you'd rather see your cuts than read printouts, the free RAG Chunk Visualizer & Cost Calculator demo does this in the browser: paste a document, pick structure-aware or fixed-size chunking, and see each chunk as a colored block with a token estimate and quality badges for chunks that are too small, too large, or cut mid-sentence. Your text stays on your device.

For the mental model behind it, the free RAG Quick-Start covers the five-step pipeline (chunk, embed, retrieve, rerank, generate) and the three decisions that move quality most, starting with structure-aware chunking, in 6 pages.

When you're ready for the whole territory, The Complete Guide to Retrieval-Augmented Generation spends 46 pages on every stage, including chunking that respects your documents' structure, hybrid search, evaluating retrieval and generation separately, and a full worked example.

This article was written with the assistance of AI and reviewed by the author. The sample document is made up for this article; the script is the author's own sketch and the output shown is its real output on that sample only.