- Book: RAG Pocket Guide: Retrieval, Chunking, and Reranking Patterns for Production
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You ship a RAG bot. The retriever scores well on Recall@5. Then a customer support ticket lands: the bot answered a config question with the phrase "set the timeout to" and stopped there. No number. No unit. Just a sentence that ends mid-clause. You pull the trace. The top-ranked chunk reads:
...connections idle for too long. To prevent this, set the timeout to
That is the entire chunk. The next chunk in the document, the one that actually contains "30 seconds and enable keepalive", was never retrieved. The chunker split a single sentence across two chunks, the embedder vectorized both halves, and the retriever ranked the first half above the second because the first half contains the trigger phrase the user query matched on. The model was handed a context window that ends in a comma and a dangling preposition and did the only thing it could: paraphrased the half-sentence and called it an answer.
This is the boundary-bug failure mode, the most under-instrumented bug in the average RAG pipeline. Recall@K is silent. Faithfulness scores look fine because the half-sentence is in the context. Users notice it before any dashboard does.
How chunkers create half-sentences
Most production chunkers fall into one of three buckets, and all three can break boundaries.
Fixed-size character chunkers. Take a document, slide a window of N characters with a stride of M, emit each window. LangChain's CharacterTextSplitter is the canonical example. Cheap, deterministic, and indifferent to what is inside the window. A chunk of 500 characters that lands in the middle of a sentence ends in the middle of a sentence. Add overlap and you get half-sentences in two chunks instead of one.
Token-count chunkers. Same shape, different unit. Count tokens with tiktoken or the embedder's tokenizer, emit windows of N tokens. The breakage is identical. A token boundary in BPE tokenizers does not align with a word boundary, let alone a sentence boundary. You can split the word "timeout" into "time" and "out" and watch the embedder happily encode the suffix as if it were a standalone token.
Recursive chunkers with a separator list. LangChain's RecursiveCharacterTextSplitter tries ["\n\n", "\n", " ", ""] in order, falling back to the next separator when a chunk is still too large. This is the most common default in 2026 and it is better than the other two. It still splits on whitespace as a last resort, which is the half-sentence problem. A 500-character paragraph with no internal newlines will be split on a space, mid-sentence, every time.
The pattern across all three: the chunker has a size budget and no syntactic awareness. It honors the budget. It does not honor sentences, paragraphs, code fences, list items, or bracket pairs.
What the LLM does with a broken chunk
A model handed a context that ends in "set the timeout to" will not refuse. The 2026 generation of frontier models will:
- Paraphrase the fragment. "The timeout should be configured according to the documentation." Confident and useless. Indistinguishable from a real answer in any superficial eval.
- Hallucinate the missing tail. "Set the timeout to 60 seconds." The number was never in the context. The model filled in a plausible default from its prior. Faithfulness checks catch this if you run them per-claim, miss it if you check the answer as a whole.
- Mention the cutoff. "The retrieved context appears to be truncated." The most useful failure mode and the rarest. Frontier chat models in 2024-2025 have become more likely to flag truncation, but it remains unreliable, and it does not happen at all when the cutoff is a comma rather than an obvious mid-word break.
The first two are silent failures. The third is a noisy failure that your users will still log as a bug.
The deeper problem is that retrieval ranks chunks by similarity to the query, and a half-sentence containing the keyword often ranks higher than the complete sentence three chunks later, because the half-sentence is denser. Less surrounding context dilutes the embedding. So the broken chunk wins. The complete one loses.
Detection: validate every chunk before it hits the index
Before you fix the chunker, instrument the existing one. The cheapest, highest-signal check is a boundary validator that runs over every chunk at index time and emits structured warnings to your logs. Three checks cover most of the damage:
import re
from dataclasses import dataclass
@dataclass
class ChunkIssue:
chunk_id: str
kind: str
detail: str
SENTENCE_END = re.compile(r'[.!?][\"\')\]]?\s*$')
SENTENCE_START = re.compile(r'^[\"\'(\[]?[A-Z0-9]')
def boundary_issues(
chunk_id: str,
text: str,
) -> list[ChunkIssue]:
issues: list[ChunkIssue] = []
stripped = text.strip()
if not stripped:
return issues
if not SENTENCE_END.search(stripped):
last = stripped[-40:]
issues.append(ChunkIssue(
chunk_id, "no-sentence-end", last,
))
if not SENTENCE_START.match(stripped):
first = stripped[:40]
issues.append(ChunkIssue(
chunk_id, "no-sentence-start", first,
))
pairs = [("(", ")"), ("[", "]"), ("{", "}")]
for open_c, close_c in pairs:
if stripped.count(open_c) != stripped.count(close_c):
issues.append(ChunkIssue(
chunk_id,
f"unbalanced-{open_c}{close_c}",
f"{stripped.count(open_c)}/{stripped.count(close_c)}",
))
fences = stripped.count("`" * 3)
if fences % 2 != 0:
issues.append(ChunkIssue(
chunk_id, "broken-code-fence", str(fences),
))
return issues
Run this over a chunked corpus and 10-30% of chunks will fail at least one check. The rate depends on the source. Markdown with code blocks fails the fence check often. Legal text with parenthetical citations fails the bracket check. Plain prose fails the sentence-boundary check whenever a fixed-size splitter ran over it.
The point of the validator is not to reject chunks. It is to give you a number. "32% of our chunks end mid-sentence" is a metric you can chart, alert on, and watch fall when you fix the chunker.
The next step up is to log the validator output as structured events on a sample of every indexing run. Tag each event with the source document, the splitter version, and the chunk size. Then a regression in the chunker (someone bumps chunk_size from 500 to 800 to "improve recall") shows up as a step change in the boundary-issue rate and you catch it the same day, not three weeks later when a user types a question that lands on one of the new broken chunks.
Fix one: structure-aware splitters
The core fix is to teach the splitter what the document is. A markdown document is not a 50,000-character string. It is a tree of headings, paragraphs, list items, and code blocks. Splitting at the boundaries of those nodes produces chunks that are syntactically complete by construction.
LangChain's MarkdownHeaderTextSplitter splits on heading levels and attaches the heading hierarchy to each chunk's metadata. A chunk pulled from ## Configuration > ### Timeouts carries that path along with it, which the retriever can also score against. The chunks are paragraph-aligned because the splitter never crosses a heading or paragraph break.
For HTML, the same idea applies with HTMLHeaderTextSplitter. For Python or other code, language-aware splitters in the same family split on top-level definitions (functions, classes) so a chunk never ends in the middle of a function body.
A minimal hand-rolled markdown splitter, for the case where you want to control what counts as a boundary:
import re
from dataclasses import dataclass
@dataclass
class Chunk:
text: str
heading_path: list[str]
HEADING = re.compile(r"^(#{1,6})\s+(.+?)\s*$", re.MULTILINE)
def split_markdown(
md: str,
max_chars: int = 1200,
) -> list[Chunk]:
headings: list[tuple[int, int, str]] = []
for m in HEADING.finditer(md):
level = len(m.group(1))
headings.append((m.start(), level, m.group(2)))
headings.append((len(md), 0, ""))
chunks: list[Chunk] = []
path: list[str] = []
for i in range(len(headings) - 1):
start, level, title = headings[i]
end, _, _ = headings[i + 1]
path = path[: max(level - 1, 0)] + [title] if title else path
body = md[start:end].strip()
if not body:
continue
if len(body) <= max_chars:
chunks.append(Chunk(body, list(path)))
continue
for para in re.split(r"\n\s*\n", body):
para = para.strip()
if para:
chunks.append(Chunk(para, list(path)))
return chunks
Two things this does that a fixed-size splitter does not. First, it never crosses a heading boundary, so the heading hierarchy is preserved as metadata. Second, when a section is too large for one chunk, it falls back to splitting on blank lines (paragraph boundaries) rather than character count. A paragraph that is itself longer than max_chars will produce an over-sized chunk; that is a deliberate choice. Two half-sentence chunks of the right size are worse than one over-sized complete chunk.
For code, the equivalent is an AST-aware splitter. Tree-sitter has parsers for most languages and gives you a structured tree to walk. Split on top-level nodes (functions, classes, methods), keep imports as a header chunk, and never split inside a function body. The chunks come out longer than a fixed-size splitter would emit, but they are complete units of code, which is what the retriever and the model both need.
Fix two: overlap windows that actually help
Overlap was supposed to fix this. The standard advice (chunk_overlap=50 so each chunk shares its last 50 characters with the next) does not survive the boundary problem. If chunk N ends mid-sentence, chunk N+1 starting at character N_end - 50 also starts mid-sentence. The overlap gives you two chunks that share the same broken half.
Useful overlap is sentence-level, not character-level. The next chunk starts at the most recent sentence boundary before the cutoff. Both chunks are sentence-aligned, and the overlap region contains complete sentences the previous chunk also covered.
import re
SENTENCE = re.compile(r"(?<=[.!?])\s+(?=[A-Z0-9])")
def sentence_aligned_chunks(
text: str,
target_chars: int = 1000,
overlap_sentences: int = 2,
) -> list[str]:
sentences = SENTENCE.split(text.strip())
chunks: list[str] = []
buf: list[str] = []
size = 0
for s in sentences:
if size + len(s) > target_chars and buf:
chunks.append(" ".join(buf))
buf = buf[-overlap_sentences:]
size = sum(len(x) for x in buf)
buf.append(s)
size += len(s)
if buf:
chunks.append(" ".join(buf))
return chunks
The regex is intentionally simple. It will fight you on common abbreviations ("Dr. Smith", "e.g.") and on numbered lists. For production, use a sentence segmenter that knows about abbreviations: pysbd, nltk's Punkt, or spaCy. The cost is a heavier dependency for a job a 30-line splitter can almost do.
Fix three: post-chunk merge
The third pattern most retrieval systems need: a merge step that runs after retrieval, before the prompt is built.
After the retriever returns its top-K chunks, walk them in document order and check whether any two are adjacent. If chunk N and chunk N+1 from the same document both made the cut, concatenate them in the prompt rather than presenting them as two separate chunks. The model gets a longer, complete passage; the retriever gets credit for finding both halves of a two-chunk argument; the boundary bug heals at read-time even when the index step missed it.
def merge_adjacent(retrieved: list[dict]) -> list[dict]:
by_doc: dict[str, list[dict]] = {}
for c in retrieved:
by_doc.setdefault(c["doc_id"], []).append(c)
merged: list[dict] = []
for doc_id, chunks in by_doc.items():
chunks.sort(key=lambda c: c["chunk_index"])
run = [chunks[0]]
for c in chunks[1:]:
if c["chunk_index"] == run[-1]["chunk_index"] + 1:
run.append(c)
else:
merged.append(_concat(run))
run = [c]
merged.append(_concat(run))
return merged
def _concat(run: list[dict]) -> dict:
return {
"doc_id": run[0]["doc_id"],
"chunk_index": run[0]["chunk_index"],
"text": " ".join(c["text"] for c in run),
"score": max(c["score"] for c in run),
}
The chunk_index field is something you set at index time: a monotonic integer per document. Without it, "adjacent" is undefined and you cannot merge safely.
What to do with this on Monday
The order that pays back fastest, with the least disruption to your pipeline:
- Run the boundary validator over your existing index. Get a number. If it is over 5%, the rest of the list is worth doing.
- Switch to a structure-aware splitter on the next reindex. Markdown-aware for documentation, AST-aware for code, sentence-aligned for prose with no markup.
- Replace character-overlap with sentence-overlap. Two-sentence overlap on a sentence-aligned splitter is sufficient for almost every retrieval shape.
- Add the post-retrieval merge step. It is fifteen lines of code and it heals two-chunk answers without any reindex work.
The chunker is the part of a RAG pipeline that gets the least review and produces the most silent failures. Validate it like you would validate any other production input.
If this was useful
The RAG Pocket Guide covers chunking strategies as a system, not as a single splitter choice — structure-aware splitting, overlap design, the merge step, and the eval slice that catches boundary regressions before users do. If your RAG pipeline is producing answers that "feel half-finished," the chunking chapters are the place to start.
Top comments (0)