Most teams debugging a disappointing RAG system start at the end of the pipeline. They swap the model, tune the prompt, raise top k. The answer quality barely moves, because the problem arrived long before the query did.
A retrieval system can only return what it indexed. Everything downstream inherits whatever the ingestion step produced, and no amount of reranking repairs a chunk that was scrambled on the way in.
Ingestion Sets The Ceiling
Parsing is the step most teams underestimate. Plain text and Markdown are simple. PDFs, which hold the majority of enterprise knowledge, are not. Multi column layouts, repeating headers and footers, tables whose cell boundaries are implicit, images with text baked in, nested sections with inconsistent formatting. Naive extraction with a general purpose library produces text with merged columns, broken paragraphs and lost table structure.
The damaging part is that a bad parse does not throw. It becomes a chunk, gets embedded, and sits in the index looking exactly as trustworthy as everything around it. Layout aware parsers exist for this reason, and so does the cleaning pass that strips boilerplate, page numbers and repeated headers before anything gets embedded.
Metadata belongs to this stage too. Every chunk should carry its source document, the section heading it came from, its page or location, and a modification date. That is what makes filtered retrieval possible later, and it is what lets an answer be traced back to a specific page instead of arriving as an anonymous fragment.
Chunking Is The Highest Leverage Parameter
Chunking is the single most impactful decision in the pipeline. The tension is simple: small chunks retrieve precisely but lose surrounding context, large chunks keep context but retrieve imprecisely and eat the context window.
Fixed size chunking, typically 256 to 1024 tokens with 10 to 20 percent overlap, is a reasonable baseline for uniform prose. It fails on structured documents, where it splits tables down the middle and separates a heading from the content it introduces.
Semantic chunking follows the document instead. Section headers and paragraph breaks define the boundaries, or embedding similarity between adjacent sentences marks the point where the topic shifts. The result is a chunk that represents one idea, which is exactly what makes its embedding useful rather than a blur of unrelated sentences.
Document type decides the approach. Technical documentation responds well to section based splitting because it is already structured. Meeting transcripts benefit from topic shift detection. Legal contracts need hierarchical chunking that preserves nested clauses. Code needs AST aware boundaries at function and class level rather than arbitrary line counts. Keep 50 to 200 tokens of overlap regardless of chunk size, so an idea sitting on a boundary is complete in at least one chunk.
Hybrid Retrieval Covers Both Failure Modes
Dense retrieval with embeddings understands meaning. A query about reducing customer churn will find a chunk about retention strategies even with no shared keywords. Its weakness is exact strings: a query for error code E-4217 can miss the chunk containing that code, because the embedding maps it into a general error handling region of the space.
Sparse retrieval with BM25 has the opposite profile. It nails exact matches, rare terminology and proper nouns, and it misses every semantic connection.
Running both and merging the two ranked lists with reciprocal rank fusion beats either alone, and it does so consistently across benchmarks. RRF scores by rank position rather than raw similarity, which sidesteps the calibration problem of combining scores from two different systems. Add a cross encoder reranker on top, 50 to 200 milliseconds to reorder the candidates by real relevance, and the context reaching the model gets noticeably cleaner.
One thing that surprises people: three highly relevant chunks usually produce a better answer than ten of mixed relevance. Filling the context window is not the goal.
What To Fix First
Work backwards from the failure. If the system answers the wrong topic entirely, retrieval is the suspect. If it finds the right document but garbles the specifics, look at chunk boundaries. If the text itself is wrong in ways no human would write, the parser did it, and everything downstream has been faithfully serving that error.
The order matters because the cost of fixing each stage is wildly different. Reindexing with new chunk boundaries takes a weekend. Repairing a corpus that was parsed badly means going back to the source documents and starting over.
The full breakdown of every stage, from ingestion through evaluation and the advanced patterns like agentic RAG and GraphRAG, is in our guide to RAG pipelines.
Top comments (0)