Retrieval augmented generation looks trivial in a tutorial chunk your documents, embed them, do a similarity search, stuff the results in the prompt. It runs on the first try. Then you point it at a real corpus, ask a real question, and it confidently returns
garbage.
I spent a chunk of the last two weeks building RAG from scratch no framework and, more importantly, measuring it at every step. Not "does it feel better," but a golden set of queries scored on whether the right answer actually came back. Here's what breaks,
why, and what fixes each thing with the numbers.
Step 1: Break it on purpose
Before improving anything, I built the crudest possible RAG split text every 300 characters, embed, cosine search and went hunting for failures on a technical doc. I found three distinct ways it breaks, and naming them precisely matters, because each has a
different fix.
Failure 1 chunk boundary corruption. The top ranked result for a query was:
"dles everything required to run a node, so there are no external runtime depende"
It starts mid word ("bundles") and ends mid word ("dependencies"). Fixed size chunking cuts through words and sentences with no regard for meaning. Even when retrieval ranked the right chunk first, the chunk itself was a broken fragment useless as context for the
model.
Failure 2 answer fragmentation. I asked "what happens to accumulators after a crash?" The complete answer was a chain of reasoning a write happens before the record proceeds, so on restart the value is restored, so processing resumes cleanly. But that chain was scattered across three different chunks, none of which contained the whole thing. Retrieval surfaced pieces no single chunk had the answer.
Failure 3 vocabulary sensitivity. A query phrased as the user would naturally say it ("how to set up?") scored far worse than one using the document's own words ("how do I install the binary?") the scores dropped and clustered together, meaning the system couldn't confidently rank anything. The same information need, phrased two ways, gave wildly different results.
The lesson from step one RAG has multiple, distinct failure modes, and "just use embeddings" addresses none of them.
Step 2: Fix the chunking and measure the limit
The obvious fix for boundary corruption is to chunk on sentence boundaries instead of character counts, and to overlap consecutive chunks so an answer spanning a boundary survives in at least one chunk.
Boundary corruption fixed. Every chunk now starts and ends at a real sentence.
But here's what measuring revealed that a tutorial wouldn't chunking alone can't fully fix fragmentation. I tested three chunk sizes on the crash recovery question:
- Small chunks (300 chars) the answer's conclusion ranked #1, but its supporting logic landed at rank #3 still split.
- Large chunks (800 chars) kept the answer together, but the bigger chunk diluted the embedding so it ranked worse overall.
That's a genuine, unavoidable tension small chunks give precise embeddings but fragment long answers large chunks keep answers whole but blur the embedding. There's no single right chunk size it depends on how long your answers tend to be. Chunk coherence turned out to matter more than any ranking trick, a theme that came back hard later.
Step 3: Hybrid search and learning to distrust an easy win
Embeddings capture meaning but blur exact terms (function names, error codes). Keyword search (BM25) is the opposite. Combining them running both and fusing the rankings with
Reciprocal Rank Fusion should cover each other's blind spots.
But first, a lesson in evaluation itself. My initial metric was recall@3 "is the answer in the top 3 results?" Everything scored 10/10. Great, except a metric everything passes tells you nothing it couldn't distinguish good retrieval from great. I switched to recall@1 (is it the top result?) and average rank, and the real picture appeared 6/10, with several answers present but poorly ranked.
Then I measured hybrid search honestly, and the result was not the tidy win I expected:
- On the prose doc hybrid tied plain vector search. It fixed the fragmentation case (pulled the crash answer from rank 3 to rank 1) but hurt the semantic only queries, where BM25's keyword noise polluted the fusion.
The honest finding hybrid search isn't a free upgrade. It helps on exact term queries and can hurt on purely semantic ones. On a small prose corpus, plain embeddings are hard to beat.
Step 4: The finding that reframed everything chunk coherence
To test whether hybrid search pays off at scale, I indexed a real 24-file codebase (Flask, ~470 chunks) and ran the same comparison. At first, every method failed recall barely above zero. When every approach fails equally, the problem isn't the ranker it's upstream.
The cause my prose-oriented chunker was splicing unrelated functions into the same chunk. The chunk containing def send_file was full of leftover code from a completely
different function. The identifier was present, but the chunk's meaning was a muddle, so its embedding matched nothing cleanly.
The fix was structure aware chunking parse the code's AST and make one chunk per function or class coherent semantic units, with accurate line numbers for citations. Rerunning the comparison on properly chunked code, hybrid search finally won cleanly it
was the only method to rank exact identifier queries first, because now the identifier actually defined its chunk.
The deepest lesson of the whole exercise:
Retrieval quality is dominated by chunk coherence, not ranking cleverness. Incoherent
chunks capped recall near zero no matter which ranker I used. Hybrid search only helps once
chunks are coherent and queries contain exact terms. Chunk your data well before you
reach for a fancier retriever.
Step 5: The part chunking can never fix synthesis
Even with perfect chunking, some answers genuinely span multiple chunks (like that crash recovery chain). No single chunk will ever hold them. The fix isn't retrieval at all it's retrieving the top-k and letting the model synthesize across them.
This is why real RAG returns multiple chunks instead of one. I wired the retrieval into an agent as a search_code tool that returns snippets with their file and line numbers, and
instructed the model to cite them. Now the agent retrieves several chunks, assembles the complete answer, and cites app.py:969-993 for each claim.
And then the payoff I verified it. I asked the agent how a request gets dispatched, it cited specific lines, and I opened those lines to confirm the code was actually there. It was. It even noticed that the checkout I'd indexed differed from the released version. The answer wasn't just fluent it was verifiable every claim traceable to source.
The whole arc, in one table
| Failure | Fix | Measured result |
|---|---|---|
| Chunk boundary corruption | Sentence boundary chunking | Fixed chunks are clean |
| Answer fragmentation | Overlap + top-k synthesis | Partly by chunking, finished by the agent |
| Vocabulary sensitivity | Hybrid (BM25 + vector) | Helps on exact terms, not semantic only |
| (discovered) Chunk incoherence on code | AST structure aware chunking | Recall near zero → hybrid wins |
What I'd tell someone starting with RAG
- Measure, with a golden set. "Feels better" is not a result. And distrust any metric everything passes recall@1 tells you far more than recall@3.
- Chunk coherence beats ranking cleverness. Get your chunks right semantic boundaries for prose, AST for code before you reach for hybrid search or a reranker.
- Hybrid search and reranking are situational, not default. They earn their place on large, identifier heavy corpora, and can hurt on small semantic ones. Measure per corpus.
- Some answers span chunks that's why you retrieve top-k and let the model synthesize. Retrieval isn't about finding the one perfect chunk.
-
Cite your sources. An answer you can trace back to
file:lineis one you can verify. That's the difference between a demo and something you'd trust.
The naive version runs on the first try. Everything that makes it actually work comes from knowing how it fails and measuring your way out.
This is from a from scratch agent I built to understand agent internals including a retrieval pipeline evaluated with a golden set harness across multiple corpora. Code and per day notes on GitHub:
github.com/Yashwanth-Brahma/miniagent
Top comments (0)