A post arguing that RAG is simpler than people think made the rounds, and I nodded through the whole thing with the specific satisfaction of someone who learned it the expensive way.
If you built a retrieval-augmented system in the last couple of years, you probably did what I did. You compared vector databases. You read about chunking strategies. You tuned the embedding model. You added a reranker. You drew a diagram with six boxes in it and felt like a serious engineer.
Then you shipped it and the answers were mediocre, and almost none of the mediocrity came from the parts you spent your time on.
Where the failures actually came from
For me, in rough order of how much damage they caused:
The documents were bad. Outdated pages sat next to current ones with no way to tell them apart. The same policy existed in three versions across a wiki, a PDF, and a Slack thread. Retrieval worked perfectly and confidently handed the model a document from 2023. No amount of embedding quality fixes a corpus that contradicts itself.
Permissions were an afterthought. The moment a system can surface any document, it can surface the wrong document to the wrong person. Retrieving with the user's own access rights rather than a service account's is not a feature to add later. It is a property of the architecture, and retrofitting it is miserable.
The prompt assembly was sloppy. I had spent weeks on which chunks to retrieve and about ten minutes on how to present them to the model. Whether sources are labeled, whether the model is told what to do when the context does not contain the answer, whether the ordering implies priority — these mattered more than the retrieval nuance.
And I had no way to tell whether a change helped. That one is the real killer.
Evaluation is the whole game
If you take one thing from this: write down thirty real questions with known good answers before you optimize anything.
It does not need to be sophisticated. A spreadsheet is fine. What it gives you is the ability to make a change and know whether it helped, which is the difference between engineering and redecorating. Without it, every tuning decision is a vibe, and you will happily add a reranker that makes things slightly worse while feeling productive.
Thirty questions from actual users beats three hundred synthetic ones, because real questions are weirder. They are ambiguous, they use internal jargon, they assume context, and they ask about things that are genuinely not in your documents. That last category is where most systems embarrass themselves.
The boring baseline is stronger than it looks
The other thing I underrated: plain keyword search, plus metadata filtering, plus a modest amount of context, gets you remarkably far.
Not always. Semantic search genuinely helps when users and documents use different vocabulary, which is common. But keyword matching is precise, debuggable, and instant to explain when someone asks why a particular result showed up. Hybrid approaches win in practice partly because they let you reason about failures.
Metadata filtering may be the most underrated component in the entire stack. Restricting retrieval by date, document type, product area, or team eliminates a whole class of confidently wrong answers, and it costs almost nothing. I have seen a filter on "current version only" improve output more than a model upgrade did.
How I would start over
If I were building this again from zero, I would spend the first week on the corpus, not the code. Find out what documents exist, which are authoritative, what the access rules are, and how stale things get. That work is unglamorous and it determines the ceiling of everything downstream.
Then I would build the dumbest possible pipeline: search, take the top handful of results, put them in a prompt with clear labels and an explicit instruction to say when the answer is not present. Measure it against the question set. Only then start adding machinery, one piece at a time, keeping whatever the numbers justify.
My suspicion is that a lot of teams have complex retrieval systems where a simple one would perform the same, and the complexity persists because nobody can prove which parts are earning their keep.
The pattern is not unique to this technology. When a field is new, we mistake the interesting part for the important part. The interesting part was the vector math. The important part was knowing which document is true.
Top comments (0)