I wanted to understand RAG properly, and reading about a technology is not the same as building one. Building one means choosing a document to build it on, and that choice turned out to matter more than I expected.
I took a document from my own field: a roughly 170-page design guide for engineering plastics and injection moulding, the kind of reference I spent twenty years working alongside. That did two jobs at once. The result is useful to the people I come from, rather than a demonstration over a corpus nobody needed searched. More importantly, I could judge whether an answer was actually correct, which is exactly what you cannot do on material you do not know.
That second point turned the project into an exercise in measurement rather than a demo, and it is why the things the measurements turned up are the most useful part of the story.
First, what RAG is
One definition, because everything below depends on it. RAG, retrieval-augmented generation, means the language model does not answer from memory. Each question first triggers a search across your own documents, and the passages that come back are handed to the language model together with the question, with an instruction to answer from those and nothing else.
Two things follow. Every answer can point at the part of the guide it came from, and anything the guide does not cover produces a refusal instead of an invention. The second consequence matters more: once the language model may only use what the search hands it, the search decides the quality of the answer. A capable language model given the wrong three paragraphs will not notice; it will simply write a more fluent wrong answer than a weaker one would. That is why most of what follows concerns the search rather than the language model.
How the search works, and what the system is made of
Searching by meaning rather than by wording requires both sides of the comparison to exist in the same form. An embedding model provides that form: it converts a piece of text into a list of numbers arranged so that passages about the same topic land close to each other. The question goes through the same conversion, and the search then reduces to finding the stored pieces nearest to it.
Because the guide has to be converted before anyone can ask anything, the system falls into two phases: once per document the guide is cut into pieces, each is embedded and stored, and then on every question the question is embedded and matched against that store. The same embedding model must serve both phases, since vectors from different embedding models cannot be compared.
Those pieces are called chunks: 223 from the body text, plus 185 figure entries and 17 table entries, held in three collections in Qdrant, a database built for nearest-neighbour lookup. Keeping figures and tables apart mattered later, when the chat model had to be told which kind of content it was looking at. That chat model, Qwen3.5-9B, is the language model that writes the answers, and it runs alongside the embedding model on llama.cpp on a single GPU. Everything stays on that machine, which is not a technical footnote: a licensed design guide is not something you upload to a third party to make it searchable.
One component sits in front of all this, because not every message is a design question and not every design question is covered by the guide. A classifier separates greetings and help requests from content questions, then a similarity threshold decides whether the nearest chunks are close enough to answer from.
What an engineer gets out of it
None of that machinery is visible to the person using it. A designer types a concrete question, such as how thick a wall can be before the surface sinks, and the assistant answers from the guide while naming the section, figure or table it used. That citation is the point rather than a nicety, because an engineer about to commit a wall thickness to a steel mould has to check where the number came from. The refusal works the same way from the other direction: a confident answer to something the guide never addresses would be worse than none.
So the value is not that the assistant saves you from reading 170 pages. It is that it finds the right two of those pages in seconds and shows you which two.
Round 1: where the guide gets cut
The first link in that chain is the cutting, and it is easy to underestimate. A chunk spanning two unrelated subsections produces a vector that represents neither properly, and no amount of search quality afterwards recovers what the cut destroyed.
Hierarchy-aware chunking follows the guide's own chapters and subheadings, so a boundary lands where the guide itself changes subject. The naive alternative cuts every N characters regardless. Which of the two wins was never really in doubt, so I did not build the naive pipeline to find that out. I built it to see the size of the difference, because "structure helps" is an intuition, and a number is something you can act on.
The naive run produced 137 chunks against 223, and inspection showed why that mattered: 86.1% bridged two or more unrelated subsections, 77.4% began mid-sentence, and 31.4% exceeded the embedding model's token limit. These are not cosmetic flaws but chunks that do not correspond to a single coherent idea, which is exactly what the vector is meant to capture.
The retrieval scores followed. One definition first, since these numbers recur: retrieval works like asking a librarian for the right page, so Recall@1 measures how often the first suggestion is correct, and Recall@5 counts a hit anywhere in the top five.
Recall@1: 0.843 for hierarchy-aware chunking against 0.708 for naive chunking, a gap of 13.5 percentage points.
Round 2: which chat model follows the rules
Putting the right chunks in front of the chat model does not guarantee that it stays inside them. Twice it did not: once inventing a formula for a table that exists in the guide only as a drawn image, and once claiming a figure was absent when it was plainly there. Both were settled with prompt rules that tell the model what to do when the context is incomplete instead of letting it improvise.
Those rules are only worth writing if the chat model follows them consistently, which turned the next question into one about the model itself. My assumption was that the larger Qwen3-14B would be the safer default and the newer, smaller Qwen3.5-9B the speed option.
Repeated runs said otherwise. At one prompt configuration the 14B model fell back into the already-fixed hallucination on 5 out of 5 attempts, while the 9B held steady across both of its checked runs. The deciding axis was therefore not raw capability but reliability under repetition, and the smaller model shipped.
Round 3: the questions I had not been asking
Every improvement so far was measured with questions written in the guide's own vocabulary, which is a flattering test, because that is the one condition the system was tuned for. Real users do not write that way.
Rephrased as a non-expert would type them, using "why do parts get weird dents" instead of the formal defect name, Recall@5 fell from 1.000 to 0.400. The content was unambiguously in the guide, so the failure lay entirely in the matching, and widening the search did not recover it. The problem is therefore not how many candidates are considered but how the question is represented in the first place.
That leaves a documented limitation rather than a solved one. The obvious next steps stop relying on meaning-matching alone: rewriting the question into formal terms before searching, adding keyword matching to catch a specialist's exact wording, or re-ranking the closest candidates in a second pass.
How all of this was measured
None of those findings would have surfaced without something to measure against, so the evaluation ran on three levels.
Retrieval was scored across 425 synthetic questions, giving Recall@1 of 0.899, Recall@5 of 0.995 and an MRR of 0.943, where MRR rewards the correct answer for landing near the top rather than merely appearing somewhere.
Answer quality was scored by an external LLM-as-Judge, deliberately not the chat model that writes the answers, since a language model grading its own output is not evidence.
Faithfulness at 4.44 and conciseness at 4.08 sit lowest, consistent with the hallucination work above, while relevance at 4.96 sits near the ceiling, which is what good retrieval should produce.
End to end, 10 of 10 scripted user journeys passed at roughly 18 seconds per answer, a figure that tracks answer length rather than slowness. Three simulated personas of rising expertise were then scored for satisfaction.
The most expert persona was the most satisfied, reassuring precisely because that was the expected direction. Several of these figures rest on small samples, which is worth naming: 5 runs against 2 in the chat-model comparison, three personas here. They are directional evidence, enough to act on but not proof at any rigorous confidence level.
| Metric | Result |
|---|---|
| Recall@1, hierarchy-aware against naive chunking (body-text subset) | 0.843 against 0.708 |
| Recall@1 / Recall@5 / MRR (all 425 questions) | 0.899 / 0.995 / 0.943 |
| Recall@5, formal against colloquial phrasing | 1.000 against 0.400 |
| LLM-as-Judge, 1 to 5: faithfulness / relevance / completeness / conciseness | 4.44 / 4.96 / 4.64 / 4.08 |
| User journeys passed, average response time | 10 of 10, roughly 18 s |
| Persona satisfaction, 1 to 5 (novice / intermediate / expert) | 3.50 / 3.83 / 4.67 |
| Runs that fell back into the fixed hallucination, one prompt setup | Qwen3-14B 5 of 5, Qwen3.5-9B 0 of 2 |
Key takeaways
Where a document gets cut decides how well it can ever be searched. The 13.5-point gap came from nothing more than respecting the guide's own outline.
A bigger language model is not automatically a safer one. Reliability under repetition proved worth more than raw capability.
A system tested only in its own vocabulary looks better than it is. The 60-point collapse on plain-language phrasing is invisible to any demo, because demos are written by whoever built the thing.
Conclusion
Building on ground I already knew is what made the measurement possible. On an unfamiliar document I could have collected exactly the same numbers and still not known which answers were right, and every finding above came from being able to tell the difference.
The colloquial-phrasing gap is still open, and it is the first thread I would pull next.
Side note: a multilingual variant exists, but it is not the subject here. It depends on a cloud translation service, which breaks the fully local property everything above rests on, and it has not been through this evaluation. Every number here refers to the local, English-language system.
About the Author
Szilárd Galambos spent 20 years as a mechanical engineering group lead at Robert Bosch, and is currently on a deliberate career break to build expertise in data science and AI. With a background in engineering mathematics and hands-on experience in n8n workflow automation, Linux server administration, and AI integration, he bridges the gap between traditional engineering thinking and modern data-driven approaches.
Connect on LinkedIn.
Top comments (2)
The 60 point drop on colloquial phrasing is the most honest benchmark number I have seen in a RAG writeup. Dense embeddings almost always fall over on shop floor terms like 'weird dents' versus 'sink marks' because general vector models never saw those specific synonyms paired during training. A cheap lexical fallback or a fast query expansion step usually recovers most of that without needing a heavy re-ranker pipeline.
Thanks. Transparency matters more to me than a tidy looking result, so it is good to see someone value that.
I should be upfront that I am a mechanical engineer who built this to understand RAG, not a retrieval specialist, so I can only really speak to the things I actually measured.
Query expansion I never tested, and of the three suggestions it is the one I find most convincing, for exactly the reason you give.
The lexical fallback I never tested either, so I cannot argue with it from data. What I can offer is what I saw when I dug into why the colloquial questions fail. Retrieval was not coming back empty. It was coming back with a topically adjacent section: a question about dents opposite a moulded-in rib returned the surface finishing chapter instead of wall thickness and ribs. The similarity scores still cleared the threshold I use, so nothing in the system flagged those answers as weak. The word "sink" appears in twelve different subsections of the guide, so there is plenty of nearby material to drift into. What I do not know is how a keyword fallback would score a question that shares almost no words with the section it should find.
The re-ranker is the one I did test. A cross-encoder over the top 10, and on my 425 formal questions Recall@1 went from 0.899 to 0.906, so 0.7 percentage points. Sixteen results improved and fourteen got worse. I left it out, because that is not worth the added latency on every question. Separately, on the colloquial set, widening the candidate pool from 5 to 8 changed nothing at all: same 0.400, the same six questions missing. I never ran the re-ranker on the colloquial set itself, so that stays an open question rather than something I settled.
Thanks again for all three suggestions. This is the most useful kind of comment to get on a writeup like this one.