When a RAG answer comes out wrong, the first fix most people reach for is the prompt, or a bigger model. Often neither can help, because the model never saw the text it needed. Hamel Husain and Shreya Shankar put it in one line: "The retrieval component is a search problem."
I'd already built a retrieval step without thinking of it as RAG, and I wanted to know how well it actually worked. So I copied that lookup rule into a small controlled demo, measured what it retrieved, and then tested what the model did with those results.
The retrieval I already built
My Nagamese generator turns English posts into Nagamese for small businesses. In my testing, general AI models handle Nagamese poorly, so the translate path gives Gemini a glossary.
First it simplifies the English into short sentences. Then it builds every phrase of one to four words from them, and skips lookups for a short list of stop words, very short words and numbers. That only removes single word lookups like "the"; the model still reads the whole post.
Each phrase is looked up as an exact key in a dictionary I compiled from published word lists. No fuzzy matching, no embeddings. If an entry is stored as "dish", the word "dishes" finds nothing.
Whatever matches goes into the Gemini prompt under one instruction: use exactly as provided. Gemini writes the Nagamese with that glossary in hand.
That lookup is retrieval. It decides what the model sees, the same way a vector search does in a document chatbot. And I never measured it. There's no labelled set of inputs with the entries each one should get, so I couldn't tell you how often the right entry reaches the prompt.
Score the search on its own
Measuring retrieval doesn't need a model at all. You need labels: for each input, the entries a correct answer depends on. Then plain code can count what came back. Jason Liu makes the same point, that retrieval metrics "don't require LLM evaluation and provide quick feedback cycles."
For each input I count three things:
- Found: needed entries that came back. Found out of needed is recall.
- Extra: entries that came back but weren't needed. Found out of everything that came back is precision.
- Wrong sense: extras that match the words but not the meaning, like the fruit "date" in "save the date". They're a subset of extras, counted separately because they do the most damage.
A post's retrieval is complete when every needed entry came back and no wrong sense entry did. Other extras, like coffee arriving next to cold coffee, don't break completeness; they show up in precision instead.
To test this without publishing anyone's dictionary or inventing Nagamese, I built a small demo with a made up café glossary: 25 entries such as cold coffee → frosk mavi and cookie → dorpa. The target words are deliberately made up. The model has no reason to know these words beforehand, so a correct glossary word should only reach the answer when retrieval supplies it. That isolates retrieval failures from the model's existing knowledge of the language.
There are 24 English posts, each labelled with the entries it needs, and tagged by the kind of match it takes. One post carries two tags ("Our weekend offer: two coffees for the price of one" is a phrase and a plural), so the tags add up to 25:
| Tag | Posts | Example |
|---|---|---|
| Written as is | 6 | "Try our cold coffee this weekend." |
| Plural or tense | 5 | "Fresh cookies come out of the oven at four." |
| Phrase | 4 | "The chocolate cake is back on the menu." |
| Typo | 4 | "Free home delivry on weekends." |
| Synonym | 3 | "Fresh biscuits with every tea." |
| Wrong sense | 3 | "Save the date: our anniversary is on the 12th." |
The code, posts, labels and runs are on GitHub.
Five retrievers, scored in code
I ran five retrievers over the same posts. All of them are plain code:
- Exact match: my glossary rule, copied from the Nagamese code. Every one to four word phrase, looked up as an exact key.
-
Normalised: the same, with a small normaliser on both sides for plurals,
-ingand-ed. - Fuzzy: compares each entry with every phrase of the same length using the Dice similarity of their character trigrams (shared three letter chunks, counted twice, over the total), and matches at 0.75 or above.
- Normalised + fuzzy: everything either one finds.
- Oracle: exactly the labelled entries. Not a real retriever; it shows what the model does when retrieval is perfect.
My exact rule gets every post where the words are written as they are in the glossary, and 10 of 24 posts in all.
It gets no plural and no typo. "Cookies" doesn't find "cookie", and "chocolte cake" doesn't find "chocolate cake". In a real glossary for real posts, those are everywhere.
Normalising fixes the plurals and fuzzy matching fixes the typos. Together they reach 18 of 24.
They also pull in things they shouldn't. "We are rolling out new opening hours" now retrieves the bread roll, and "mint condition" retrieves the herb. A looser match finds more of the right entries and more of the wrong ones. Without labels you only see the first half of that.
These spelling rules don't recognise synonyms. "Biscuits" will never match "cookie" by spelling, and "food to go" will never match "takeaway". That's where semantic retrieval can help: embeddings can find related terms that spelling misses. Anthropic's write up on contextual retrieval makes the reverse point too: embeddings "can miss crucial exact matches", which is why keyword matching and embeddings are usually combined.
| Retriever | Posts complete | Needed entries found (recall) | Extra entries | Precision | Wrong sense |
|---|---|---|---|---|---|
| Exact match (my rule) | 10 of 24 | 22 of 36 | 7 | 22 of 29 | 2 |
| Normalised | 14 of 24 | 29 of 36 | 8 | 29 of 37 | 3 |
| Fuzzy | 18 of 24 | 31 of 36 | 7 | 31 of 38 | 2 |
| Normalised + fuzzy | 18 of 24 | 33 of 36 | 8 | 33 of 41 | 3 |
| Oracle | 24 of 24 | 36 of 36 | 0 | 36 of 36 | 0 |
None of this needed a model call. It runs in milliseconds, and it already tells me what to fix in my own glossary step before I look at a single translation.
Then the model
I took three retrievers into the generation test: my original exact rule as the baseline, normalised + fuzzy as the best real retriever, and the oracle as the perfect retrieval reference. For each, openai/gpt-oss-120b through Groq rewrote every post with the entries that retriever found, told to use each glossary word exactly as given. No temperature was set and reasoning was left at the model's default. Code checks the result: every needed made up word present, no wrong sense word. I ran it three times on 15 November 2026.
Every post falls into exactly one of three groups for each retriever: retrieval was complete, a needed entry was missed, or nothing was missed but a wrong sense entry came back. No post had both errors.
| Retriever | Posts right, runs 1 / 2 / 3 | Complete retrieval | A needed entry missed | Only a wrong sense entry |
|---|---|---|---|---|
| Exact match (my rule) | 10, 10, 10 | 10 of 10, every run | 0 of 12, every run | 0 of 2, every run |
| Normalised + fuzzy | 19, 19, 16 | 18, 18, 16 of 18 | 0 of 3, every run | 1, 1, 0 of 3 |
| Oracle | 19, 20, 21 | 19, 20, 21 of 24 | none | none |
When retrieval was complete, the answer was almost always right: every time with exact match, and 16 to 18 times out of 18 with the looser retriever.
When retrieval missed an entry, the post was wrong in every run. Part of that is by design: the model can't guess "dorpa". With a real language it might guess a word it already knows, which doesn't fix the miss. It only hides it.
The more interesting failure was a wrong match. When the retriever returned an entry that matched the spelling but not the meaning, the model used it. It wrote "Save the zimbo" and "our new seats are in lerpa condition", following the instruction to use the glossary even when the entry didn't belong. Exact match returned 2 wrong sense entries and normalised + fuzzy returned 3, so over three runs there were 15 chances, and the model used the wrong entry 13 times. The two times it didn't, it ignored the bread roll in "rolling out", which is the right call and the opposite of what the prompt told it to do.
Perfect retrieval still isn't perfect answers. Handed exactly the right entries, the model got 19 to 21 of 24. It left "biscuits" and "latte" as they were instead of using the cookie and coffee entries, sometimes kept "milkshak" as written, and twice spelled "dovra velt" as "davra velt". Those are generation problems. The oracle row is what separates them from search problems.
The oracle gives me a ceiling. Even when retrieval gets every entry right, the model still makes mistakes, so I now have two separate problems to work on: what reaches the prompt, and what the model does with it. Without the oracle, those failures would look identical in the final post.
So the search decided most of the outcome. Search galat, toh answer galat.
What I'd change in my glossary step
The demo is deliberately modelled on my existing lookup rule. It doesn't prove these changes will improve real Nagamese translations, but it gives me concrete hypotheses to test:
- Normalise both sides. Plurals and tenses were the cheapest misses to fix in the demo, and they're common in real posts.
- Add fuzzy matching with a threshold I can see. Typos in business posts are normal. The threshold is a tradeoff, and the labels show where it starts pulling in wrong entries.
- Give ambiguous entries a sense. Words like date, roll and mint need a note the model can see, or a rule that skips them unless the context fits. The model will use whatever it's given.
- Build the labelled set from real inputs. Start with 20 to 30 real posts and have someone who speaks Nagamese verify which entries each one needs. Keep a separate set aside for testing, so tuning the lookup doesn't also decide its reported score.
What this doesn't prove
This is one small synthetic set: 24 posts and 25 entries, labelled by me. The made up words make retrieval decisive by design. With a real language, a model may already know some words, which hides retrieval misses instead of fixing them. The normaliser and the fuzzy threshold were tuned by hand on the same 24 posts they're scored on, so the combined retriever's numbers are a proof of concept, not evidence of how it does on new inputs; a held out set would test that. And it's one model, three runs.
What it does show is the method, and it's cheap: label what each input needs, score the search in code, then look at the answers.
What I'd tell someone building RAG
- Write down what should come back. For each test question, the passages or entries a correct answer depends on.
- Score retrieval before reading answers. Recall, precision and wrong matches, counted in code.
- Keep keyword matching. Microsoft's hybrid search docs note that product codes, jargon, dates and names "perform better with keyword search". Don't replace it with embeddings; combine them when your labels show each solving a different problem.
- Run an oracle. Hand the model the right passages and see what it still gets wrong. That's the part a better prompt or model can fix; the rest is search.
RAG evaluation doesn't need to start with an LLM judge or another round of prompt engineering. Start with the search. Write down what should be retrieved, measure what actually comes back, and see what changes when you give the model exactly what it needs.
If the answer is still wrong with perfect retrieval, you have a generation problem. Until then, a bigger model might just be a more expensive way to hide a search problem.
Top comments (0)