2 Methods for Improving RAG Retrieval: A Real Client Case Study

Most RAG advice defaults to the trendy stuff — agentic RAG, multi-hop reasoning, exotic re-ranking pipelines. But a lot of the time, the actual fix for bad retrieval is much simpler than that. This is a real case study from a client project where two straightforward changes took recall from around 50–60% up to 95%+ — without touching the model, the chunking strategy, or reaching for anything trendy.

Recall, for context, measures how often your retrieval system actually finds the correct document(s) for a given query. It's arguably the single biggest lever in RAG quality: no amount of downstream prompt engineering fixes an LLM that was never given the right context to begin with.

The Setup

This was a fairly standard internal RAG chatbot for customer service staff — built to help them find information faster across the company's systems. The pipeline followed the usual shape: pull data from source systems, chunk it, generate embeddings, load everything into a vector index (Azure AI Search, in this case), and connect a chatbot on top that retrieves relevant chunks and generates an answer.

The domain was spa and wellness services. The two key document types were:

• Locations — spas and gyms, each with a description covering available treatments, massages, and equipment, plus city and region
• Experts — massage therapists, personal trainers, etc., each with a description of the services they offer

Standard indexing approach: join all the text fields (description, city, region) into one content field for full-text search, and generate an embedding of that same field for vector search.

Where It Broke

User queries looked like "Swedish massage in Helsinki" — and the correct answer should be: locations and experts in Helsinki that specifically offer Swedish massage. In testing, this setup only retrieved the right documents about 50–60% of the time.

Vector search turned out to be a non-starter for this use case. It's built for fuzzy, semantic matching — great when you want conceptually related results, but actively unhelpful when you need an exact match on a specific service and a specific city. Vector search would happily return other types of massage, or other cities that were semantically close to Helsinki (other Finnish capitals, other cities in the same region) — none of which is what the user actually asked for.

BM25 (full-text search) was better, but still not good enough, for a subtler reason: it ranks based on term frequency. Picture a location that offers every massage type except Swedish massage — its description might mention "massage" six times, and happen to also mention "Swedish meatballs" on their menu, racking up seven total keyword matches. Meanwhile, a location that offers only Swedish massage, in Helsinki, might get just two matches for "massage" and one for "Helsinki" — three total. Despite being the exact right answer, it ranks below the irrelevant location with more incidental keyword hits.

There was also a language-specific wrinkle: this project was largely in Finnish, where words conjugate heavily. "In Helsinki" isn't just "Helsinki" — it takes a different word form depending on context. If a document uses a different conjugation than the one in the user's query, plain keyword matching misses it entirely.

Fix #1: Extract Structured Fields at Index Time

The first change: add a services field to the search index — a clean, structured list of services, rather than leaving that information buried inside a free-form description paragraph.

That structured data didn't exist in the source system, so it had to be derived. The fix: run each location/expert description through an LLM at indexing time, prompting it to extract the specific services mentioned and return them as a clean list. That list becomes the new services field on the document.

At the same time, the vector embedding field got dropped entirely — it wasn't earning its keep for this exact-match use case, so there was no reason to keep paying to generate and store it.

Fix #2: Structure the Query, Not Just the Index

The second — and bigger — change happened on the query side. Instead of running the user's raw natural-language query directly against the search index, an LLM first parses the query into a structured filter.

"Swedish massage in Helsinki" becomes something like:

{
  "city": "Helsinki",
  "services": ["Swedish massage"]
}

That structured query then runs as an actual filter — city == "Helsinki" AND services contains "Swedish massage" — instead of a fuzzy text or vector match. This is exact, precise filtering instead of approximate relevance scoring, which is exactly what this use case needed.

The Result

After both changes, correct-document retrieval jumped to close to 100% in the next round of user testing — up from the original 50–60%. The team didn't have exact recall numbers at that stage of the project (feedback was being evaluated more qualitatively at the time), but the shift was obvious: nearly all the retrieval complaints users had previously raised simply disappeared, leaving only a small number of edge cases — mostly caused by underlying data quality issues, addressed separately.

The Tradeoffs

Nothing here was free:

• Indexing got more expensive. Every document now runs through an LLM call to extract services at index time. For a tool serving hundreds of internal users and saving them real time, this cost was easily justified — but it's a real, ongoing cost to account for.
• Query latency increased slightly. Structuring the query now requires an extra LLM call before retrieval even happens. In practice this was mitigated by using a smaller, faster model for this step — the inputs (a short user query) and outputs (a small structured filter) are both tiny, so the added latency was minor relative to the overall retrieval-and-generation flow.

The Bigger Takeaway

You don't always need agentic RAG, multi-hop retrieval, or whatever the latest research paper is proposing. Sometimes the fix is much more mundane: look clearly at what your users are actually asking for, and match your retrieval method to that. Here, the queries were fundamentally exact-match lookups (a specific service, in a specific city) dressed up as natural language — so the right fix was structured filtering, not fancier semantic search.

There's also a nice bit of symmetry worth noting: RAG stands for retrieval-augmented generation — using retrieval to make LLM outputs better. But it works in the other direction too. Here, LLMs were used to make the retrieval itself better — extracting structured metadata at index time, and structuring the query at search time. Both directions are worth having in your toolkit.

---

If your RAG system's problem looks like "the model gives bad answers," it's worth first checking whether the real problem is upstream — whether retrieval is actually finding the right documents in the first place.