DEV Community

Cover image for Improving the scores of a RAG
Serhiy Kucherenko
Serhiy Kucherenko

Posted on

Improving the scores of a RAG

In the previous articles (part 1 and part 2) I compared my own RAG against five other systems. One of its numbers bothered me since then: recall@5 = 0.60. In plain terms, for 4 questions out of 10 the page with the answer was actually retrieved, but ranked below the top 5 that the LLM gets to see. So the answers tended to avoid specifics ("settles instantly, see section 4.2.3" instead of "5 seconds"), and one question failed so badly that the judge scored the answer 0.

The cause was measured earlier, and it is quite illustrative. The same page (p26 of the SCT Inst rulebook, the one that says "5 seconds") ranks differently depending on nothing but the phrasing of the question:

  • "target maximum execution time", the spec's own words: rank 1
  • "how fast does an SCT Inst payment settle?", a user's words: rank 9

A user asks casually while the rulebook is written formally, so their embeddings end up far apart. The chunk size makes it worse: a ~300-word chunk embeds as an average of everything in it, meaning a single bare fact competes with the rest of its own chunk.

There are two known ways to close this gap, so I measured both.

Hypothesis

A vocabulary mismatch can be attacked in two places:

  • At query time: rewrite the casual question into several spec-vocabulary variants and search with all of them. Cheap, no re-indexing needed, and very popular (RAG tutorials call it multi-query or RAG-Fusion).
  • At index time: give every chunk back the context it lost when it was cut out of its document, so that a bare "5 seconds" embeds as what it actually is. This is Contextual Retrieval as described by Anthropic; they report large reductions in failed retrievals. Costs a one-time re-index.

Note: there is a third popular trick, HyDE, where you embed a hypothetical
answer instead of the question. I rejected it without testing: in a compliance-heavy domain, a hallucinated draft steering the retrieval is exactly the failure mode this project exists to avoid.

Both experiments follow the same rule as before: measure on the golden set first, and the production path changes only if the number justifies it. For reference, reranking passed through the same gate earlier: it lifted recall from 0.60 to 0.70 but costs about a minute per question, so it never shipped.

Development

Fix 1: multi-query 

flowchart LR
    Q[casual question] --> RW["LLM rewriter<br/>(Haiku, ~$0.0001)"]
    RW --> V1[variant 1]
    RW --> V2[variant 2]
    RW --> V3[variant 3]
    Q --> R0[retrieve top-10]
    V1 --> R1[retrieve top-10]
    V2 --> R2[retrieve top-10]
    V3 --> R3[retrieve top-10]
    R0 --> F["reciprocal rank fusion<br/>(already in the codebase)"]
    R1 --> F
    R2 --> F
    R3 --> F
    F --> K[top-5 to the LLM]

An LLM rewrites the question into spec-vocabulary variants, each variant retrieves separately, and the ranked lists are merged with the same reciprocal-rank-fusion function that hybrid search already used. Nothing hallucinated can leak in this way: whatever the phrasing, retrieval only ever returns real corpus text.

The first surprise came before any quality result. Two identical eval runs returned 0.50 and 0.70. The reason was that the rewriter ran at the default temperature, so every run searched with different variants. If a retrieval mode has an LLM inside and the temperature is not pinned, the recall number is not reproducible. With temperature 0 the result became stable: 0.60, twice.

Which is... exactly the baseline. No improvement. The hits did move though: multi-query recovered the currency question (the one behind the judge-0 answer) and lost a different one, one for one. That trade was the most useful outcome of the experiment, because it showed the vocabulary gap is real and reachable, and also that rewriting alone cannot cash it in. A better phrasing still lands on the same diluted chunk embeddings.

I stopped there. With ten golden questions every hit is worth ±0.10, and tuning variant counts until the number goes up would be overfitting the eval, not improving retrieval.

Fix 2: contextual retrieval

For contrast, this is what indexing looked like until now. Each chunk is embedded exactly as stored:

flowchart LR
    PDF[rulebook PDF] --> CH[chunk ~300 words]
CH --> E["embed(chunk)"]
CH --> ST[store verbatim chunk]
E --> DB[(pgvector)]
ST --> DB

And with contextual retrieval, two LLM steps get inserted before the embedding. The same PDF feeds both paths: its full text produces a one-time summary, and that summary is the shared context for writing a short blurb per chunk:

flowchart LR
    PDF[rulebook PDF] --> CH[chunk ~300 words]
PDF -->|full text| S["LLM summary<br/>(once per doc)"]
S --> B["LLM blurb per chunk<br/>(one call per page)"]
CH --> B
B --> E["embed(blurb + chunk)"]
CH --> ST[store verbatim chunk]
E --> DB[(pgvector)]
ST --> DB

The blurb is one or two sentences situating each chunk: which rulebook it is from, which rule it belongs to, what its numbers are about. It is prepended only for the embedding. The stored text, the one that gets cited and shown as evidence, stays the verbatim spec passage. A bare "5 seconds" now embeds as "SCT Inst rulebook, target maximum execution time... 5 seconds", which is what a casual question is actually reaching for.

Two implementation choices kept it cheap and safe:

  • All chunks of a page get their blurbs in one call, against a one-time per-document summary. That is about $0.5 for the whole 484-chunk corpus, instead of the naive chunk-times-whole-document approach.
  • If a blurb call fails, that page just embeds bare chunks with a warning instead of aborting the run. In practice it never happened: 484 of 484 chunks got contextualized.

Results

Recall@5 per retrieval configuration, with the per-query cost of each:

  • plain index (baseline): 0.60
  • multi-query rewrites: 0.60, at ~1s + $0.0001 per query (and 0.50 to 0.70 until the temperature was pinned)
  • reranking, from an earlier experiment, for scale: 0.70, at about a minute per query
  • contextual index: 0.80, at zero per-query cost
  • contextual index plus multi-query: 0.80, the hits only reshuffle, no net gain

The contextual lift is strictly additive: every question the baseline got right, plus two it did not, including the one behind the only judge-0 answer. It also carries through end to end. The answer eval on the new index scores 96.3 mean with a 100% pass rate, up from 84.8 and 90%. Asked live, the formerly failing question now answers "SCT Inst payments are made in euro" and cites the rulebook pages.

One more detail: promoting the fix to production cost zero additional API dollars. The deploy tooling copies the chunks table, vectors included, from the local database to the cloud one, so production serves the exact index the numbers were measured on.

Note: the earlier six-system comparison scored this same system 84.8, fourth place out of six. On the new index it evals at 96.3, a hair under that comparison's winner (LlamaIndex, 96.5). This is not a re-ranking of the comparison: different index, different day, and the other five systems would deserve the same fix applied. But it does suggest the gap was never about the framework choice. It was about the index.

Conclusion

  • The cheap query-side trick did nothing, and the index-side fix did everything. Multi-query is what every tutorial reaches for first because it needs no re-indexing; on this corpus it moved recall not at all. Contextual retrieval, for about $0.5 one time, moved it from 0.60 to 0.80 with zero per-query cost.
  • The failed experiment still paid for itself. Its one gained question predicted what the real fix would deliver, and it surfaced that an unpinned LLM temperature makes an eval non-reproducible. A measured "no" is material, not waste.
  • Ten questions is a small ruler. Every hit is worth ±0.10, so I stopped tuning the moment the temptation appeared. The numbers above are honest rather than optimized.
  • Same discipline as always: nothing ships on vibes. Reranking measured well and stayed benched because of latency, multi-query measured flat and stayed benched, contextual retrieval measured +0.20 at zero query cost and shipped the same day.

Out of Scope

  • Sentence-window / small-to-big chunking, the other index-side fix from the same playbook; it stacks with this one but was not measured here
  • Re-running the full six-system comparison on contextual indexes (every system would need the same treatment to keep it fair)
  • BM25/hybrid contextualization: the blurbs currently improve only the vector side

Sources

Top comments (0)