Query rewriting is the first stage of most RAG pipelines and the one that gets tuned by vibes. So I built a real BM25 index over a real corpus, applied real rewrites, and scored them against hand-labelled relevance.
👉 Live, runs in your browser: https://dev48.in/ai/days/day82-query-rewriting.html
| rewrite | recall@3 | precision@3 | MRR | mean query length |
|---|---|---|---|---|
| verbatim (keeps stopwords) | 0.688 | 0.375 | 0.938 | 3.4 |
| raw (drops stopwords) | 0.688 | 0.375 | 0.938 | 2.9 |
| expand (synonyms) | 0.938 | 0.542 | 0.938 | 5.8 |
| expandNoisy (wrong synonyms) | 0.688 | 0.375 | 0.813 | 5.8 |
| hyde (pseudo-answer) | 0.563 | 0.292 | 0.323 | 16.9 |
| decompose | 0.688 | 0.375 | 0.938 | 2.9 |
The claim I had to withdraw
I wrote this assertion and it failed:
t("INVERSION: expand raises recall but LOWERS precision", () => {
return exp.recall >= base.recall && exp.precision <= base.precision;
});
Expansion raised both — recall 0.688 → 0.938 and precision 0.375 → 0.542.
That is not a flaw in the measurement. It is what a genuine vocabulary mismatch looks like: the documents say automobile and vehicle where the queries say car, so the added terms are not noise, they are the missing words.
The precision/recall trade-off everyone quotes is a property of synonym quality, not of expansion. The control proves it — the same mechanism, same code path, identical query length (5.8 tokens either way), but with plausible-and-wrong synonyms:
precision 0.375 → 0.375, MRR 0.938 → 0.813
Interesting detail: precision@3 does not move. The wrong synonyms are rare terms, so they pull in few documents at a cut-off of three. The damage shows up in where the first relevant hit lands — which is the metric a RAG pipeline actually lives on, because the generator only ever sees a truncated list.
Why recall is a trap here
Look at the definition. expand only adds terms, never removes them:
expand: (q) => {
const base = tokens(q), out = base.slice();
for (const w of base) for (const s of (SYNONYMS[w] || [])) out.push(s);
return out;
}
A superset of the query can only retrieve a superset of the documents. Recall cannot fall. So "our rewrite improved recall" is close to a tautology for any additive rewrite, and tuning on it will approve every expansion you ever write, including the bad ones.
HyDE lost to doing nothing
MRR 0.938 → 0.323. Per-query, the pseudo-answer version is worse on five of eight.
The mechanism is mundane. A HyDE pseudo-answer is query terms plus answer-shaped filler — this, document, explains, typically, generally, process, step, first, then, finally. Those are real terms in the corpus, and two of my sixteen documents are majority filler:
"The process you should generally follow to file a complaint"
"This document explains what you should typically do first"
A pseudo-answer retrieves those because it looks like them.
The control is the important half: strip the filler and keep everything else, and HyDE's MRR is identical to the raw query, to machine precision. It is the filler, not the method. A corpus of prose with no boilerplate documents would behave differently — which is the real lesson, since you cannot know that without measuring your own corpus.
A rewrite reweights, it does not just add
BM25 scores a term by how rare it is:
| term | documents containing it | IDF |
|---|---|---|
| alternator | 1 | 2.0794 |
| automobile | 2 | 1.7047 |
| car | 0 | 3.4340 |
| you | 6 | 1.0986 |
Adding one common word contributes almost nothing. Adding many common words — which is what a pseudo-answer is made of — pulls generic documents to the top.
Verification
The page scores from an inverted index, accumulating term by term. The verifier builds a dense term-document matrix and computes each document's score from the BM25 formula directly — the transpose of the page's loop, written from the formula rather than from its code. The relevance labels are retyped, so a label typo would surface as a disagreement instead of being copied along.
263 asserts, 0 failures. 22 in-page checks.
Top comments (0)