A "lexical query" is not the same thing as a query that needs lexical matching.
My retrieval benchmark sorts every query into two groups: ones that contain an identifier — a file path, a version number, an error code — and ones that don't. Queries with identifiers are supposed to be the easy case for keyword search. They were the case where it lost by the widest margin.
The reason: for 54% of those queries, the identifier never appears in any of the documents that answer them. There is nothing for exact matching to match against.
The usual advice says keyword search handles keyword queries, dense retrieval handles the rest, and hybrid gives you both. That advice is why most people run hybrid search. None of it held up here, and the part I was confident about is the part that broke worst.
The setup
BEIR / CQADupStack, the unix and android subforums: 70,380 documents and 1,771 judged queries. The task is finding duplicate questions — given a Stack Exchange question, find the earlier question that asks the same thing.
Before any retrieval runs, a classifier labels each query lexical or semantic. It only looks for lexical signals: something shaped like an identifier, or text in backticks. Semantic is just everything left over. There's no equally reliable way to look at a query and tell that it needs meaning-based matching, the way you can tell that a string looks like a hex literal. 214 queries end up labeled lexical, 1,557 semantic.
The BM25 side uses Porter stemming and identifier-aware tokenization. Anserini's Lucene analyzer, which produced the published BEIR numbers I compare against, stems by default, and the dense side gets that kind of robustness for free from how it tokenizes. The unstemmed numbers are in the repo as a control.
The result
Hybrid belongs in that table too, since the advice this post argues with is partly about hybrid:
Recall@3, macro-averaged
overall lexical semantic
BM25 (identifier-aware
+ Porter) 0.2255 0.3070 0.2143
Hybrid RRF (equal weight) 0.2964 0.3952 0.2828
Hybrid RRF + Porter 0.3173 0.4076 0.3049
Dense (all-MiniLM-L6-v2) 0.3834 0.4842 0.3695
Equal-weight RRF lands between its two inputs and below dense on its own. Dense beats the stemmed fusion by +0.0660 R@3 overall — 257 queries better against 107 worse, p = 2.3e-15 — and it wins both layers (+0.0766 lexical, +0.0646 semantic). Blending a 0.23 system with a 0.38 system on equal terms costs more than it gains.
The direction is stable when the two subforums are run as separate datasets: all 108 signs hold. Two confidence intervals don't clear zero, both for dense against stemmed fusion on android's lexical layer, where n drops to 96 (R@3 +0.0568, p = 0.093; MRR +0.0478, p = 0.281). That's a power limit rather than a reversal — the same cell was already borderline before stemming — but it's the one place the merged result isn't independently confirmed.
Weighting fixes most of the fusion gap. Giving BM25 only 10-20% of the score edges past dense (R@3 0.3894 vs 0.3834), and the lexical layer gains the most (0.5003 vs 0.4842). But that alpha was chosen on the same queries the number is reported on, so it's an optimistic ceiling rather than a validated gain. The defensible reading: tuned hybrid is roughly a wash against dense on this corpus, and equal-weight RRF is a bad default.
Dense wins overall, which is no surprise — that roughly matches the published BEIR ordering, and the dense nDCG@10 of 0.4071 is close enough to the reference number for this model that I trust the harness.
The surprise is where it wins. Dense beats BM25 by +0.18 on the lexical layer and +0.16 on the semantic one. The gap is bigger on exactly the queries full of file paths and version strings. A paired bootstrap over 10,000 resamples keeps the lexical interval clear of zero, and a sign test on the overall comparison gives 429 wins against 74 losses, p = 8.1e-62. It also holds when I run the two subforums as separate datasets instead of one merged pool (unix +0.2202, p = 3.4e-07; android +0.1940, p = 2.2e-05), so it isn't an artifact of merging them.
Why
That 54% explains the whole thing. BM25 can only match xorg.conf against documents that literally contain xorg.conf, and in a corpus of duplicate questions, most documents that do are about somebody else's unrelated problem. The question that actually duplicates yours describes the same situation but never types the path — or writes /etc/X11/xorg.conf, or
a different version number, or leaves it out. Worse, a rare token gets a very high IDF, so on the rare occasion it does match, it dominates the score and pushes up a document whose only connection to the query is that one string. The bi-encoder ignores the identifier entirely and matches on the problem description around it, which the duplicate really does
share.
So the classifier does exactly what it was written to do, and still measures the wrong thing. It tells you whether a query contains something identifier-shaped. I had been reading it as "this query needs exact matching." Those turn out to be different questions,
and on this task the answers differ more than half the time.
How far this goes: finding duplicate questions is close to the worst case for the assumption that identifiers are shared, because the query and the target are two different people describing one problem in their own words. In log search or code search the identifier is shared by construction, and I'd expect the usual advice to hold there. So the finding isn't "dense beats BM25 on lexical queries." It's that a query containing keywords and a query helped by keyword matching are two different things — and if you're routing between retrievers based on what a query looks like, you're assuming they're the same thing.
The useful version: how often a query's identifiers show up in its own correct answers is something you can measure on your corpus before building anything, and it tells you whether a keyword arm is worth having.
The check that made this believable
A benchmark that calls every difference real is worthless. So I pointed the same statistics at a comparison where I expected to find nothing.
That comparison was my identifier-aware tokenizer. It emits everything the plain one does, plus the full identifier — strictly more information, so it should only ever help, but in fact it didn't. On the lexical layer it landed 0.0125 below the plain tokenizer, with a confidence interval spanning zero and only 10 of 214 queries changing at all. p = 0.75. No
effect, and the same non-result shows up in both subforums separately.
That's what makes the p = 8.1e-62 above worth anything. The tests can tell a real difference from an imagined one.
Three limits on all of this:
- The bootstrap resamples queries. It covers variation in which questions I happened to test, not in the model, the preprocessing, or the relevance judgments.
- One embedding model, one corpus, one task. The 54% is a fact about the dataset that no choice of model would change, but everything built on top of it was measured on a single setup.
- These comparisons came after the fact. They show the ordering here isn't luck. They don't show it transfers anywhere else.
The repo has the rest of it: why equal-weight RRF ends up worse than dense alone, a tokenizer change that quietly cost recall, a leakage audit of CQADupStack, and the per-subforum replication. Code, full per-layer metrics, and the significance script
hybrid-rag-eval
This project compares BM25, dense, and hybrid retrieval on CQADupStack, with every metric split by query type — lexical (identifier-bearing) vs semantic. BEIR already shows dense beats BM25 overall; splitting by layer shows it wins by an even wider margin on identifier-bearing queries.
Setup
./scripts/download_data.sh # ~65 MB into data/cqadupstack/
python -m pytest tests/ -q
python run_eval.py # BM25 rows only, no dependencies
The BM25 rows, fusion and the metrics are standard library only. Dense
retrieval and reranking import sentence-transformers optionally; when it is
unavailable those three rows are reported as skipped and the BM25 rows still
complete. To run the full table:
python -m venv .venv
.venv/bin/pip install --index-url https://download.pytorch.org/whl/cpu torch
.venv/bin/pip install sentence-transformers pytest
.venv/bin/python run_eval.py
To reproduce the significance tables below (paired bootstrap + sign test on the paired comparisons, permutation test on the layer gap, all metrics, all layers) run with the venv interpreter so…
.
Top comments (2)
the 54 percent is the finding and the tokenizer null result is what makes me believe it. a change that is strictly more information landing 0.0125 below baseline, 10 of 214 queries moved, p 0.75, is a clean power check and almost nobody pre-registers one.
the thing i'd want stated louder is the corpus class. CQADupStack is duplicate question retrieval, so the target is by construction a paraphrase of the query written by a different person. that's close to the best case that exists for dense and the worst for lexical, and it's also exactly why your 54 percent is so high. someone restating your problem in their own words has no reason to repeat your file path.
flip the corpus and the ordering flips. where the identifier is in the answer because the answer is the artifact, error code lookups, api docs, log search, part numbers, the lexical arm earns its keep and we see it hold. so i read your result as a strong argument against routing by query surface, which is your actual claim, rather than against lexical retrieval generally.
your diagnostic is the reusable part and it deserves a name. measuring identifier presence in your own gold answers before you pay for a keyword arm is a twenty minute check nobody runs.
missing row is the hybrid number itself. dense beats BM25 on both strata, granted, but the decision a practitioner faces is fusion versus dense alone, and the post argues against hybrid's rationale without reporting it.
i co-founded backboard, we run hybrid BM25 plus vector in production, so treat the corpus class claim as interested.
Thanks! Your framing is better so I'm taking it.
The missing hybrid data: Equal-weight RRF is 0.2964 R@3, 0.3173 with the stemmed keyword arm, against dense at 0.3834. Dense beats the stemmed fusion by +0.0660 — 257 queries better, 107 worse, p = 2.3e-15 — and wins both layers. Run as two separate datasets the direction holds everywhere, though on android's lexical layer (n = 96) the interval doesn't clear zero. Weighted fusion at 10-20% BM25 does edge past dense (0.3894 vs 0.3834). But I picked that alpha on the same queries I'm reporting it on, so it's a ceiling, not a result. So my position is narrower than the post sounds — equal-weight RRF is a bad default.
Corpus class: Agreed, and it's more than a caveat. This started as a work problem — support questions coming in from different clients, phrased differently each time. The 54% is a fact about duplicate-question retrieval, not about identifiers: the target is someone else's paraphrase of your problem, and they have no reason to type your file path. n error-code lookup or log search the answer is the artifact, and I'd expect the number to collapse.
That's why the measurement matters more than my result. For each query with an identifier, check whether it appears in any of that query's correct answers. On CQADupStack it's 46%. On your corpus it could be 90%, and then a keyword arm obviously earns its place. Twenty minutes, and it tells you whether anything else in my post applies to you.