Hybrid search in Python: BM25 + vectors with Reciprocal Rank Fusion, alpha swept 0 to 1

#rag #python #search #machinelearning

Should you fuse BM25 with vector search, and what weight should each side get? I ran both retrievers separately and fused them with Reciprocal Rank Fusion (RRF) across three different query styles, sweeping the fusion weight from pure vector to pure BM25.

Result: no single retriever wins everywhere. Pure BM25 beat pure vectors by 49 points of MRR on exact code identifiers (0.772 vs 0.279). Pure vectors beat pure BM25 by 24 points on hand-written paraphrase questions (0.768 vs 0.566). An RRF blend with 80% weight on BM25 rank was the only setting that stayed within 9 points of the best score on every query type.

The setup

Corpus: the English FastAPI docs, 151 Markdown files, chunked with the heading-aware-plus-heading-path chunker from my chunking benchmark, the strategy that scored best there. 2,713 chunks.
BM25: the same from-scratch implementation as the chunking post (k1=1.5, b=0.75), reused unchanged.
Vectors: sentence-transformers/all-MiniLM-L6-v2, 384 dimensions, cosine similarity via normalized dot product. A small, CPU-friendly model (model card).
Fusion: score(d) = alpha / (60 + rank_bm25(d)) + (1 - alpha) / (60 + rank_vec(d)). Standard RRF with k=60, the constant from the original Cormack, Clarke and Buettcher paper, with a weight alpha instead of a flat 50/50 sum. alpha=1.0 is pure BM25, alpha=0.0 is pure vector search. Each retriever contributes its own top 50 candidates; a document missing from one side gets rank 200 for that side's term.
Three query sets, on purpose:
1. Heading queries (773): section headings used verbatim. They overlap with the indexed heading-path prefix by construction, so this is an easy benchmark, kept for continuity, not as the main result.
2. Identifier queries (305): inline-code spans like response_model or BackgroundTasks that appear in exactly one chunk. This tests exact-token recall, the case hybrid search is usually sold on.
3. Paraphrase queries (30): questions I wrote by hand, like "How do I run code after sending the response to the client?" for the background-tasks page, avoiding the target page's own vocabulary. A hit counts any chunk from the target file. Small sample, so treat the percentages as approximate.

Results

MRR by alpha (0.0 = pure vector, 1.0 = pure BM25):

alpha	heading MRR	identifier MRR	paraphrase MRR
0.0	0.491	0.279	0.768
0.2	0.581	0.471	0.794
0.4	0.642	0.569	0.806
0.5	0.679	0.696	0.751
0.6	0.710	0.735	0.731
0.8	0.782	0.753	0.722
0.9	0.813	0.768	0.678
1.0	0.859	0.772	0.566

hit@1 / hit@5 for the identifier and paraphrase sets:

alpha	identifier hit@1	identifier hit@5	paraphrase hit@1	paraphrase hit@5
0.0	0.207	0.397	0.667	0.933
0.4	0.515	0.656	0.667	0.967
0.8	0.662	0.889	0.533	0.933
1.0	0.692	0.889	0.367	0.867

What the numbers mean

BM25 wins big on identifiers, vectors win big on paraphrases, by more than I guessed. MiniLM's hit@1 on identifier queries was 0.207. A 384-dim sentence model wasn't trained to preserve rare exact tokens like BackgroundTasks; it embeds the gist of the sentence around it, and one API reference page looks a lot like another. BM25's IDF weighting does the opposite: a rare token dominates the score.

Pure BM25 also won the heading benchmark, and that number is misleading. The chunker prepends each chunk's heading path to the indexed text, so querying with the heading is close to a substring search, BM25's best case. Real users don't type section titles. The 30 paraphrase questions are the fairer read on "does semantic search help", and there pure vectors beat pure BM25 by 24 MRR points.

No fixed alpha is best for all three. alpha=0.4 is best for paraphrases, alpha=1.0 for identifiers and headings. Taking the worst-case regret across the three query types, the minimum is at alpha=0.8: 1.9 points below the best identifier score, 8.4 below the best paraphrase score, 7.7 below the best heading score. alpha=0.5, the "obvious" 50/50 choice, loses 18 points on headings.

Honest caveats

The paraphrase set is small and I wrote it. 30 questions is enough to see a 24-point gap, not enough to trust the third decimal place.
The heading benchmark favors whatever puts the query text into the index, BM25's easiest case.
One small local embedding model. Larger or API-hosted embedding models could close some of the identifier gap. I didn't test that.
RRF's k=60 wasn't swept here. Only alpha was. (I did sweep k in a follow-up.)
Single corpus. FastAPI's docs are unusually well structured. A messier corpus would likely show a smaller BM25 edge on identifiers.

The fusion code

K_RRF = 60

def rrf_fuse(bm25_ranked_ids, vec_ranked_ids, alpha, top_k=5):
    """bm25_ranked_ids, vec_ranked_ids: lists of doc ids, best first.
    alpha=1.0 is pure BM25, alpha=0.0 is pure vector."""
    rank_bm = {doc_id: r for r, doc_id in enumerate(bm25_ranked_ids)}
    rank_vec = {doc_id: r for r, doc_id in enumerate(vec_ranked_ids)}
    fallback = max(len(bm25_ranked_ids), len(vec_ranked_ids)) * 4
    candidates = set(rank_bm) | set(rank_vec)
    scored = [
        (
            alpha / (K_RRF + rank_bm.get(d, fallback) + 1)
            + (1 - alpha) / (K_RRF + rank_vec.get(d, fallback) + 1),
            d,
        )
        for d in candidates
    ]
    scored.sort(reverse=True)
    return [d for _, d in scored[:top_k]]

Run BM25 and your vector index independently, each returning its own top-N (50 was enough here), then fuse. No score normalization needed: that's the point of using ranks instead of raw scores, which differ in scale between BM25 and cosine similarity.

Latency on 2 vCPUs over 2,713 chunks: BM25 1.82 ms median, vector encode+search 23.99 ms. A hybrid query costs roughly what the vector half costs.

What I'd do in your pipeline

Don't default to alpha=0.5. On this corpus it lost more than any other tested value on at least one query type. Start from 0.7-0.8 (BM25-leaning) and tune on a real query log.
If your users type API names, error codes or config keys, weight BM25 heavily. Small embedding models are bad at exact tokens; that's structural, not a tuning problem.
If your users type full questions, don't skip vectors. BM25 alone lost 24 MRR points on plain paraphrases.
Measure your own alpha with a golden-set eval in CI built from real traffic instead of a guess.

The full script (bench_hybrid.py), raw results and corpus fetch script are on the original post.

I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: https://recallrun.dev/learn/ai-engineering/?utm_source=devto&utm_campaign=hybrid

DEV Community