Chirag (Srce Cde) for AWS Community Builders

Posted on Jul 12 • Originally published at srcecde.me

Reciprocal Rank Fusion (RRF): how it works and when to skip it

#rag #ai #machinelearning

Hybrid search is commonly used in the search systems which runs several retrievers for a given same query (For ex: a keyword retriever, a dense embedding retriever) and each retriever returns its own top-k ranked list of documents. User cares about a single list and not lists of list from different retrievers, so you have to merge those lists into a single list. The most common way to merge the ranked results from different retrievers is to use Reciprocal Rank Fusion.

In this post I'll walk through three things -

RRF equation and what it actually computes
Why RRF works smoothly across different retrievers but averaging the scores across retrievers fail
Tradeoff when RRF is the wrong tool

Why averaging scores across retrievers do not work

Suppose there are two retrievers. Each retriever returns the top-k matching document list along with the score. So to merge the results into a single list, why not average their two scores and sort by average?

Picture the retrievers as judges scoring the contestants.

Judge 1: BM25
Judge 2: Dense retriever

BM25 scores can range from 0 upto tens with no fixed ceiling and the exact number depends on term frequency and corpus statistics. Dense retriever scores using the cosine similarity which is bounded between -1 and 1. Both the retrievers has very different range in terms of scores. For example - For a single document say BM25 assigned the score of 32 and Dense retriever assigned it 0.9 (quality match) and the average becomes (32 + 0.9)/2 = 16.45. 16.45 is almost half of 32 and it means that the judge 2 (dense retriever) score barely moved the result. It just shifted the average by less than half a point 0.45. The larger score (BM25 - 32) decided and dominated the result.

Point to note here is that whenever you average two quantities whose ranges differ by an order of magnitude, the one with the wider range dominates the sum, so naive averaging doesn't blend two retrievers at all. It quitely picks whichever retriever emits the bigger raw numbers and lets it win not because that retriever is right more often, but because of the scale its scores happen to live on.

The fix here is to put the two judges on equal scale and RRF does it with a very simple move. It throws the scores away entirely and keeps only each retrievers ranking.

RRF equation explained

We have three documents (A, B, C) and two retrievers that ranked them like this (deliberately skipping the score (distance, similarity) here):

BM25: A, B, C
Dense: B, C, A

Each retriever here represents one of our judges and each document is a contestant. Every judge has handed in a ranking placing the contestants first, second, third and our job is to turn two rankings into one final standing using RRF. RRF says for each document, go to every judge, look at the place/rank it gave that document and turn that place into 1 / (60 + rank/place). Then add those numbers up across both judges. The 60 is a constant I'll explain shortly.

Let's calculate for document B first. We find B on each retrievers ranking and read off its place/rank:

BM25 placed B second - 1 / (60 + 2) = 1/62 = 0.0161
Dense placed B first - 1 / (60 + 1) = 1/61 = 0.0164

Add them: 0.0161 + 0.0164 = 0.0325. That is B's fused score. Now the same for A and C:

A was placed first, third - 1/61 + 1/63 = 0.0164 + 0.0159 = 0.0323
C was placed third, second - 1/63 + 1/62 = 0.0159 + 0.0161 = 0.0320

So the final standing is B (0.0325), then A (0.0323), then C (0.0320). And B wins here.

Why B wins? B was ranked near the top by both judges, a first and a second. A was ranked first by BM25 but last by the dense retriever. C never did better than second. RRF rewards the contestant both judges rank near the top over the one a single judge loves and the other ranks low. So the broad agreement beats one strong opinion.

Now that we've worked out the numbers, here is the same rule written as a formula:

RRF_score(d) = Σ_r  1 / (k + rank_r(d))

Reading it, term by term: d is the document we're scoring. Σ_r means we sum over every retriever r (our two judges). rank_r(d) is the rank retriever r gave document d, counting from 1 at the top. And k is that constant, set to 60 in our example. So the formula says exactly what we just did by hand: turn each retriever's rank into 1 / (k + rank), then add across retrievers.

Two choices in that formula are deliberate, and both are worth understanding.

First is the reciprocal, the 1 / rank shape. We want better rank to worth more, so you take one over the rank, which makes 1/1 > 1/2 > 1/3. Rank 1 beats rank 2 beats rank 3, and the reward keeps shrinking the further down the list you go. That is the "reciprocal" in reciprocal rank.

Second is the + k. Its job is to soften how much the very top of each list dominates. Consider what happens without it: rank 1 would score 1/1 = 1.0 and rank 2 would score 1/2 = 0.5, a two-fold drop between first and second. A single retriever putting a document first would then almost decide the whole result. With k = 60, rank 1 scores 1/61 ≈ 0.0164 and rank 2 scores 1/62 ≈ 0.0161, nearly the same. A large k makes the judges generous: being in the top few matters, but the exact place barely does. A small k makes them strict: first place towers over everything below. k is the knob for that behaviour, and 60 is the value from the original 2009 paper, a reasonable default and not a law. If you want RRF to trust a confident #1 more, you lower k.

Why RRF is scale-invariant

Let's look at the example where BM25 assigned 32 as score & dense retriever assigned 0.91 along with the RRF equation. Within RRF equation there is no variable which takes input as retrievers score. It only looks at the rank of each document within the retrievers output. Rank doesn't care what scale the scores were on. If you multiply every one of BM25's scores by a thousand, take the log or add a constant, the documents will still come out in the same order, so their ranks don't change, so their RRF contributions don't change. Any transformation that preserves the ordering leaves RRF untouched. The scale is discarded the moment we read off the places/ranks and that's what makes RRF robust in the ways people like.

A retriever whose scores live on a wild scale can't overpower the others, because its magnitudes never enter the sum. The scale-mismatch problem from earlier (averaging) simply doesn't arise.
A retriever that returns broken scores (miscalibrated, all negative, or NaN where it failed) can't poison the result through those numbers as long as it produces a sensible ordering, RRF can use it, because the scores are ignored.
You don't need to normalize anything, tune relative weights, or know each retriever's score distribution. The ranking is already a common currency across all of them.
Averaging fails because it trusts scores that aren't comparable; RRF sidesteps the problem by not trusting any score at all, and reading only the order.

But look at the last point - Not trusting any score also means discarding everything the scores were telling you beyond their order and that is where RRF starts to cost you.

The tradeoff: what scale invariance throws away

RRF gets its scale invariance by discarding the scores and the margins between scores and it can be a fine trade but sometimes the margin was the whole point.

Each judge scores the contestants and ranks them by those scores. The margin lives there: how far ahead the winner actually was. RRF never looks at the scorecards. It keeps only the placing the scores produced, 1st, 2nd, 3rd. A dominant win and a near-tie come out as the same placing, so the margin on the scorecards is gone. If that margin was what set a strong match apart from a weak one, RRF has already discarded it. Let's try to understand it with an example.

Suppose the dense retriever returns first document with 0.95 as cosine similarity and every other match is scored at 0.30. That gap is the retriever telling you, that the first document is the answer and the rest of the matches are noise and RRF hears none of it. RRF only records rank 1, rank 2, rank 3 evenly spaced, and the retriever's confidence (which lived entirely in the size of that gap) is gone. So RRF is blind to by how much one document beat another.

This blindness shows up in three practical places.

When one retriever is much stronger than the other - Plain RRF gives every retriever an equal vote. If your dense retriever is genuinely good and BM25 is just along for the ride, a weak ranker still gets an equal say and can pull the result down. A 2025 study, Balancing the Blend, documents exactly this and calls it the weakest-link effect: a single weak retrieval path measurably drags the fused result down, so it pays to check each path's quality before fusing rather than tipping everything into one equal-weight pot. There is a partial fix worth knowing: weighted RRF attaches a weight to each retriever's term, so you can make the dense ranker count more than BM25. But that only tunes how much each retriever counts. It still can't express how much one document beat another inside a ranker, which is the margin we lost above and can't get back.

When you need to know whether anything is actually good - Recall the three fused scores: 0.0325, 0.0323, 0.0320. You can put a threshold on numbers like these, but it won't mean what you want: the score reflects how highly the retrievers ranked a document, not how relevant it is, so there's no value you can use to define "this match is good." The moment your system needs to decide whether any document is good enough (to abstain and return nothing, to gate a RAG answer, to drop weak matches before they reach an LLM), RRF has already discarded the quantity that decision depends on.

When you have very few candidates or many ties - With a short list or lots of tied ranks, the 1 / (k + rank) curve is coarse, k dominates the arithmetic, and the fused ordering turns mushy, while k quietly becomes a number you're tuning with nothing to guide you.

⚠️ Important Note

An RRF score is not a relevance score. Its size is set by k and the number of retrievers, not by how good any match is. With two retrievers and k = 60, a document ranked near the top by both scores close to the 2/60 ≈ 0.033 ceiling whether it is a perfect match or a mediocre one. So the score encodes consensus rank, not relevance and RRF cannot support abstention, confidence gating, or any decision that depends on how good the top result actually is. If your system's job includes saying "I don't have a good answer for this," RRF has thrown away the only thing that answer depends on. This is the single most important limitation of the method.

What research says

There's a common piece of advice that RRF is the sophisticated choice and score-based fusion is a naive trap and this framing is too simple.

The research backs this up, and it cuts against the usual advice that RRF is the sophisticated choice. In the most careful comparison (Bruch et al.), a tuned convex combination of scores beats RRF on the standard BM25-plus-dense benchmark, across all nine datasets they test. Two things fall out of that work worth carrying: RRF's k is a real parameter, not a free lunch, and the value that helps in-domain doesn't transfer to new data, so "no tuning needed" mostly means the parameter was left at 60. And the usual objection to score fusion, that picking a normalizer is fragile, mostly isn't true: reasonable choices come out equivalent once you tune the weight.

It's worth knowing the origin of RRF because it tells us when RRF is still right. RRF was built for metasearch, combining ranked lists from systems whose scores you genuinely can't see, trust or compare. That's the setting it was made for and it's a good default there.

When to use RRF, and when to skip it

Putting it together:

Averaging the raw scores: broken by the scale problem we started with. Don't.
RRF: puts every retriever on equal footing by keeping only ranks. Its robust and needs no configuration, and clearly better than averaging.
A tuned, normalized convex combination: puts retrievers on a common scale while keeping the magnitudes, and beat RRF on all nine datasets in Bruch's main BM25-plus-dense comparison (though not in every one of their other model pairings), as long as you can tune the weight. For two retrievers that weight is a single number α; for more it is one weight per retriever, summing to one.

The choice between the second and third isn't about which is more advanced. It's about your situation.

Reach for RRF when you can't see or trust the scores, the retrievers sit on genuinely incomparable scales, or you have no labeled queries to tune with. Probably a cold start, a zero-shot setup, a grab bag of black-box retrievers that only hand you ranks. It's also a fine first baseline.

There is a standing operational argument for RRF even when you could tune. RRF has nothing to maintain. A convex combination's weight is fit to a snapshot of your data, and it drifts as the corpus, the embedding model, or the score distributions change, so it needs re-tuning and an evaluation harness to catch when it has gone stale. RRF has no such knob to go stale. That robustness is a real feature, not a consolation prize for the cold-start case.

Skip RRF when any of these is true:

You have even a few dozen labeled queries to tune on. With two retrievers, tuning α is a one-dimensional search over a single number between 0 and 1, so it takes very little data (Bruch found under 5% of a training set was enough) and with more retrievers it is a few more weights, still cheap. Once the weights are roughly right, the convex combination is ahead.
You have a retriever whose scores are calibrated, so the margins carry real signal. This is the real crux of the keep the magnitudes argument: it only helps if the magnitudes mean something down the line. An uncalibrated dense model whose scores are a scale artifact gives you margins that are noise, and discarding them, which is exactly what RRF does, is then the right move rather than a loss.
Anything downstream needs absolute relevance (abstention, thresholding, confidence gating), which, as we saw, RRF structurally cannot provide.

The short version: RRF when you can't tune or trust the scores; a normalized, tuned convex combination the moment you can.

Two approaches in code

To make the difference concrete, here are both, small enough to read in full.

RRF first. Notice that the retrievers scores never appear and the function only ever looks at each document's position in each list:

from collections import defaultdict

def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Fuse ranked lists into one. Each list is doc ids, best first.
    Only rank position is used; scores never enter."""
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

For each ranked list, it walks the documents in order, turns each document's position into 1 / (k + rank), and adds that onto the document's running total. That is the whole method.

Now the alternative that keeps the magnitudes. First a small helper to min-max normalize a retriever's scores into [0, 1], then a weighted blend:

def minmax(scores: dict[str, float]) -> dict[str, float]:
    lo, hi = min(scores.values()), max(scores.values())
    if hi == lo:
        return {d: 0.0 for d in scores}
    return {d: (s - lo) / (hi - lo) for d, s in scores.items()}

def convex_fuse(dense: dict[str, float], bm25: dict[str, float],
                alpha: float = 0.8) -> list[tuple[str, float]]:
    """alpha weights the dense retriever; 1 - alpha weights BM25.
    alpha is the single knob, tunable on a few dozen labeled queries."""
    d, b = minmax(dense), minmax(bm25)
    docs = set(d) | set(b)
    fused = {x: alpha * d.get(x, 0.0) + (1 - alpha) * b.get(x, 0.0) for x in docs}
    return sorted(fused.items(), key=lambda kv: kv[1], reverse=True)

The alpha here is the single knob (how much to trust the dense retriever versus BM25), and it's the one number you'd tune on a labeled set.

The thing to notice is what survives to the output. In rrf, the input scores are never read, so the result can tell you an order but nothing about how good any match is. In convex_fuse, the normalized scores flow through into the final number, so that number still carries a notion of relevance you can threshold on. That difference (magnitude discarded versus magnitude kept) is the entire tradeoff, in two functions.

Every number here is from published benchmarks. Which one wins on your data is a different question, so run both on your own queries and measure before you commit.

Final takeaway: RRF works by keeping ranks and throwing away magnitudes. Everything else about it, good and bad, comes from that.

Thank you for reading!

DEV Community