Retrieval gets the right documents into the pile; a reranker gets them into the right order. Most RAG pipelines stop at the first step and wonder why the answer is subtly off. The fix is a cheap second stage that reorders the shortlist — and understanding why it works comes down to one architectural difference between two kinds of encoder.
The bi-encoder — fast recall, coarse order
The retriever every RAG system starts with is a bi-encoder: it turns the query into one vector and each document into one vector independently, then ranks by cosine similarity. Document vectors are computed once, offline, and an approximate-nearest-neighbour index searches millions of them in milliseconds.
bi = SentenceTransformer("all-MiniLM-L6-v2")
doc_emb = bi.encode(corpus) # PRECOMPUTED, offline, indexed
q_emb = bi.encode(query)
scores = util.cos_sim(q_emb, doc_emb)[0]
topN = scores.topk(N).indices.tolist() # a broad shortlist, fast
It pays for that speed with a blind spot: each document is squashed into a single vector before it ever sees the query, so the score can't reflect how this query and this document interact. The result is strong recall (the useful docs are somewhere in the top-N) but coarse precision at the very top — a doc that merely shares surface words can out-rank the one that actually answers the question. And in RAG the model reads the top chunk, so a wrong #1 becomes a wrong answer that no downstream prompt fixes.
The cross-encoder — slow, but reads the pair jointly
A cross-encoder concatenates [query] [SEP] [doc] and runs the pair through the transformer together, so every query token attends to every document token. It emits one calibrated relevance score.
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, corpus[i]) for i in topN]
rel = ce.predict(pairs) # e.g. [0.41, 0.96, 0.34, 0.04]
That joint attention models exactly the query-doc interaction the bi-encoder threw away — far more accurate. But there is nothing to precompute, because the score depends on the query, so you cannot run it over a whole corpus. A million forward passes per query is a non-starter.
Retrieve-then-rerank — each stage where it's strong
Combine their strengths. Stage 1: the bi-encoder sweeps the whole corpus for a broad top-N (recall, cheap). Stage 2: the cross-encoder re-scores just those N pairs and reorders them into a precise top-k (precision, sharp). You pay the expensive model only N times per query.
def retrieve_then_rerank(query, N=50, k=5):
# STAGE 1 — cheap, over the whole corpus
scores = util.cos_sim(bi.encode(query), doc_emb)[0]
shortlist = scores.topk(N).indices.tolist()
# STAGE 2 — expensive, only over N pairs
rel = ce.predict([(query, corpus[i]) for i in shortlist])
order = sorted(range(N), key=lambda j: rel[j], reverse=True)
return [shortlist[j] for j in order[:k]]
The doc the bi-encoder buried at rank 2 or 3 is now #1; the keyword decoy drops. This reordered top-k is what the LLM actually reads.
Tuning N — the recall/latency dial
N is the crucial knob. The reranker can only reorder what stage 1 surfaced: if the true answer sits at bi-encoder rank 60 and you rerank the top 50, it never gets a chance — bigger N means higher recall. But every extra shortlisted doc is another cross-encoder forward pass, so latency grows linearly with N. Typical N is 50–200; measure recall@N on your own data and pick the smallest N past the knee of the curve. Concretely, stage 1 is a sub-millisecond ANN lookup and stage 2 is a few ms per doc that batches well, so reranking ~100 docs adds tens of milliseconds — a bounded, predictable tax for a large jump in top-order quality.
In production you rarely host the cross-encoder yourself: reach for Cohere Rerank, Voyage rerank, or an open bge-reranker, drop it between your vector store and the LLM, and hand the precise top-k to the model. Because the #1 chunk is now the real answer, the model is properly grounded.
Pick a query, slide the top-N cut, and watch the two rankings reorder side by side at https://dev48v.infy.uk/ai/days/day56-rerankers.html
Top comments (0)