Mayur Agnani

Posted on Nov 3 Originally published at theaicommit.com

How to Choose How Many Search Results a Reranker Should Rescore

#rag #performance #llm #ai

Engineer's view. This is the bug where you fetch ten rows with LIMIT, then filter them in app code, and the row you needed was eleventh. A reranker only reorders the results your vector search returned. If it gets too few, the right passage never reaches it. If it gets too many, you pay latency for distractors.

TL;DR. A reranker reads the question and each search result together, so it can fix an order the vector search got wrong. It can only fix what it was shown, so measure how deep the right answers sit before you choose how many to send.

Explain Like I'm 5

Imagine a librarian who sorts books by a one-line summary on each spine. You ask a question. The
librarian grabs the shelf of books whose summaries look closest and hands you a stack.

Then a careful reader opens each book in the stack and checks it against your question. The reader
is slow, but almost never wrong.

The careful reader can only check the books in the stack. If the right book was left on the shelf,
nobody finds it. And a taller stack means a longer wait.

The Problem

You have shipped this bug before, in a system with no model in it.

A page showed a user's ten newest documents. The query ran with LIMIT 10. Then the app removed the
ones the user was not allowed to see. Some pages came back with three items, and support asked
where the rest went. The rows existed. They were rows eleven to forty, and the filter never saw
them. The fix was not a smarter filter. It was fetching enough rows for the filter to work with.

Retrieval for a model has the same shape. Your app embeds the question, asks the vector database
for the nearest chunks, and puts the top few into the prompt. Many teams add a reranker in
between, a second model that rescores those chunks. What the reranker receives is decided by one
number, usually picked once and never measured.

I measured it on a public set of 300 science claims over 5,183 abstracts, using a common small
embedding model. For about one claim in seven, the right abstract sat below rank 20. For one claim in
fourteen it was below rank 100. Here is one claim: "Insulin increases risk of severe kidney
failure." The search ranked the right abstract 28th. The reranker moved it to first. With the
common setting of 20 candidates, it would never have seen it.

The fix: measure how deep your right answers sit, then send the reranker the smallest number of
candidates past which quality stops rising, and keep only a handful for the prompt.

The Fix: Measure the Depth Where Reranking Stops Paying

Why does the search rank the right passage so low?

An embedding model turns text into a list of numbers, a vector, so that similar texts land
close together. It does this to every document once, at indexing time, before any question
exists. A whole abstract becomes 384 numbers. At query time the database only compares vectors.

That is why it is fast, and why it misses. The kidney abstract covers amputation, blindness,
kidney failure and more. Its single vector is an average of all of that. Nothing in it can know
that this question cares only about kidneys.

What does the reranker do differently?

A cross-encoder is a model that reads the question and one document together, as one input,
and returns a relevance score. Every word of the question can attend to every word of the
document. It sees that "severe kidney failure" appears in both, and that the top-ranked abstract
is about lesions, not risk.

It cannot be precomputed, because it needs the question. So it costs one model call per
candidate. On my laptop, the reranker took about 6 milliseconds per pair.

So why not send it everything?

Two curves decide it. Recall at N is the share of right answers inside the top N, and it is
the ceiling: the reranker cannot promote what it never saw. Latency is N times the cost per pair,
a straight line. Quality after reranking follows the ceiling up, then flattens where the ceiling
flattens. Past that knee you pay for candidates that are mostly distractors. A few distractors
fool the reranker, so quality can even dip.

Is the knee in the same place for every corpus?

No. I ran the same two models on 648 finance questions over 57,638 forum posts. The knee came at
20, and going deeper made the ranking worse, even as more right answers got in. Reranking only the
top 5 was worse than no reranker at all.

What This Means for You

When this matters. You retrieve chunks for a model: a chat bot over your docs, a support
agent, a code search tool, an agent that calls a search tool. It matters most when the corpus is
specialized, because a general embedding model averages away the detail that decides relevance.
It also matters when you added a reranker and never checked what it gets.

How it affects you. Too shallow, and your reranker is an expensive way to reshuffle the same
answers, while the right one waits at rank 30. Too deep, and every query pays hundreds of
milliseconds for nothing. For an agent, it compounds. NVIDIA reports that adding a reranker cut an
agent's tokens per query by about 14%, because it read fewer wrong chunks and searched less.

What to do about it. Start with no setup. Find the number in your code that sets how many
results go to the reranker, and the number that sets how many go into the prompt. Write both
down. Many teams find the first is the vector database's default.

Next, label 50 to 100 real questions with the chunk that answers them. A spreadsheet is enough.
Run the search at depth 200 and record where each right chunk lands. That one histogram tells you
how deep you need to go.

Then score the deepest list with the reranker once, and compute quality at each depth from those
scores. Pick the smallest depth near the best, inside your latency budget.

Implementing It

The change.

Four jobs touch this number, and one engineer may own all of them.

Job 1, whoever owns retrieval. Fetch more than you keep. The search returns N, the prompt gets
k. With pgvector, N is the LIMIT:

-- before: the search result IS the prompt
SELECT id, body FROM chunks ORDER BY embedding <=> $1 LIMIT 5;
-- after: fetch N for the reranker; k is chosen later
SELECT id, body FROM chunks ORDER BY embedding <=> $1 LIMIT 50;

If the index is approximate, check its search breadth too. An HNSW index with
hnsw.ef_search below N cannot return N good candidates.

Job 2, whoever calls the reranker. Score the pairs, sort, keep k. Run locally with
Sentence Transformers:

from sentence_transformers import CrossEncoder
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def rerank(query, rows, keep=5):
    scores = ce.predict([(query, body) for _, body in rows], batch_size=64)
    ranked = sorted(zip(rows, scores), key=lambda r: -r[1])
    return [row for row, _ in ranked[:keep]]

Or call a hosted reranker. Cohere returns indexes into your list, best first:

curl -X POST https://api.cohere.com/v2/rerank \
  -H "Authorization: Bearer $CO_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "rerank-v5.0-pro", "query": "...", "documents": ["...", "..."], "top_n": 5}'

A hosted call is billed by usage, so N drives cost as well as latency there.

Job 3, whoever owns the eval. Score the deepest list once, then derive every depth from it.
Reranking the top 20 is sorting the first 20 of those scores, so one pass gives the whole sweep:

for n in (5, 10, 20, 50, 100, 200):
    top = sorted(zip(ids[:n], scores[:n]), key=lambda x: -x[1])
    record(n, recall_at(ids, relevant, n), ndcg_at_10([d for d, _ in top], relevant))

sweep() and choose_depth() in code_example.py do this over a labeled set.

Job 4, whoever assembles the prompt. Keep k small and put the reranked best first. A reranker
that finds the answer is wasted if the prompt then takes 20 chunks anyway.

Getting N wrong raises no error. The model answers from a near-miss chunk, and nobody notices.
One sign is visible, though. If rerank time stays flat when you raise N, the database is
returning fewer rows than you asked for.

How you know it worked.

Two numbers per depth: recall at N before reranking, and nDCG@10 after. nDCG@10 is a 0-to-1
score for how well the top ten are ordered, with credit for putting right answers high. On my
science set, no reranker scored 0.645. Reranking the top 50 scored 0.694, and the top 200 scored
0.691. The chosen depth should sit at that knee. Log rerank time per query too. It should come
out near N times your per-pair cost. If quality never rises above no reranker, the reranker does
not fit your domain, and depth will not save it.

When a Reranker Is the Wrong Tool

When the first search is already right. If your histogram shows nearly every right chunk at rank
1 to 3, a reranker adds latency to fix nothing. Spend the effort on chunking or on labels instead.

When your latency budget is tight. Six milliseconds per pair is a small model on a laptop. A large
reranker is many times slower. Under a 100-millisecond budget you may only afford 10 candidates,
and a better embedding model may buy more than a reranker can.

When the reranker was trained on different text. The one I used learned from web search queries.
On science claims it added about five points of nDCG@10. On finance questions it added two at
best. That is real, but modest. Test a
reranker on your own labeled questions before you trust its leaderboard score.

When the answer needs several chunks together. A reranker scores each chunk alone. It cannot see
that two middling chunks answer the question only as a pair.

My test has limits. It used one small embedding model, one small reranker, public test sets and
one laptop. Your knee will sit somewhere else.

Three questions before you set N:

Where does the right chunk land for your real questions, at depth 200?
What does one reranker call cost you, in milliseconds and in money?
Does reranking beat no reranking on your own labels at all?

Glossary

embedding model — a model that turns text into a vector, so similar texts sit close together.
cross-encoder — a model that reads a question and one document together and returns a relevance score.
Recall at N — the share of right answers found in the first N search results.
nDCG@10 — a 0-to-1 score for how well the top ten results are ordered.
HNSW — a common approximate vector index; its search breadth limits how many good neighbors it returns.

DEV Community