Last year I took over a support chatbot for a client with about 40,000 help center and internal wiki chunks in a vector database. Retrieval was pure dense vector search. The model answered confidently from the wrong chunk about a third of the time. We did not re-embed anything, did not switch databases, and did not fine-tune a model. We added a reranker between retrieval and generation in one afternoon, and answer accuracy on our test set went from 61% to 84%. The reranker cost less than $40 that month.
That is the pattern I keep seeing. Teams assume bad RAG answers mean they need a better embedding model, a bigger LLM, or a new vector database. Usually they need a cheap second pass over what retrieval already found. This post covers how reranking works, what it costs, how to measure the lift, and when to skip it.
Why Vector Search Alone Ranks the Wrong Chunks
Most RAG retrieval uses a bi-encoder. The embedding model compresses the query into one vector and each document chunk into one vector, then returns the closest chunks. It is fast because every document vector is precomputed, so a query over millions of chunks takes milliseconds.
The weakness is structural. The query and the document never see each other. Each becomes a single point in vector space, and similarity is measured afterward. That compression throws away the details that decide relevance: negation, qualifiers, part numbers, and the difference between "refund policy for annual plans" and "refund policy for monthly plans." Those two queries produce nearly identical vectors, so the same wrong chunks win.
The short version of vector search fundamentals: bi-encoders are built for recall. They find 50 plausible candidates well. They do not pick the best one well.
How Two-Stage Retrieval Works
The fix is to split retrieval into two stages with different jobs. Stage one is recall: run your fast bi-encoder search and pull the top 50 to 100 candidates. Cast a wide net; ordering can be sloppy.
Stage two is precision: a reranker scores each candidate against the query. Rerankers are cross-encoders. Instead of embedding the query and document separately, a cross-encoder takes the query and one candidate in a single input and returns a relevance score. The model reads both texts at once, so it catches the interactions a bi-encoder misses: "annual" versus "monthly," "not supported" versus "supported," error code EX-2204 versus EX-2240.
Cross-encoders are slow per pair, so you never run one over the whole corpus. You run it over 50 to 100 candidates, keep the top 5 to 10, and put only those into the LLM prompt. The pipeline is straightforward: the user query hits stage one recall, which runs BM25 plus dense vector search through a bi-encoder and returns the top 50 to 100 candidates. You merge and dedupe that list, then a cross-encoder scores the query against each candidate and returns the top 5 to 10. Only those chunks go into the LLM prompt.
The reranker is a filter between two systems you already have. No re-embedding, no migration, no changes to chunking. That is why it is the biggest quality gain per dollar in most RAG stacks I audit.
Add BM25 for Hybrid Recall First
A reranker can only score what stage one returns. If the right chunk never makes the candidate list, no reranker will save you. Dense vector search has a known blind spot: exact strings. Part numbers, error codes, SKUs, and uncommon acronyms embed poorly because the model has rarely seen them.
BM25, the classic keyword ranking algorithm, catches those literal matches. Run BM25 and dense search in parallel, then merge the ranked lists with reciprocal rank fusion to cover both semantic similarity and exact matches. Most vector databases support hybrid search natively now, including Weaviate, Qdrant, and Elasticsearch; the Elasticsearch hybrid retrieval docs show one implementation. If your corpus contains identifiers of any kind, hybrid retrieval is not optional.
On the client project I mentioned, hybrid retrieval alone moved recall of the correct chunk into the top 50 from 78% to 91%. The two changes compound: better recall gives the reranker better raw material. If you are building a system from scratch, our walkthrough on building a RAG system for a business knowledge base covers chunking and indexing decisions that feed this pipeline.
What Reranking Costs: Latency and Dollars
You have three realistic options in 2026, and the cost difference between them is smaller than most teams assume.
| Option | Type | Cost | Added latency (100 pairs) |
|---|---|---|---|
| Cohere Rerank 3.5 | Hosted API | $2 per 1,000 searches, up to 100 docs per search | roughly 150 to 300 ms |
| Jina reranker | Hosted API | Free tier of 1 million tokens, then per-token pricing | roughly 200 to 400 ms |
| bge-reranker-v2-m3 | Self-hosted, open weights | Fractions of a GPU hour; an L4 runs about $0.70 per hour | roughly 60 to 120 ms on GPU |
Take 10,000 queries per month with 50 candidates reranked per query. With Cohere, each query is one search call since 50 docs fit the 100-doc limit, so 10,000 searches cost $20 per month. Check the Cohere pricing page for current rates. Self-hosting bge-reranker on a modest GPU cuts that to near zero at this volume but adds an ops burden; the model card on Hugging Face lists supported languages and sizes. Under roughly 100,000 queries per month, I tell clients to pay the API bill and skip the infrastructure.
Latency is the real cost to watch. A hosted reranker adds 150 to 300 milliseconds to every query. For a support chatbot where the LLM already takes two to four seconds to answer, that is invisible. For a typeahead search box, it is unacceptable. There is an offset: better chunks mean you send 5 chunks to the LLM instead of 15 mediocre ones, cutting input tokens per call. Prompt size dominates the LLM bill, so smaller, better prompts often pay for the reranker by themselves, compounding nicely with our LLM token cost optimization techniques.
A Working Reranker in 20 Lines of Python
Here is the self-hosted version with sentence-transformers, assuming stage one returned candidate dicts with a text field.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
def rerank(query: str, candidates: list[dict], top_k: int = 5) -> list[dict]:
pairs = [(query, c["text"]) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [c for c, _ in ranked[:top_k]]
candidates = hybrid_search(query, top_k=50) # BM25 + dense, merged
context = "\n\n".join(c["text"] for c in rerank(query, candidates, top_k=5))
The hosted version is one POST to the Cohere or Jina endpoint with the query and candidate texts, then reading back the indices in score order; the Cohere rerank reference documents the request shape. Either way, this is an afternoon of work. The hard part is proving the change helped, which is where most teams get lazy.
Measure the Gain With a Small Labeled Set
Do not ship a reranker on vibes. Build a small evaluation set first: 50 to 100 real queries from your logs, each paired with the chunk that actually answers it. It takes a few hours of manual labeling, and you will reuse it every time you touch retrieval.
Run each query through old and new pipelines. Track two numbers. First, recall at k: for what share of queries does the correct chunk land in the top 5? Second, end-to-end answer quality: grade the final answers pass or fail against the labeled chunk. Teams that also worry about wrong answers from the generation step itself should pair this with detection work; our guide on catching AI hallucinations in business outputs covers that side of the stack.
On the support chatbot project, recall of the correct chunk in the top 5 went from 64% to 89% after adding hybrid search plus the reranker. End-to-end answer accuracy went from 61% to 84%. Those are the receipts you want before and after, not a demo that feels better.
Try two or three rerankers against your labeled set before committing; the gap between models on your data matters more than leaderboard positions.
When Reranking Is Not Worth It
Reranking is not always worth it. Skip it, or defer it, in these cases.
Tiny corpora. If your whole knowledge base is under roughly 2,000 chunks, vector search alone usually retrieves the right material, and a reranker adds latency without moving accuracy.
Corpus fits in context. If the whole relevant document set fits in the prompt, retrieval quality barely matters. Send more chunks instead.
Broken stage one. If your labeled set shows the correct chunk is missing from the top 50, the problem is recall, not ranking. Fix RAG chunking strategies, metadata, or hybrid retrieval first. A reranker cannot score a chunk it never sees.
Hard latency budgets. Real-time autocomplete, sub-200-millisecond search UIs, and voice pipelines with tight turn-taking cannot absorb an extra 150 to 300 milliseconds. Cache reranker results for repeated queries if you must have both.
Also skip reranking with a general LLM like GPT-4o as the scorer. It works, but it costs 10 to 50 times more per query and it is slower. Use the right tool class. The same principle applies to wiring retrieval into tools: our MCP guide for small businesses explains why protocol fit matters more than raw model power when connecting systems.
What Most Guides Get Wrong
Most reranking guides start with a leaderboard and end with a benchmark table. That ordering is backwards. The model you pick matters far less than whether the correct chunk reaches the candidate list at all, which is a stage one problem that no reranker can fix. I have audited stacks where the team benchmarked four rerankers on MTEB while their chunker split every document mid-table, guaranteeing the answer-bearing text never survived retrieval intact. Guides also treat reranking as a search quality tool only and ignore its second effect: shrinking the prompt. Sending five sharp chunks instead of fifteen loose ones cuts input tokens on every call, which often pays the reranker bill before accuracy gains are even counted. Another common miss is the claim that you must self-host to get costs down. At under 100,000 queries a month, a hosted API at $2 per 1,000 searches is cheaper than the engineering hours to keep a GPU alive. Guides that demo reranking with toy Wikipedia corpora also hide the failure mode that actually hurts businesses: exact identifiers like part numbers and error codes that dense search flattens into noise. Finally, almost none of them tell you to build the labeled evaluation set first. Without 50 to 100 graded queries, you cannot tell whether the reranker helped, and you will ship on demo vibes.
Conclusion
If your RAG system has more than a few thousand chunks and users complain about wrong or vague answers, add a reranker before you touch anything else. The order of operations I follow on every engagement: hybrid BM25 plus dense retrieval first, a cross-encoder reranker second, then, only if the labeled set still shows gaps, re-embedding or fine-tuning. Most teams never reach step three.
Budget one afternoon for the integration, a few hours for the labeled evaluation set, and $20 to $50 per month in API costs at small business query volumes. In return you typically get a 15 to 25 point jump in answer accuracy, fewer hallucinated answers grounded on the wrong chunk, and smaller prompts that cut your LLM bill. Reranking is the rare upgrade that is cheap, fast to ship, and easy to measure. Ship it first.
Top comments (0)