Cohere Rerank costs $2 per 1,000 search requests via hosted API — but the same model deployed privately through Model Vault runs $3,250 to $6,500 per month regardless of whether you fire one query or a million. That gap isn't a pricing footnote. It's the entire story of how reranking decisions should be made in 2026.
Reranking has shifted from a model-selection problem to a deployment-topology problem. The choice between hosted API, private instance, and native database integration determines your cost structure, operational complexity, and true total cost of ownership far more than which reranker model you pick. Teams that obsess over benchmark scores while ignoring topology are optimizing the 20% variable and ignoring the 80% fixed.
Here's why that matters: most RAG failures aren't model problems — they're retrieval pipeline problems. Reranking is the highest-leverage fix you can apply, but only if you deploy it in a topology that matches your traffic patterns and latency budget.
What Does a Reranker Actually Do in a RAG Pipeline?
A reranker is a second-stage precision layer that re-scores the candidates your first-stage retrieval already pulled — typically the top 50 to 100 documents — to move the most relevant chunks to the top of the LLM context window. First-stage retrieval (BM25, vector search, or hybrid) is built for speed and recall. It compresses each document into a vector or bag of terms, which loses detail. Reranking examines that shortlist closely and catches distinctions the first stage misses.
The performance gap is real. Cross-encoder rerankers outperformed BM25 on the MS MARCO passage-ranking leaderboard by a wide relative margin — the approach that first topped that benchmark. Reranking also counteracts what researchers call context rot: LLM attention is highest at the start and end of the context window and degrades when relevant information sits buried in the middle. A reranker moves the best chunks into those high-attention positions before the LLM ever sees them.
Think of it this way: your retriever casts a wide net and pulls 50 fish. Most are roughly the right species. The reranker is the person who sorts through the catch and puts the prize fish on top. Skip that step and your LLM context fills with approximately-right chunks instead of the actually-right ones.
Which Reranker Models Should You Compare?
The model landscape has bifurcated along two axes: latency budget and deployment model. Here's how the top contenders stack up.
Cohere Rerank is the production default for accuracy-first pipelines. Cohere Rerank 3.5 is the standard model, while Rerank 4 Fast is a newer variant optimized for lower latency, per ToolChase's review. Teams that adopt Cohere Rerank routinely see 10–30% improvements in end-to-end RAG answer quality without changing their embedding model or LLM. Latency runs approximately 595–603ms, which Voyage Rerank 2.5 matches — both target pipelines that can absorb ~600ms.
Jina Reranker v3 is the only top-tier reranker under 200ms latency, hitting 81.33% Hit@1 at 188ms, making it the choice for sub-200ms production RAG budgets. It supports a 131k-token context window and listwise scoring of 64 documents at once. The newer v3.5 is a 0.6B-parameter listwise reranker that beats Qwen3-Reranker-4B on BEIR and runs up to 56% faster than v3, with a 9.6 nDCG@10 gain on semi-structured retrieval. It's a drop-in replacement for v3 — no code changes required for existing users.
Nemotron edges Jina on accuracy at 83.00% Hit@1 but costs you 243ms, per the same latency benchmark comparison.
Ettin Reranker is the open-source contender. Six CrossEncoder models (17M–1B parameters) released May 19, 2026, under Apache 2.0 license. The 150M variant is 2.3x faster than comparable ModernBERT-base rerankers, making it a strong self-hosting starting point.
| Model | Pricing | Latency | Best For |
|---|---|---|---|
| Cohere Rerank 3.5 | $2 per 1,000 searches | ~595–603ms | Accuracy-first hosted pipelines |
| Jina Reranker v3 | — | 188ms | Sub-200ms latency budgets |
| Cohere Model Vault | $3,250–$6,500/month | Dedicated instance | Private deployment, predictable volume |
| Ettin (150M) | Free (Apache 2.0) | — | Self-hosted open-source RAG |
How Does Reranking Pricing Actually Work?
Cohere's hosted Rerank API is priced at $2 per 1,000 search requests, where one search equals one query paired with up to 100 documents under 500 tokens each. Documents longer than 500 tokens are automatically split into chunks, and each chunk counts toward the document total. That sounds simple. It isn't.
Here's the catch: Cohere's public pricing page renders per-token tables client-side via Sanity CMS, making them invisible to scrapers and most LLM crawlers that view the static HTML. For a vendor whose public pitch centers on transparency and retrieval accuracy for enterprise customers, that's a strange choice. You'll need to cross-reference OpenRouter, AWS Bedrock, or Cohere's own docs to get the full picture.
The bigger pricing trap is private deployment. Cohere Model Vault costs $3,250–$6,500/month for dedicated Rerank instances, creating a fixed cost floor that applies regardless of query volume. A 50-developer team using Model Vault incurs $3,250–$6,500/month in fixed instance costs, as Model Vault pricing is capacity-based and independent of team size. You're paying for idle capacity when traffic drops.
And that's just the reranker. In private search operations, Embed indexing and refresh costs can dominate total spend even when rerank API calls appear small. Buyer operations costs — privacy review, source verification, monitoring, incident response, and human QA — are not included in model pricing and must be budgeted separately.
When Should You Choose Private Deployment Over Hosted API?
The industry narrative says private deployment saves money and gives you control. The data says otherwise for most teams.
Model Vault private deployment is not automatically cheaper, safer, more private, or more accurate than hosted API. It should only be chosen when privacy, predictable volume, latency, dedicated capacity, or procurement requirements justify dedicated infrastructure. Private capacity creates a fixed instance-hour floor where low traffic still requires full billing — and embedding refresh and idle capacity dominate private search operations.
Here's the decision framework I'd use:
- Start with hosted API when search volume is moderate, data-handling policy allows hosted inference, and you want costs to track usage. Pay-per-query elasticity means you pay nothing when traffic is zero.
- Move to Model Vault only when you have hard privacy constraints, predictable high volume that makes the fixed floor economical, dedicated latency requirements, or procurement rules that mandate private infrastructure.
- Consider database-native reranking when you want to eliminate external communication entirely. MongoDB Atlas Native Reranking (public preview) performs reranking directly within the aggregation pipeline, improving search quality by up to 30% while processing queries entirely inside the database.
The pattern I've observed: topology determines 80% of TCO while model choice is increasingly a commodity variable with diminishing returns at the top end. Pick your deployment topology — database-native, cloud-managed, or API-hosted — before evaluating models. The model is the last decision, not the first.
What Latency Budget Should You Target?
Latency is the axis most teams get wrong because they treat it as a spec sheet number rather than a user experience constraint. A reranker is a synchronous hop that the user waits on. That tension between accuracy and speed is the whole story.
For sub-200ms production RAG budgets, Jina Reranker v3 is the only top-tier option at 188ms with 81.33% Hit@1. Nemotron hits 83.00% Hit@1 but costs you 243ms — that 55ms gap matters when you're inside a real-time user interaction.
For accuracy-first pipelines that can absorb ~600ms, Cohere Rerank 3.5 and Voyage Rerank 2.5 both sit at approximately 595–603ms. These target async or batch workflows where the user isn't staring at a spinner.
Jina Reranker v3 uses a late interaction architecture, encoding queries and documents separately into token embeddings and comparing them to produce sortable scores. This makes it agnostic to the first-stage retrieval method — BM25, vector, or hybrid. The reranker works the same regardless of how candidates arrived.
There's also a contrarian efficiency story worth watching. Leaderboards and vendor guides assume larger models deliver better accuracy, with Qwen3-Reranker-4B and seven-times-larger models positioned as top-tier. But Jina Reranker v3.5 (0.6B parameters) beats Qwen3-Reranker-4B on BEIR and runs 56% faster. Ettin's 150M parameter variant is 2.3x faster than comparable ModernBERT-base rerankers. The efficiency frontier is shifting, and smaller models are winning where they shouldn't — at least according to conventional wisdom.
How Do Open-Source Rerankers Compare to Hosted APIs?
The open-source reranker landscape has matured significantly in 2026, and the tradeoffs have shifted.
The Ettin family — six CrossEncoder models from 17M to 1B parameters, all Apache 2.0 licensed — lets you deploy without an API call or licensing fee. The 150M variant is the recommended starting point for most server-side RAG pipelines, offering a defensible size/accuracy trade-off. The open training recipe and distillation dataset make domain-specific fine-tuning possible for teams with labeled relevance data, which is a meaningful differentiator from most model releases.
But there's a caveat: all performance figures are author-reported. Independent evaluation is pending. Wait for third-party benchmarks before deploying the 1B model in high-stakes production workflows.
Jina's late interaction architecture deserves attention here too. By encoding queries and documents separately, it enables reusable document-side states — you pre-compute document embeddings once and reuse them across queries. That's a structural advantage for self-hosting because it reduces per-query compute. The Elasticsearch integration runs Jina models natively inside Elasticsearch, which is the database-native topology pattern applied to an open-source model.
The real question isn't open-source versus hosted. It's whether your team can operate the infrastructure after the initial setup enthusiasm fades. If you have GPU capacity and the engineering bandwidth to maintain it, open-source rerankers eliminate per-query costs. If you don't, the hosted API's per-search pricing is insurance against operational debt.
What Should You Deploy First?
Start with topology, not models. Here's the sequence:
- Define your latency budget. If you need sub-200ms, your field narrows to Jina Reranker v3 immediately. If you can absorb ~600ms, Cohere and Voyage enter the conversation.
- Estimate monthly query volume. Below roughly 1.6 million searches per month, hosted API at $2 per 1,000 searches is cheaper than Model Vault's $3,250/month floor. Above that break-even, private deployment starts making economic sense — but only if volume is predictable.
- Audit your hidden costs. Embedding refresh cycles, privacy review, monitoring, human QA, and incident response aren't in the model price. Budget them before making an economics claim.
- Pick your model last. By the time you've settled topology and latency, the model choice is often obvious.
The teams that win with reranking aren't the ones who pick the highest-scoring model on a leaderboard. They're the ones who match their deployment topology to their actual traffic patterns, budget honestly for operational costs, and treat the model as a swappable component rather than a strategic commitment. If your embedding model choice and reranker topology are misaligned, no amount of benchmark optimization will save you.
The open question for your team: are you paying for idle capacity you don't need, or are you paying per-query for volume that would be cheaper on a dedicated instance? Run the numbers on your actual traffic distribution before signing anything.
Originally published at SaaS with Alex
Top comments (0)