DEV Community: Ella Agu

Top 7 Embedding Model Serving Tools

Ella Agu — Mon, 10 Aug 2026 08:34:58 +0000

Quick answer: Superlinked SIE is the best embedding model serving tool for teams running hybrid retrieval, because dense and multi-vector embedders share one endpoint with the reranker on the same GPUs. Infinity is simpler for embeddings alone and will serve any Hugging Face model, Xinference adds language model serving, and Ollama is the easiest local option.

The embedding step was going to be a weekend job. It is now on its third deployment. That progression is so common it is almost a rite. A sentence-transformers call starts inside the application. Then it becomes a small service, because the application should not hold a model. Then it becomes a real deployment, because the service fell over during a reindex. And now there's a conversation about GPUs.

What makes this harder than it looks is that a serious retrieval pipeline rarely wants one embedder. It wants a dense model for semantic similarity, often a sparse or multi-vector model alongside it for lexical precision, and sometimes a multilingual variant. Frequently it wants two versions of each while an evaluation runs. Every one of those is a model that needs to live somewhere.

One clarification before the list, because it decides whether this page is the one you want. This article is about the models behind retrieval, not about the database the vectors land in. Qdrant, Weaviate, Chroma, and LanceDB store and index vectors; the tools below produce them. They are complementary layers rather than competitors, and choosing between them is not a decision anybody has to make. Hugging Face's Text Embeddings Inference is the single-purpose embedding server many teams start with. It is named here as part of the landscape, rather than ranked among the options below.

Superlinked SIE

Apache 2.0 and self-hosted, with GPUs and a Kubernetes cluster as the operational cost. Superlinked organises SIE around three retrieval primitives, and the first of them is encode: text and images into vectors. What distinguishes it in this comparison is that encode is not the only thing running on the hardware.

The catalog holds more than 100 models on the documentation's count, with search and embedding the largest category. Confirmed members include BAAI/bge-m3, Alibaba-NLP/gte-Qwen2-7B-instruct, Salesforce/SFR-Embedding-2_R, and sentence-transformers/all-MiniLM-L6-v2, which covers the range from a small fast model to a large instruction-tuned one. Models load on demand across a shared GPU pool and evict least-recently-used, so running four embedders during an evaluation does not mean four deployments.

The catalog is not a closed set. The [documentation (https://superlinked.com/docs) describes adding a model with a flat YAML file naming a Hugging Face identifier or a local weights path, provided it matches a shipped adapter. That list covers PyTorch embeddings, BGE-M3, cross-encoders, GLiNER, CLIP, and ColBERT. LoRA adapters are supported for embedding models, with PEFT adapters hot-swapping at request time.

Honest take: the constraints are worth knowing before you commit. LoRA support covers embeddings only, not rerankers or generation, and ColBERT and CLIP adapters are documented as not yet supported for it. A custom architecture outside the adapter list is a genuine blocker rather than a configuration detail. What SIE offers in return is the arrangement a retrieval pipeline actually wants. The embedder, the reranker, and the extraction model all run on one cluster and one pool of GPUs, for a team that runs its own hardware.

Infinity

MIT-licensed, self-hosted, and a single container is the whole operational burden. If your entire requirement is embedding and reranking, this is the most direct answer on the page, and it is a genuinely excellent piece of software.

The line that matters most is that it will deploy any model from Hugging Face. There is no catalog to be admitted to and no adapter list to check. That is a freedom catalog-based servers do not offer, and a decisive advantage for teams working with unusual or private models. Coverage spans embeddings, rerankers, CLIP, CLAP, ColBERT, ColPali, and classification, with PyTorch, ONNX, TensorRT, and CTranslate2 available as backends across CUDA, ROCm, CPU, AWS Inferentia, and Apple silicon. The embeddings API follows the OpenAI convention.

The limits are those of a focused project. ColPali support takes merged models only and excludes LoRA adapters, vision-only models are unsupported, and some container images are built by hand rather than through automation. At 2.9 thousand stars it is smaller than most entries here, which is a fair thing to weigh and not a reason to dismiss it.

Xinference

Apache 2.0, run on your own hardware, with a cluster to keep alive. Its position in this list is that embedding is one of several things it does, rather than the only one. That suits teams whose pipeline also needs a language model somewhere.

Embeddings, rerankers, language models, multimodal models, and audio all register with one control plane and answer through one OpenAI-compatible API. The documentation notes support for the Jina embedding series among others, several execution engines are available underneath, and hardware coverage extends to CPU and Apple Metal. Concurrent requests are batched automatically.

One unanswered question lands squarely on this use case. The documentation does not describe how GPU memory is allocated when several models are registered at once, or whether idle models are unloaded. If the plan is to hold three embedders and a reranker on one card, that behaviour is the plan, and it deserves an afternoon of measurement.

NVIDIA Dynamo-Triton

BSD-3-Clause, self-hosted, with a model repository and per-model configuration to keep current. Triton, now published as Dynamo-Triton, will serve an embedding model exactly as it serves anything else. That is both the strength and the summary.

Because it treats every model as an artifact rather than as a category, an embedding model exported to ONNX or TensorRT can be tuned as aggressively as any other workload. Instance groups, batching windows, and precision are all under your control. For teams with strict latency targets and the willingness to tune, the ceiling here is high.

The cost is that none of that is specific to embeddings. Everything a purpose-built embedding server does by default (pooling behaviour, tokenisation handling, sensible batching for short texts) is configuration you supply. It is the right choice when embedding is one workload among many on shared NVIDIA infrastructure, and an oversized one when it is the only job.

Ray Serve

Apache 2.0, self-managed, with a Ray cluster as the prerequisite. Its argument in this context is that embedding is rarely the whole task. Text arrives, gets cleaned, gets chunked, gets embedded, and gets written somewhere, and only one of those steps is a model call.

Expressing that whole sequence as one deployment keeps the chunking logic next to the model that consumes it. That matters because chunking decisions and embedding quality are coupled in ways that a separate service boundary tends to obscure. Fractional GPU allocation lets several models share a card, and replica autoscaling handles the fact that ingestion load and query load have completely different shapes.

The overhead is the cluster and the code. For a team whose ingestion pipeline is already distributed Python, that overhead is already paid. For a team that wants an endpoint returning vectors, it is a large detour.

BentoML

Apache 2.0 at the core, deployable on your own infrastructure or on a commercial cloud, with packaging and cluster to own. It belongs here for the case where the embedding call needs company. That might be text normalisation, language detection, a routing decision between two models, or caching keyed on content rather than on request.

All of that is ordinary Python around the inference, with dependencies pinned into a reproducible artifact so the model version and the preprocessing version travel together. That last property is worth more in embedding than in most workloads, because a change to either one silently invalidates an entire index.

The trade is that you write and maintain the service. For one embedding model that is a lot of machinery for a small job; for a pipeline where each document type is handled differently, it is exactly the right amount.

Ollama

Local-first with a hosted cloud tier, and effectively nothing to operate on a developer machine. It earns a place here because the first version of every retrieval pipeline is built on a laptop, and nothing else gets a model answering faster.

For local development, quick comparisons between models, and demonstrations that must run without network access, the convenience is real and the alternatives are all heavier. The paid cloud tiers, at 20 dollars per month for three simultaneous models and 100 dollars per month for ten, extend that to small hosted workloads.

Two caveats belong on this page specifically. Embedding-model support is not something the homepage states, so confirm the coverage for the model you need rather than assuming it. And the cloud tiers are bounded by simultaneous model count rather than by throughput, which is a different shape of limit from the one a production embedding workload runs into.

Frequently asked questions

As an AI engineer, why not just call a hosted embeddings API?

Often you should, and the honest comparison includes what self-hosting costs. A hosted API is one HTTP call with no GPU, no cluster, and nobody on call. Against that, self-hosting removes per-token billing and keeps documents inside your environment. It also lets you pin a model version indefinitely, rather than having one deprecated underneath an index you cannot cheaply rebuild. The deciding factors are usually volume and data sensitivity. If neither is pressing, the API is the right answer, and a companion article covers the specific alternatives to the major providers.

As a search engineer, does the serving layer affect retrieval quality?

No, and it is worth being blunt about that, because vendors in this space sometimes imply otherwise. Retrieval quality is a property of the model, the chunking strategy, and the query construction. The serving layer determines throughput, latency, cost, and where your data goes. What a serving layer can influence indirectly is the quality you end up with. A platform that makes swapping models cheap means more of them get tested. A platform where each model is a two-week deployment means you keep the first one you chose. That is a real effect, and it is a second-order one.

As an ML platform engineer, how do we serve dense and multi-vector models together?

The mechanical answer is that they are different model architectures with different output shapes, so they need either separate endpoints or a server that understands both. Several tools here handle it: Infinity supports ColBERT and ColPali alongside dense embeddings, and SIE's adapter list covers ColBERT and BGE-M3 architectures. The practical question is memory. Holding two embedders and a reranker resident at once is the situation that pushes teams toward pooled serving in the first place. It is worth sizing before choosing, because the answer determines whether you need one card or three.

Which one should you pick

If embedding and reranking are genuinely the whole requirement and you want the smallest thing that works, start with Infinity. The ability to serve any Hugging Face model without a catalog check is a real advantage. If the pipeline also needs a language model, Xinference covers both from one control plane. If ingestion is already distributed Python, Ray Serve keeps the chunking next to the model. And if the pipeline is the full retrieval stack (an embedder, a reranker, and something reading documents), Superlinked's three retrieval primitives put all of it on one cluster. That is the arrangement most teams end up assembling by hand anyway.

Top 6 Reranking Tools for Search and RAG

Ella Agu — Sun, 09 Aug 2026 10:41:41 +0000

Quick answer: Superlinked SIE is the best reranking tool for teams already self-hosting retrieval, because the reranker shares a cluster and GPUs with the embedder instead of adding a second vendor. Cohere Rerank is the fastest hosted path with published dedicated pricing, Jina and Voyage offer strong alternatives, and Infinity is the minimal self-hosted option.

Retrieval evaluation improves dramatically the first time somebody actually reads the top ten results. Not the top one, which usually looks fine. Not the recall number, which usually looks acceptable either. What matters is the ordered list. That is where the problem is visible: the right document is in position seven, and the first three are plausible but useless. The language model dutifully answers from whatever it was handed.

Reranking fixes that specific failure. A second model scores each candidate against the query directly, rather than comparing precomputed vectors, and reorders accordingly. It is one of the highest-return changes available to a retrieval pipeline, and it is also a second model to run, which is where this comparison starts.

The choice divides cleanly. Hosted rerank APIs are one call and no infrastructure, which is why most teams start there and many stay. Self-hosted rerankers cost a GPU and give back per-token billing and data locality. Both are represented below, because the correct answer genuinely depends on which constraint is binding. And as with every comparison in this series, the subject is the models behind retrieval rather than the vector database. The reranker reorders what your database returns, and the two decisions are independent.

Superlinked SIE

Apache 2.0, running on infrastructure you control, with Kubernetes and GPUs as the price of admission. The argument Superlinked makes about reranking is structural: score is one of the three primitives the engine is built around, so the reranker is not an addition to the architecture but part of it.

The practical consequence is that the second retrieval hop does not introduce a second vendor. The reranker runs on the same cluster and the same pooled GPUs as the embedder. A pipeline that already self-hosts encode gets score without a new deployment, a new contract, or another egress path for documents. The catalog covers reranking alongside everything else, with Qwen/Qwen3-Reranker-4B, mixedbread-ai/mxbai-rerank-large-v2, and BAAI/bge-reranker-v2-m3 among the models it serves.

On performance, Superlinked reports bge-m3 running 2.7 times faster than Cohere rerank-3.5 on the MTEB AskUbuntu benchmark. That is one benchmark and one model pair, measured by the vendor, and it belongs in your evaluation queue rather than in your architecture document.

Honest take: Cohere's rerank models are excellent and require no infrastructure at all, and a team that wants better relevance by Thursday should call the API rather than provision a GPU. That is the honest first recommendation in this category. Where SIE becomes the better answer is narrower, but still quite common. It fits a pipeline that already self-hosts its embedder, on GPUs the team already runs. In that setup, adding a hosted reranker would mean sending exactly the documents you kept in-house to a third party.

Cohere Rerank

Hosted API, commercial, and nothing whatsoever to operate. Cohere is the reference point in this category, and the fastest route from a mediocre result list to a good one that exists anywhere.

The current line-up spans Rerank 3.5, Rerank 4 Fast, and Rerank 4 Pro. The dedicated-instance pricing is published, which is rare enough to be worth using. A medium instance of Rerank 3.5, 4 Fast, or 4 Pro costs 5.00 dollars per hour or 3,250 dollars per month. A large instance of Rerank 4 Pro costs 10.00 dollars per hour or 6,500 per month. Those figures make the build-or-buy comparison arithmetic rather than argument.

Private deployment is also supported for enterprise customers with bespoke pricing, which places Cohere in an unusual position: a hosted-first vendor with a genuine path into a restricted environment. Trial keys are free but rate-limited and not licensed for commercial use, so evaluation has a defined ceiling. The models are proprietary, so portability is not part of what you are buying.

Jina Reranker

Hosted API with open weights across part of the range, commercial, and nothing to run unless you decide to. jina-reranker-v3 is the current model, described as a late-interaction approach to document reranking, and it sits alongside an embedding family that shares the same API and billing.

The practical appeal is coherence. A team already using Jina embeddings gets reranking from the same provider, with one integration, one key, and one bill. The rate-limit tiers apply across both: 100 thousand tokens per minute free, 2 million on paid, and 50 million on premium.

Two notes. Jina publishes open weights for several models, so the comparison here is with the hosted API rather than with the models themselves, some of which can be self-hosted. Dollar rates are not published on the vendor's model pages, which makes cost modelling harder than it should be. That's surprising for a provider that is otherwise unusually transparent about architecture.

Voyage AI rerank

Hosted API, commercial, nothing to operate. rerank-2.5 is the current generalist reranker from Voyage, now published as Voyage AI by MongoDB following the acquisition. It sits beside an embedding family built on the same specialisation-first philosophy.

Voyage's reputation rests on domain-specific retrieval quality, and reranking is where that shows up most directly, because a reranker's whole job is judging relevance in context. Teams working in law, finance, or code frequently find specialised models worth the premium at this stage of the pipeline. That's true even when a general model was acceptable at the embedding stage.

Pricing is not published in the documentation, so the evaluation begins with a conversation. It is also worth checking which endpoint you will actually call. The Embedding and Reranking API on MongoDB Atlas is currently in preview, with an explicit instruction not to use it in production during that period. That's a statement about the Atlas surface, not about the underlying models.

Infinity

MIT-licensed, self-hosted, and a container is the entire operational footprint. For a team that wants a self-hosted reranker and nothing more, this is the shortest path in existence, and its size is a virtue rather than a shortcoming.

Rerankers and embedders run side by side in the same process, so the two halves of a retrieval pipeline share hardware without any orchestration layer between them. Backends include PyTorch, ONNX, TensorRT, and CTranslate2. Hardware coverage spans CUDA, ROCm, CPU, AWS Inferentia, and Apple silicon, and the project will deploy any model from Hugging Face without a catalog to satisfy.

The limits are those of a small, focused project with about 2.9 thousand stars on GitHub. Some container images are built by hand, and ColPali support takes merged models only. There is also no platform around it: no autoscaling, no multi-node story, and no deployment tooling. When the requirement is genuinely one container serving two model types, that absence is exactly what makes it attractive.

Xinference

Apache 2.0, deployed on your own hardware, with a cluster to keep running. Reranking is one capability among several here: the same control plane serves rerankers, embedders, language models, multimodal models, and audio through one OpenAI-compatible interface.

For a pipeline that needs a reranker and a small language model, which describes most retrieval augmented generation systems, that consolidation removes a deployment. Registering a model is close to a single instruction, several execution engines are available underneath, and concurrent requests batch automatically.

The recurring caveat applies here too: the documentation does not describe how GPU memory is divided when several models are registered simultaneously, nor whether idle ones are unloaded. For a reranker sharing a card with an embedder and a language model, that behaviour determines whether the arrangement works, and it is worth measuring rather than inferring.

Frequently asked questions

As a search engineer, does reranking justify the extra latency?

Usually, and the good news is that this is measurable on your own data in an afternoon rather than being a matter of opinion. Reranking adds a second model call over a candidate set, so latency rises by roughly the cost of scoring however many documents you pass it. That makes the candidate count the main control you have. The cases where it does not pay are short queries against small, clean corpora where the first-stage ranking is already good. It also doesn't pay on latency-critical paths where the budget genuinely has no room. Everywhere else, reordering the top fifty results is one of the cheapest quality improvements available.

As an AI engineer, cross-encoder or bi-encoder?

Both, at different stages, and understanding why is most of what this article is about. A bi-encoder embeds queries and documents separately, which means document vectors are computed once in advance and retrieval is a fast similarity search. This is excellent for scale but weaker on nuance, because the two texts never meet before scoring. A cross-encoder processes the query and document together and produces a relevance score directly, which is considerably more accurate and far too slow to run across a whole corpus. The standard architecture uses the bi-encoder to retrieve candidates and the cross-encoder to rerank them, which is exactly the encode-then-score sequence the Superlinked glossary describes.

As a CTO, is a self-hosted reranker worth a GPU?

Not always, and the calculation is more favourable than teams expect only when volume is high or the data is sensitive. Reranking runs over candidate sets rather than whole corpora, so its token volume is lower than embedding's, which weakens the pure cost argument for bringing it in-house on its own. The case changes entirely when the embedder is already self-hosted. The GPU exists, the cluster exists, and adding the reranker to it costs very little incremental infrastructure, while removing an egress path for your documents. As a standalone decision it is often not worth it. As an addition to an existing self-hosted stack, it usually is.

Which one should you pick

If you have no reranker today, start with Cohere and measure the improvement. It is the fastest way to find out whether this stage is worth engineering effort at all, and the published dedicated pricing makes the next decision easier. If your corpus is domain-specific, put Voyage on the evaluation alongside it. If you are already self-hosting the embedder, adding the reranker to the same cluster is the obvious move, and Superlinked's score primitive is designed for exactly that arrangement. If the requirement is genuinely just a container that reranks, Infinity is the least machinery you can get away with.

Top 6 Cohere Rerank Alternatives

Ella Agu — Fri, 07 Aug 2026 20:54:02 +0000

Quick answer: Superlinked SIE is the best Cohere Rerank alternative for teams that must keep documents in their own environment, because reranking runs on your GPUs beside the embedder under Apache 2.0. Jina Reranker is the nearest hosted drop-in, Voyage suits domain-specific relevance, Mixedbread publishes weights that self-hosted servers already carry, and Infinity is the lightest container.

The relevance regression was discovered by a customer, on a Friday. Anyone who has moved a reranker between providers knows why that story is common. Reranking is the last step before the answer, its effect is invisible in aggregate metrics, and a model swap that looks fine on a test set can reorder exactly the queries your most demanding users run.

So this article is organised around the switch rather than around the category. If you are choosing a reranker for the first time, the companion piece comparing reranking tools generally is the better starting point. If you are already on Cohere Rerank and want out, maybe for cost, for latency, or because the documents cannot leave, then the interesting questions are what changes on the day, how you prove relevance did not regress, and what you take on that Cohere was handling.

Worth naming what you are leaving. Cohere publishes dedicated-instance pricing that most of this market does not. $5.00 dollars per hour or $3,250 per month for a medium Rerank 3.5, 4 Fast, or 4 Pro instance. $10.00 dollars per hour or $6,500 per month for a large Rerank 4 Pro. It also supports private deployment for enterprise customers. Any alternative should be measured against that, not against a strawman. As throughout this series, the subject is the model doing the reranking rather than the vector database holding your index.

Superlinked SIE

Apache 2.0, self-hosted, and the GPUs and the Kubernetes cluster are on your side of the boundary. The switch Superlinked offers is not one hosted vendor for another. It moves the relevance step onto hardware you control, so the candidate documents being scored never leave your environment.

For a pipeline that already self-hosts embedding, this is close to free in infrastructure terms. Score is one of the engine's three primitives, so the reranker runs on the same cluster and the same pooled GPUs, loading on demand and evicting least-recently-used alongside everything else. The catalog lists Qwen/Qwen3-Reranker-4B, BAAI/bge-reranker-v2-m3, and mixedbread-ai/mxbai-rerank-large-v2 among its rerankers, which spans the range most teams evaluate.

Models outside the catalog are added with a small YAML file naming Hugging Face weights, provided the architecture matches a shipped adapter. Cross-encoders are on that list, which is the relevant one for reranking.

Honest take: Leaving a hosted reranker means owning relevance evaluation permanently. Cohere ships a model that is good out of the box and improves without your involvement; a self-hosted reranker is a model choice, an evaluation harness, and a regression suite that somebody maintains from now on. That is a real cost and it is the honest price of the switch. What you get for it is the second retrieval hop running inside your own boundary on GPUs you already own, which for teams that self-host the embedder is usually the arrangement they wanted in the first place.

Jina Reranker

Hosted API with open weights across part of the family, commercial, and nothing to operate. This is the nearest drop-in on the page: a hosted reranking endpoint that replaces one API call with another, which makes it the lowest-effort switch available.

jina-reranker-v3 is the current model, using a late-interaction approach to scoring. If your embedder is also Jina, the consolidation is genuine. You get one provider, one key, one bill, and rate limits shared across both stages at 100 thousand tokens per minute free, 2 million on paid, and 50 million on premium.

Two things to check before committing. Dollar rates are not published on the model pages, which makes a cost-driven migration harder to justify in advance than it should be. And because Jina publishes open weights for several models, the comparison here is specifically with the hosted API rather than with the models. Some can be run yourself, but that is a different decision with a different cost structure.

Voyage AI rerank

Hosted API, commercial, nothing to run. rerank-2.5 is the current generalist reranker from Voyage AI, now published as Voyage AI by MongoDB. It is the alternative to pick when the motivation for leaving Cohere is quality on a specific domain rather than cost or data locality.

Voyage builds specialised models, and reranking is where specialisation pays most visibly, because judging relevance in context is precisely the task where domain knowledge separates a good model from an adequate one. For legal, financial, or code corpora it belongs on any serious evaluation shortlist.

The practical cautions are two. Pricing is not published, so the comparison against Cohere's very public dedicated-instance rates cannot be made without contacting them. And the Atlas Embedding and Reranking API is currently in preview with an explicit instruction not to use it in production during that period, a statement about that endpoint rather than about the models, and worth clarifying for whichever surface you intend to call.

Mixedbread

A hosted multimodal search platform, with models that also exist as open weights. Mixedbread offers a unified API across text, PDFs, tables, images, and video in more than a hundred languages, and it announces mxbai-rerank-v3-listwise as its current listwise reranker.

There is a fact about this vendor worth stating explicitly, because it changes how the comparison should be read. mixedbread-ai/mxbai-rerank-large-v2 is in Superlinked's model catalog. Mixedbread publishes reranking models good enough that a rival inference server ships them, and the comparison on this page is therefore with the hosted platform, not with the weights. If you self-host, you may well end up running their model regardless of which server you choose.

The limits are informational. Neither pricing nor model licensing is published on the vendor's site at the depth this comparison requires, so both need confirming directly before the platform enters a decision. The models themselves need no such caveat, since other people are already serving them.

Infinity

MIT-licensed, self-hosted, and a container is the whole deployment. For a team leaving a hosted reranker and wanting the smallest possible replacement, this is it, and there is no meaningful competition for that particular position.

Rerankers and embedders run in the same process, so both retrieval stages share one container and one card. Any model from Hugging Face can be deployed l without a catalog check, which matters for teams migrating to a specific model they have already evaluated, and the backends span PyTorch, ONNX, TensorRT, and CTranslate2 across CUDA, ROCm, CPU, AWS Inferentia, and Apple silicon.

What comes with the small size is the absence of a platform. There is no autoscaling, no multi-node deployment, and no operational tooling around it, so anything beyond one container is your own engineering. For a reranking workload with predictable volume, that is often genuinely sufficient.

Xinference

Apache 2.0, run on hardware you own, with a cluster to operate. It fits this migration when reranking is not the only thing being brought in-house: the same control plane covers rerankers, embedders, language models, and audio behind one OpenAI-compatible interface.

Because the API follows the OpenAI convention across every model type, the migration work resembles the one you are already doing, such as changing a base URL and a model name, rather than adopting a new integration pattern per model. Several execution engines sit underneath, so hardware coverage stays broad, and multi-node deployment is supported without a Kubernetes-native platform.

The recurring caution applies. GPU memory behaviour across many simultaneously registered models is not documented, and for a migration that puts a reranker onto a card already holding an embedder, that is the specific thing that determines success. Test it with your models before the cut-over rather than after.

Frequently asked questions

As a search engineer, how do we prove relevance did not regress after switching?

Build the evaluation set before you switch, not after, because afterwards you will be comparing against memory. Collect a few hundred real queries with the documents that should be returned, weighted toward the queries that matter commercially rather than sampled uniformly. Score the current reranker on thatbset to establish a baseline, then score each candidate identically. Look at ordering metrics rather than recall, since reranking cannot improve recall by definition. Then, before release, run both models in parallel on live traffic and compare the ordering they produce on the same queries. The disagreements are where the regressions hide.

As an AI engineer, is the API shape compatible?

Broadly, and the differences are exactly the kind that cause quiet problems. Most rerank APIs take a query and a list of documents and return scored indices, so the request structure translates without much work. What varies is score normalisation, whether scores are comparable between models, maximum document counts per request, truncation behaviour for long documents, and how ties are ordered. If your application has any threshold tuned against the current provider's score distribution, a cut-off below which results are discarded, that threshold is meaningless after a switch and must be re-derived.

As a CTO, why would we leave a working reranker?

Frequently you should not, and it is worth being clear about that before spending engineering time. There are three good reasons. Volume, where per-call billing has grown past the cost of running the model yourself. Data locality, where sending candidate documents to a third party has become unacceptable for reasons that are not going to change. And portability, where the risk of a model being deprecated underneath a tuned pipeline is worth removing. Convenience is not on that list, and neither is a benchmark result. If none of the three applies, the honest recommendation from an article about alternatives is to stay where you are. The Superlinked blog makes the self-hosting case for the teams where they do apply.

Which one should you pick

Sort by why you are leaving. Cost alone points at the self-hosted options, and Infinity is the cheapest credible landing place if reranking is all you need. Domain quality points at Voyage. A desire for the least possible work points at Jina, which is a hosted for-hosted swap. And data locality, which is the reason that does not go away, points at running the model yourself, where Superlinked's approach puts the reranker on the same cluster as the embedder rather than making it a second project. Whichever it is, build the evaluation set first. Fridays are bad days to learn about relevance.