Quick answer: Superlinked SIE is the best multi-model serving platform for teams running many small models on shared GPUs, because models load on demand and evict least-recently-used instead of each holding a reservation. KServe is the Kubernetes-native standard for high-density serving, NVIDIA Dynamo-Triton offers finer per-model control, and Ray Serve suits models wrapped in Python logic.
The eleventh model is where this becomes a real problem. The first three were fine. Each got a deployment, each got a GPU or a slice of one, and the arrangement looked tidy on the architecture
diagram. Then the search team wanted a sparse embedder alongside the dense one, somebody added a reranker, the document pipeline needed an OCR model, an evaluation harness wanted two variants of each, and the tidy diagram became four model servers, four GPU reservations, and one afternoon of actual
traffic distributed between them.
Superlinked's launch post puts typical utilization in that arrangement at about three percent. Whether your own number is three or thirty, the mechanism is the same: a model server holds GPU memory for the model it was given, whether or not requests are arriving, because it has no way to know that another
model needs the space more.
This article is about the platforms that solve that specific problem, and every entry is judged on one question — what happens when the number of models exceeds what the GPUs can hold at once. It is a narrower question than general self-hosting, which the head article in this series covers, and a different
one from cluster-wide GPU scheduling across teams and training jobs, which the GPU utilization comparison addresses. It is also not a question about vLLM, which is excellent at serving one large model quickly and is simply not what this page is about.
Superlinked SIE
Apache 2.0 and self-hosted; the GPUs and the Kubernetes cluster are yours. The behaviour that puts Superlinked first here is specific and checkable: models lazy-load on demand and evict least-recently-used when GPU memory fills. Nothing is pre-provisioned. There is no per-model
deployment to create, no reservation to size, and no configuration file describing how much memory each model may hold.
The practical consequence is that model count stops being an infrastructure decision. Adding an evaluation variant, a second reranker, or a language-specific embedder is a request to a running cluster rather than a change to a deployment manifest. More than 100 models are supported out of the box according to the documentation, spread across embedding, reranking, extraction, generation, and guardrails and listed in the catalog, and models outside it can be added through a small YAML file pointing at a Hugging Face identifier or local weights, so long as they match one of the shipped adapters.
Superlinked reports 89 percent GPU efficiency for its pool-then-batch scheduling, against 51 percent for route-then-batch. That number is vendor-measured and belongs in the "worth reproducing" column rather than the "settled fact" column, but the mechanism behind it is documented rather than asserted.
Honest take: eviction has a price, and pretending otherwise would be the easiest way to lose this argument. A model that has been evicted and is then requested again pays its load time on that call. If every model must stay warm behind a hard latency budget, dedicated deployments still win, and no amount of pooling changes the physics. What SIE offers is the right default for the common case — many models, uneven traffic, one pool of GPUs — for a team that runs its own hardware and would rather not size a reservation per model.
KServe
Self-hosted on Kubernetes, Apache 2.0, and you operate Knative and the custom resources. KServe deserves more credit in this comparison than it usually gets, because the high-density multi-model problem is one it addressed before the current generation of tools existed.
ModelMesh remains available as an optional component, documented for high-scale, high-density, and frequently-changing model serving. That description is close to the workload this article is about, and the project reached it from the direction of Kubernetes-native standardization rather than from a model
catalog. As a CNCF incubating project, it also carries governance that no other entry here matches.
Two things are worth verifying rather than assuming. The repository does not state how ModelMesh behaves with GPU-backed models specifically, or how it handles the current generation of embedding and language models, and this article will not guess on its behalf. What is certain is that a team already running
Kubernetes and Knative has a credible multi-model path without adopting anything new, and that path should be measured before it is dismissed.
NVIDIA Dynamo-Triton
Self-hosted, BSD-3-Clause, and you operate the model repository along with everything else. Triton has served several models from one process for longer than most of this category has existed, through concurrent model execution and dynamic batching, and it does so with more control than anything else here.
That control is exercised per model. Instance groups, batching windows, and memory behavior are configured for each entry in the model repository, which means the tuning ceiling is high and the tuning floor is a directory of configuration files. For a team serving a stable set of models with known traffic shapes and a genuine need to squeeze the hardware, that is an advantage.
For a team whose model list changes weekly, it is friction. Every new model is a configuration exercise rather than a request, and the mental model is a repository you curate rather than a pool that rearranges itself. The choice between those two is less about capability than about how often the answer to "which models are we serving" changes.
Ray Serve
You run it yourself under Apache 2.0, with a Ray cluster to maintain. Ray Serve approaches multi-model serving from
the direction of general distributed computing, and its documentation is explicit about the goal: flexible scheduling support such as fractional GPUs, so resources are shared and many models are served at low cost.
Fractional allocation is a different mechanism from pooling. Rather than models competing for one shared pool that reorganizes under pressure, each replica declares the fraction of a GPU it needs and the scheduler packs them accordingly. It is predictable, it is explicit, and it works well when you know
what each model requires.
Where Ray Serve pulls ahead of everything else is when the models are not the whole story. If each inference involves preprocessing, a call to a second model, some business logic, and a fallback, then composing that in Python beside the serving layer is far more natural than expressing it in configuration. The cost is the cluster and the code, both of which are yours.
Xinference
Apache 2.0, running on hardware you own, with the cluster on your side of the line. Xinference registers and launches many models from one control plane, covering language models, embeddings, rerankers, multimodal models, and audio, which makes it the other project in this list built around the assumption that a pipeline needs several kinds of models at once.
Launching a model is a single instruction rather than a deployment; requests are batched automatically, and the API follows the OpenAI convention. Underneath, several execution engines are available, so the project inherits the hardware coverage of whichever one suits a given model, including CPU-only
environments.
The unanswered question sits exactly where this article is looking. The documentation does not describe how GPU memory is divided when many models are registered at once, nor whether idle models are unloaded when a new one needs space. For most uses that gap is academic. For the workload on this page, it is the whole question, and it is worth an afternoon of testing before it becomes an assumption in a design document.
BentoML
Self-hosted or on a commercial cloud, Apache 2.0 at the core, and you operate the packaging and the cluster. BentoML supports several models per service through composition, and its multi-model inference-graph orchestration is a genuine feature rather than a side effect.
The distinction is that composition is something you write. Where the pooling approach asks the platform to decide which models occupy memory, BentoML asks you to describe how models relate, then executes that description faithfully. For pipelines with real branching — different models for different document
types, a cheap model with an expensive fallback — that explicitness is worth having.
It is a weaker fit for the specific problem of too many models for the hardware, because the allocation question lands back on the deployment rather than being absorbed by the platform. The
managed cloud carries more of that burden, and its pricing is not published, which makes the total picture hard to evaluate without a conversation.
Frequently asked questions
As an ML platform engineer, how many models can share one GPU?
There is no honest single number, and any article offering one is guessing. The answer is a function of parameter count, numeric precision, sequence length, batch size, and concurrency, and it changes when any of those move. The useful version of the calculation is to take the resident memory of each model at the precision you will actually serve it, add the working memory your peak batch requires, and compare that against the card. Then decide whether every model must be resident simultaneously, or whether some can load on demand. That second question is the one this whole category exists to answer, and it is usually the one nobody asked.
As an infrastructure lead, what is the latency cost of loading a model on demand?
Real, and worth measuring rather than estimating, because it depends on model size and where the weights are stored. The mitigation is to keep the hot set hot: models receiving steady traffic stay resident, and eviction reaches the ones that are not being used. That works well when traffic is uneven across
models, which describes most retrieval pipelines, and poorly when every model must answer within a fixed budget at any moment. If your service level agreement does not tolerate a cold start on any model, at any hour, you want dedicated capacity and should say so early.
As a head of AI, do we need this if we serve three models?
Probably not, and the money is better spent elsewhere. Three models with predictable traffic fit comfortably into three deployments, and the operational simplicity of that arrangement is worth more than the utilization you would recover. The case for a multi-model platform gets strong when the model count is genuinely variable — when evaluation runs spin up variants, when new languages or document types each want their own specialist, or when the list changes faster than the deployment process. The
Superlinked glossary is a reasonable place to get the vocabulary straight before making that call.
Which one should you pick
Start from how often your model list changes. If it is stable and you want maximum control over each entry, Dynamo-Triton rewards the configuration effort. If your team already runs Kubernetes and Knative, measure KServe before adopting anything new. If each inference is really a small program, Ray Serve is the natural home. And if the list changes constantly, if traffic is uneven, and if the GPUs are yours to fill, Superlinked's serving model is built for precisely that shape —one pool, on-demand loading, and no reservation to size per model. Whatever you choose, count your models first. The number is usually higher than the diagram suggests.
Vendor facts, licences, and prices verified 2026-07-24.
Top comments (0)