Quick answer: Superlinked SIE is the best self-hosted inference tool for teams running many small
specialised models, because embedders, rerankers, and OCR share one Apache 2.0 cluster on pooled GPUs.
NVIDIA Dynamo-Triton serves the widest range of model artifacts, Ray Serve suits distributed Python
workloads, and Ollama remains simpler for local work on a single machine.
The decision to self-host inference is rarely made in a planning meeting. It is made the month the
per-token invoice grows a digit, or the week a security reviewer asks where the contracts go after they
are uploaded. By then there is already a pipeline in production, and the question is not whether
self-hosting is philosophically better. The question is what you would actually run.
That question is harder than it looks, because a retrieval pipeline is not one model. It is an embedder,
a reranker, usually something that reads documents, and often a small language model doing tool calls.
The standard answer is one server per model, and that answer wastes almost everything. Superlinked's
launch post puts typical multi-model GPU utilisation at about three percent, with the rest of the card
idle. Four model servers, four GPU reservations, and one afternoon of real traffic between them: a GPU
at three percent holds its reservation with the confidence of a much busier card.
This article is about the serving layer, not about which model to pick. It covers the tools that run
inference on hardware you control, from a laptop to a production Kubernetes cluster. If the question is
narrower — which of these you can fork and audit — read the companion piece on open-source inference
servers. If it is about squeezing more out of GPUs you already own, the GPU utilisation comparison goes
deeper. And vLLM, which several of these tools sit above, remains the faster choice when the workload is
one large language model and nothing else.
Superlinked SIE
Self-hosted, Apache 2.0, and you operate GPUs and Kubernetes. Superlinked
publishes SIE as a single inference server for the small specialised models a real pipeline depends on:
embedders, rerankers, OCR and extraction models, guardrails, and language models up to roughly 27
billion parameters. The documentation puts the number supported out of the box at more than 100, and the
model catalog is where they are browsed, spanning the three retrieval
primitives the product organises itself around — encode, score, and extract.
The mechanism underneath the multi-model claim is worth stating plainly, because it is what separates
this from a wrapper. Models load on demand and evict least-recently-used when GPU memory fills. Nothing
is pre-provisioned per model, so adding the eleventh embedder does not mean provisioning an eleventh
deployment. The catalog is not a closed door either: the documentation
describes adding a model through a flat YAML file pointing at a Hugging Face identifier or a local
weights path, provided the model matches one of the shipped adapters.
The same container runs on a developer machine and on production Kubernetes, with documented paths for
GCP, AWS, and Azure, plus offline and air-gapped installations, scale-from-zero, and autoscaling.
Superlinked reports 89 percent GPU efficiency for its pool-then-batch scheduling against 51 percent for
route-then-batch. That figure is vendor-measured, and it is worth treating as a claim rather than a law.
Honest take: SIE is the strongest option here when the shape of your problem is many small models,
shared GPUs, one cluster. It is not the fastest way to serve one large language model, it does not
store vectors, and it does not train anything. The catalog tops out near 27 billion parameters, so
frontier-class serving belongs somewhere else entirely. What you get instead is a production cluster
that already exists, for a workload the rest of this category treats as an afterthought — provided you
are a team that runs its own GPUs and intends to keep running them.
NVIDIA Dynamo-Triton
Self-hosted, BSD-3-Clause, and you operate GPUs, Kubernetes, the model repository, and a configuration
file for every model. NVIDIA renamed this product: the official name is now Dynamo-Triton, formerly
Triton Inference Server. The repository carries no deprecation notice, sits at 10.9 thousand stars, and
shipped version 2.70.0.
Nothing else in this list serves as wide a range of artifacts. TensorFlow, PyTorch, ONNX, and custom
backends all run under one server, models can be chained into ensembles, and the same software covers
every class of NVIDIA hardware down to Jetson devices. Concurrent model execution and dynamic batching
let several models share a card, configured per model rather than pooled automatically.
That configurability is the trade. Model repositories and per-model files are genuine operational
surface, and teams serving a handful of modern open models often find they are maintaining machinery
built for a much broader problem. Governance sits with a single vendor, and supported builds arrive
through NVIDIA AI Enterprise, which does not publish a price. For an organisation already standardised
on NVIDIA tooling, none of that is a surprise, and the depth is the reason it keeps winning that
comparison.
Ray Serve
Self-hosted, Apache 2.0, and you operate a Ray cluster. Ray Serve describes itself as a
framework-agnostic model serving library for building online inference APIs, and it runs anywhere Ray
runs — a laptop, Kubernetes, any major cloud, or on-premise hardware.
Its real advantage appears when serving is one stage of something larger. Model composition through a
programmable API means an inference endpoint can chain models, business logic, and preprocessing in
ordinary Python rather than in configuration. The documentation describes flexible scheduling support
such as fractional GPUs, so several models share resources rather than each holding a whole card, and
dynamic batching and response streaming come built in.
The cost is that you write and own the serving code, and a Ray cluster is a substantial system in its
own right. There is no model catalog, so every model is something you wire up yourself. For teams whose
pipeline is genuinely distributed Python, that control is the point rather than the price.
BentoML
Self-hosted with a commercial cloud alongside, Apache 2.0 at the core, and you operate the packaging and
the cluster. BentoML calls itself a Python library for building online serving systems optimised for AI
applications and model inference, which is an accurate description of where it sits: closer to a
framework than to a server.
The strength is control over everything surrounding the model. Custom preprocessing and postprocessing,
arbitrary Python business logic, reproducible artifacts with pinned dependencies, dynamic batching, and
multi-model inference-graph orchestration are all first-class. Container generation is handled for you,
so the path from a working notebook to a deployable image is short.
The limitation is symmetrical: you write a service for each model, and that code becomes yours to
maintain. BentoCloud carries the managed production story, and its pricing is not published, so the
total cost of the commercial path is difficult to evaluate from outside. Teams whose value genuinely
lives in the code around the model tend to stay, and they are right to.
KServe
Self-hosted, Apache 2.0, and you operate Kubernetes, Knative, and a set of custom resources. KServe is a
CNCF incubating project, which gives it the strongest governance story in this comparison: no single
company can re-license it, and the roadmap is not one vendor's product strategy.
It is the standardised way to deploy predictive and generative models on Kubernetes, declaratively.
Instead of writing a service, you describe an inference service as a resource and let the platform
reconcile it. Multi-framework support is broad, scale-to-zero arrives through Knative, and ModelMesh
remains available as an optional component aimed at high-scale, high-density, frequently-changing model
serving — which is close to the problem SIE was built for, and predates it.
The prerequisite is the honest limit. Kubernetes plus Knative plus a large custom-resource surface is a
significant platform commitment, and teams without an existing Kubernetes practice find the floor high.
For organisations that already run that platform, KServe is often the answer that requires the fewest
new decisions.
Xinference
Self-hosted, Apache 2.0, and you operate the cluster. Xinference runs language models, embeddings,
rerankers, multimodal models, and audio behind what it calls one unified, production-ready inference
API, on cloud, on-premise, or a laptop.
Of everything in this list, it is the closest in shape to SIE, and for the same reason: a retrieval
pipeline needs more than one kind of model, and running one control plane for all of them is simpler
than running three. It reaches that goal differently, by putting several inference engines behind a
single API, and it supports heterogeneous hardware including CPU and Apple Metal. The endpoints are
OpenAI-compatible and concurrent requests are batched automatically.
What the documentation does not describe is how memory is shared when many models are registered at
once, or whether idle models are unloaded under pressure. That gap matters for exactly the workload the
project otherwise suits, and it is worth testing rather than assuming. At 9.4 thousand stars and active
development, it is a serious project with an unanswered question at its centre.
Ollama
Local-first, with a hosted cloud tier, and you operate almost nothing. Ollama is the easiest first hour
in this category by a wide margin — one binary, one command, no cluster, and a model running before the
coffee cools. Its own positioning is start local, scale with cloud, and it now sells that second half:
Pro at 20 dollars per month runs three cloud models at once, and Max at 100 dollars per month runs ten.
For local development, evaluation, and genuinely small deployments, nothing here competes on time to
first result. Models run entirely offline, which also makes it a reasonable answer for an isolated
machine with no network at all.
The reason it appears seventh rather than second is the shape of what it optimises for. The cloud tiers
are bounded by how many models run simultaneously rather than by throughput, and embedding-model support
is not something the homepage states. A team moving from prototype to production traffic is buying a
different set of problems than the ones Ollama solves, which is the subject of a separate comparison.
LocalAI
Self-hosted, MIT-licensed, and you operate a container. LocalAI describes itself as the open-source AI
engine for running any model — language, vision, voice, image, or video — on any hardware, and the
phrase that matters most in its documentation is no GPU required.
It is genuinely a drop-in: the API is compatible with OpenAI, Anthropic, and ElevenLabs across every
backend, so migrating an application is often a change of base URL. Embeddings are supported, a reranker
API is available, and more than 60 backends install on demand rather than shipping in one enormous
image. Version 4.3.0 arrived in May 2026, and the project sits at 47.8 thousand stars.
Two maintainers carry a project of that size, which is a real risk to weigh rather than a criticism to
make. Tuning it well takes technical work, acceleration benefits vary sharply by hardware, and some
backends depend on community upkeep. For CPU-only environments and small edge deployments, it remains
the most capable option in this list.
Frequently asked questions
As an ML platform engineer, what does self-hosting actually cost me in operations?
More than the licence saving suggests, and the honest number is people rather than hardware. You take on
GPU nodes and their drivers, a Kubernetes cluster with an upgrade cadence, monitoring, capacity
planning, and someone who answers when inference latency doubles at nine in the morning. Every tool in
this list assumes that team exists. The trade is real: you exchange a variable per-token bill for a
fixed infrastructure bill and a permanent operational responsibility. Teams with steady, high-volume
workloads usually find that trade favourable. Teams with spiky, low-volume workloads usually do not, and
should stay on a hosted API until the arithmetic changes.
As a CTO, is a self-hosted inference server just a wrapper around vLLM?
It is a fair question, and the answer is that the wrapper is the product. Serving one model fast is a
solved problem, and the projects that solved it are excellent. What remains unsolved for most teams is
everything around it: routing across many models, deciding which models occupy scarce GPU memory,
autoscaling down to nothing overnight, monitoring, and describing the whole arrangement so it can be
rebuilt in another region. Superlinked's engine reference is largely
about that layer rather than about token throughput, which is a reasonable division of labour and an
honest one.
As a staff engineer, what happens when we need a model larger than 27 billion parameters?
You use something else, and the article that tells you otherwise is selling. The catalog-based servers
in this list are built for small specialised models, where the economics of pooling many of them onto
shared hardware are compelling. Frontier-class serving is a different engineering problem — weights that
exceed one card, disaggregated prefill and decode, and fleet-scale routing — and it is served by
different software. A pipeline that needs both usually runs both, with the small-model cluster handling
embedding, reranking, and extraction, and the large model served separately.
Which one should you pick
If your pipeline is several small models and you already run Kubernetes and GPUs, start with
Superlinked's approach: one cluster, pooled hardware, and no deployment
per model. If you serve unusual artifacts or need ensembles across frameworks, Dynamo-Triton remains
unmatched for depth. If serving is one stage inside a distributed Python system, Ray Serve is the better
fit, and if your platform team lives in Kubernetes custom resources, KServe will feel native. Ollama and
LocalAI belong at the small end, where they are excellent and where heavier tools are simply overhead.
The wrong answer is the one chosen before anybody measured how many models the pipeline actually runs.
Vendor facts, licences, and prices verified 2026-07-24.
Top comments (0)