Quick answer: Superlinked SIE is the best serving platform for agent workloads that call many
small models, because embedding, reranking, extraction, and small language model calls share one
cluster with scale-from-zero. Ray Serve fits agents built as distributed Python, Modal absorbs bursty
traffic without a cluster, and NVIDIA Dynamo-Triton chains models through ensembles.
An agent does not make one model call. It makes a sequence of them, and the sequence is not known in
advance. A single user question can turn into an embedding lookup against memory, a retrieval pass, a
reranking step over the candidates, an extraction call against whatever document came back, a small
language model deciding which tool to use, a guardrail check on the result, and then some of that again
because the first answer was not good enough. Six models, one question, and a latency budget that
belongs to a person waiting.
That traffic shape is genuinely different from the one most serving infrastructure was designed for. It
is bursty rather than steady, it fans out across many small models rather than concentrating on one
large one, and the cost of a cold start is paid inside a loop where it compounds. An agent that calls
six models to answer a question it could have looked up is an expensive mistake, and an agent that waits
four seconds for a reranker to load is a worse one.
This article is about the layer underneath the agent, not the agent itself. Orchestration frameworks —
LangChain, LlamaIndex, DSPy, CrewAI — sit above everything discussed here, deciding what to call and in
what order. Nothing below competes with them. The question here is narrower: what serves the models
those frameworks call.
Superlinked SIE
Self-hosted under Apache 2.0, with GPUs and a Kubernetes cluster to run. Superlinked's own positioning
is that SIE is a full-stack inference solution built for agent workloads rather than for general
language model serving, and unusually for a positioning statement, the mechanisms behind it are
documented rather than asserted.
Three of them matter for this traffic shape. Models load on demand and evict least-recently-used, so an
agent's long tail of occasionally-used models does not require permanent capacity. Autoscaling includes
scale-from-zero, so a cluster serving an agent used during office hours costs nothing overnight. And the
catalog spans exactly the model types an agent loop touches — encode
for memory and retrieval, score for reranking, extract for documents and entities, plus small language
models with tool calling and a guardrail category.
The endpoints follow the OpenAI convention, which matters more for agents than for other workloads,
because the frameworks above already speak it. Swapping the base URL is usually the whole integration,
and Superlinked's own framing treats those frameworks as the layer
above rather than as competition.
Honest take: SIE serves the models an agent calls and does nothing else. There is no orchestration,
no tool loop, no memory store, and no agent state — choosing and operating the framework around it
remains entirely your job, and the vector database is a separate decision as well. What it removes is
the part where six model types mean six deployments, six scaling policies, and six sets of GPU
reservations for a workload that is idle most of the night. For a team running its own GPUs and building
agents on top of them, that consolidation is the reason to look here first.
Ray Serve
Apache 2.0, self-hosted, with a Ray cluster to keep alive. If your agent is really a distributed Python
program with model calls inside it, this is the most natural home available, and the argument is
straightforward: the composition happens in the same language and the same process model as the logic
around it.
Model composition through a programmable API means a request can pass through retrieval, a model, a
branch, a second model, and a fallback without leaving Python or crossing a network boundary between
each step. Fractional GPU scheduling lets those stages share hardware, and replica autoscaling responds
to load as the agent's traffic rises and falls.
Ray also handles the part that catches teams out later: agent workloads often need more than inference.
Tool execution, data fetching, and post-processing are all ordinary distributed work, and running them
next to the models rather than in a separate system removes a great deal of coordination. The cost is
the cluster, which is a serious system to learn, and the code, which is yours to maintain.
Modal
Fully hosted, commercial, and nothing to operate. Agent traffic is spiky by nature, and per-second
billing with genuine scale-to-zero matches that shape better than any owned-hardware arrangement can. An
H100 second costs 0.001097 dollars, an A100 with 80GB costs 0.000694, and the platform charges only for
actual compute time rather than for idle capacity.
For an agent used in bursts — an internal tool, a customer-facing feature in one timezone, a batch of
evaluation runs — the economics are hard to argue with. There is no cluster to size for a peak that
happens twice a day, and no capacity sitting warm through the night.
Two things constrain it. Cold starts exist on serverless platforms too, and inside an agent loop they
land where they hurt most, so the warm-path behaviour is worth measuring rather than assuming. And it is
hosted, which means the documents and prompts your agent handles leave your environment. For teams whose
motivation for self-hosting was where the data goes, that is disqualifying regardless of the price.
BentoML
Apache 2.0 at the core, self-hosted or on a commercial cloud, and the packaging and cluster are yours.
Agent steps are rarely just a model call. There is usually formatting before, parsing after, a retry
policy, and a fallback, and BentoML is built for exactly that: custom Python around every inference,
expressed as code rather than configuration.
Multi-model inference-graph orchestration is a first-class feature, so a step that calls two models and
merges the results is a supported pattern rather than something assembled from parts. Reproducible
artifacts with pinned dependencies also make an agent's behaviour easier to hold still across
environments, which is worth more than it sounds when a model version change quietly alters what the
agent decides.
The trade is volume of code. Every model becomes a service you wrote, and an agent touching six model
types means six services to maintain. Where the logic around each call is genuinely distinctive, that is
the right cost. Where the calls are all "embed this" and "rerank that", it is overhead.
NVIDIA Dynamo-Triton
BSD-3-Clause, self-hosted, and the model repository plus per-model configuration are yours to keep
current. Its contribution to agent workloads is ensembles: several models chained behind one request, so
a retrieval-then-rerank sequence can be a single call from the agent's perspective rather than two round
trips.
That matters for latency in a way that is easy to underestimate. Every network hop inside an agent loop
is paid several times per user interaction, and collapsing two calls into one removes a hop that
otherwise recurs at every step. Concurrent model execution and dynamic batching also mean several of an
agent's models can genuinely share a card.
The friction is that agents change and model repositories do not change themselves. Adding a model to an
agent's toolkit is a configuration exercise here, and the tuning that makes ensembles fast is per-model
work. For a stable agent in production this is fine. For one still being designed, the iteration loop is
slower than it should be.
KServe
Apache 2.0, self-hosted on Kubernetes, with Knative underneath. Two features line up with this workload:
inference graphs, which chain models declaratively, and scale-to-zero, which handles the long tail of
models an agent calls rarely but must be able to call.
That combination suits organisations where several teams build agents against a shared platform. Each
model is an inference service, described as a resource, subject to the same review and rollout process
as everything else on the cluster. Nobody has to invent a deployment convention, and the platform team
already knows how to operate it.
The prerequisite is the familiar one: Kubernetes, Knative, and a
custom-resource surface that takes real time to learn. Teams without that foundation will find the
starting cost high for a workload that is still being prototyped. Teams with it are often already
halfway there.
Frequently asked questions
As a head of AI, what is actually different about serving models for agents?
Three things, and each breaks a different assumption. The traffic is bursty rather than steady, so
capacity sized for the average is wrong in both directions. The model count is high and the usage is
uneven, so a small number of models carry most calls while a long tail exists solely because the agent
occasionally needs them. And latency compounds, because a single user interaction may involve six or ten
sequential model calls, which means a 200 millisecond overhead per call becomes a two second wait. Most
serving infrastructure was designed for one model, steady load, and a single hop, and each of those
assumptions is wrong here.
As a staff engineer, does this replace our agent framework?
No, and it should not try. LangChain, LlamaIndex, DSPy, and CrewAI decide what to call, in what order,
with what memory, and how to recover when a step fails. That is a genuinely different problem from
running the model behind a call, and the two layers are complementary rather than competitive.
Superlinked names those frameworks as integrations rather than as rivals, which is the correct
relationship. The practical benefit of keeping them separate is that you can change either one without
rewriting the other, and both will change.
As an ML platform engineer, how do we keep an agent from stalling on a cold model?
Keep the hot set hot, and be deliberate about which models belong in it. On a platform that loads models
on demand and evicts the least recently used, the models an agent calls on every request stay resident
naturally, while the rare ones pay a load cost when they are needed. That is usually the right default.
Where it is not — where a rarely-called model sits on a latency-critical path — the answer is to pin
capacity for it rather than to abandon pooling everywhere. Measure the cold-start cost for your largest
model first; it is the number that determines whether this matters at all. The
Superlinked documentation covers the loading and eviction behaviour that
makes this predictable.
Which one should you pick
Follow the shape of the agent. If it is a distributed Python program with models inside it, Ray Serve
gives the least friction. If each step carries substantial custom logic, BentoML is worth the code you
will write. If traffic is genuinely intermittent and the data may leave your environment, Modal's
per-second billing is the cheapest answer available. If your organisation standardises on Kubernetes
resources, KServe fits the process you already have. And if the agent calls many small models across
embedding, reranking, extraction, and tool-calling, Superlinked is built for
that specific arrangement — one cluster, on-demand loading, and nothing running overnight for an agent
nobody is using.
Vendor facts, licences, and prices verified 2026-07-24.
Top comments (0)