A single model server can look healthy while users wait longer with every burst of traffic. Add replicas behind a generic load balancer and a subtle problem appears: requests that could reuse a model's cached prompt may land on a different replica, which must process that context again. Background agent loops can also compete with interactive requests for the same serving capacity.
That is the problem llm-d 1.0 is designed to address. Released on October 20, 2026, llm-d adds a Kubernetes-native coordination layer around inference servers such as vLLM. Its focus is not replacing vLLM. It is coordinating a fleet of model servers with cache-aware routing, traffic control, and production operations.
This guide explains when plain vLLM is enough, what llm-d adds, and how to follow the project's quickstart to deploy a small test configuration. The commands and configuration below are based on the llm-d project's published guide; no independent deployment or benchmark is reported here.
First, separate the model server from the fleet manager
vLLM serves model requests. It provides an inference engine and an API server for supported models. If one instance serves your workload and the surrounding platform already handles availability and scaling, adding a separate orchestration layer may not help.
llm-d coordinates model servers across Kubernetes. The project's vLLM integration guide describes it as a framework that coordinates a fleet of vLLM instances. Its features include prefix-aware routing, distributed KV-cache management, prefill/decode disaggregation, and wide expert parallelism. The llm-d project is built around the Kubernetes Gateway API Inference Extension and operates at the cluster-serving layer.
The short comparison:
- Plain vLLM: a simpler choice for one server, a small deployment, or a workload where generic routing is sufficient.
- vLLM behind llm-d: worth evaluating when several replicas share traffic, prompts repeat, tenants need different service priorities, or Kubernetes is already the operational home for model serving.
- Neither choice guarantees lower cost or latency. Model, hardware, concurrency, prompt reuse, and configuration determine the result. Benchmark with representative requests.
What changed in llm-d 1.0
The release announcement presents 1.0 as an operational milestone, with three capabilities especially relevant to platform teams.
Cache-aware routing. Instead of treating every replica as interchangeable, llm-d can route a request toward a replica that has a reusable prefix in its KV cache. That matters when requests share long system prompts, document context, or agent history. Reuse is an opportunity, not a promise: the benefit depends on cache retention, request similarity, and whether the selected replica has capacity.
Multi-tenant flow control. llm-d 1.0 adds priority- and fairness-aware flow control. The project describes separate queues by flow and priority, fair dispatch among tenants at a priority level, and load shedding when demand exceeds available capacity. The operational goal is to prevent a noisy background workload from overwhelming latency-sensitive traffic. Operators still need to define sensible priorities, quotas, and service objectives.
Safer operations. The maintainers describe router high availability and rolling updates, including coordinated management for disaggregated prefill and decode deployments. This matters because a serving system is not production-ready just because it returns tokens once. Upgrades, restarts, and partial failures are part of normal operation.
These capabilities add moving parts. A feature-rich router is not automatically better than a direct vLLM service. Use it when the traffic pattern or operational requirement justifies the layer.
A practical first deployment
The official quickstart is an end-to-end test of an optimized baseline in standalone mode. It uses Helm for the llm-d router and Kubernetes manifests for the model server. Its current default deploys eight replicas of Qwen/Qwen3-32B on NVIDIA GPUs, so this is not a laptop experiment: plan for substantial compatible GPU-backed Kubernetes capacity and the associated cost. The project also lists alternative configurations for other hardware, with their own model and capacity requirements.
The quickstart also downloads model weights, so startup can take several minutes. Check the current project guide before running commands because chart versions, model defaults, and hardware-specific examples can change.
1. Check prerequisites and choose a test cluster
You need a Kubernetes cluster, kubectl, Helm, and enough compatible accelerator capacity for the selected model server. The guide includes other model-server configurations for AMD, Intel, TPU, and CPU, but some alternatives use a different model. Confirm the exact model and hardware pairing in the repository before applying a configuration.
Clone the repository and load its environment definitions:
export branch="main"
git clone https://github.com/llm-d/llm-d.git
cd llm-d
git checkout "$branch"
export REPO_ROOT="$(realpath "$(git rev-parse --show-toplevel)")"
source "$REPO_ROOT/guides/env.sh"
export GUIDE_NAME="quickstart"
export NAMESPACE="llm-d-quickstart"
For a repeatable evaluation, record the commit you checked out, model ID, accelerator type, replica count, and relevant chart versions. A tutorial's main branch can change after publication, so pin and record the exact revision for a controlled test.
2. Install the inference API resources and create a namespace
The quickstart installs the Gateway API Inference Extension custom resource definitions. It derives the manifest path from the repository's environment script:
kubectl apply -f \
"https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml"
kubectl create namespace "$NAMESPACE" \
--dry-run=client -o yaml | kubectl apply -f -
This is a cluster-level change for the API resources and a namespace-level change for the test deployment. Review the manifests and your cluster's change controls before applying them to a shared or production cluster.
3. Configure access to model weights
The quickstart asks for a Hugging Face token in a Kubernetes secret named llm-d-hf-token, with the key HF_TOKEN. Keep the token out of shell history and logs where possible. Use your organization's secret-management practice for anything beyond a temporary test.
kubectl create secret generic llm-d-hf-token \
--from-literal="HF_TOKEN=${HF_TOKEN}" \
--namespace "$NAMESPACE" \
--dry-run=client -o yaml | kubectl apply -f -
Set HF_TOKEN in your environment through an approved secret-injection method first. Do not paste a real token into a shared terminal transcript or commit it to a values file.
4. Deploy the router and model servers
The standalone router chart includes its proxy. The guide supplies its base and optimized-baseline values files, then applies the GPU vLLM model-server configuration:
helm install "$GUIDE_NAME" \
"$ROUTER_STANDALONE_CHART" \
-f guides/recipes/router/base.values.yaml \
-f guides/optimized-baseline/router/optimized-baseline.values.yaml \
-n "$NAMESPACE" \
--version "$ROUTER_CHART_VERSION"
kubectl apply -n "$NAMESPACE" \
-k guides/optimized-baseline/modelserver/gpu/vllm/base/
Wait for the model-server pods to become ready before sending a request:
bash "$REPO_ROOT/guides/scripts/wait-for-pods-ready.sh" \
-l llm-d.ai/guide=optimized-baseline \
-n "$NAMESPACE"
A pod being scheduled is not the same as a model being ready. Check pod events and logs if the wait takes longer than expected. Weight downloads, accelerator allocation, image pulls, and model initialization can all affect startup time.
5. Send a request through the router
The quickstart retrieves the router service's cluster IP and sends a completion request from a temporary pod inside the cluster:
export IP="$(kubectl get service "${GUIDE_NAME}-epp" \
-n "$NAMESPACE" \
-o jsonpath='{.spec.clusterIP}')"
kubectl run curl-debug --rm -it \
--image=cfmanteiga/alpine-bash-curl-jq \
--namespace="$NAMESPACE" \
--env="IP=$IP" \
-- /bin/bash
Inside that temporary shell:
curl -X POST "http://${IP}/v1/completions" \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen/Qwen3-32B",
"prompt": "Explain what a Kubernetes readiness probe checks."
}'
This confirms the request path works. It does not establish that llm-d is faster or cheaper than direct vLLM. The quickstart is a deployment and verification path, not a benchmark result.
Make the comparison fair
A useful evaluation has two configurations: your current baseline and the candidate deployment. Keep the model, accelerator type, precision, prompt set, output length, concurrency, and warm-up policy consistent. Measure both cold-start and steady-state behavior separately.
Track at least:
- Time to first token and end-to-end latency, including high percentiles
- Input and output tokens per second under realistic concurrency
- GPU or accelerator utilization and memory use
- Cache hit behavior for repeated prefixes, if that is the feature being tested
- Error rate, queue depth, and behavior during traffic bursts
- Infrastructure cost for the full serving path, including router and supporting services
- Upgrade and recovery behavior, not just a successful steady-state run
Use a mix of repeated-context interactive requests and unrelated requests. If every test prompt is identical, the cache may look unusually favorable. If no prompts share context, a prefix-cache optimization has little opportunity to help. Include both patterns if both occur in production.
The llm-d project points to Prism and its benchmark recipes for examining published configurations and comparing runs. Treat vendor or project benchmarks as a starting point. Reproduce the configuration and traffic pattern that resemble your own service before making a deployment decision.
When should you stay with plain vLLM?
Stay with a simpler deployment if one replica meets the service objective, traffic is predictable, prompts rarely share context, and your existing load balancer and autoscaler handle the service adequately. A separate coordination layer increases the number of components to configure, observe, secure, and upgrade.
Evaluate llm-d if you operate a fleet and can identify a measurable limitation: cache locality is poor, interactive traffic is hurt by batch or agent bursts, model pools need priority control, or you need coordinated distributed inference capabilities. Start with the optimized baseline and add one feature at a time. That helps answer whether a specific capability improved your service rather than attributing every change to the platform as a whole.
A short decision checklist
Before a pilot, answer these questions:
- Traffic: Do requests repeat long prefixes, arrive in bursts, or mix interactive and background work?
- Scale: Do you need multiple model-server replicas or pools?
- Platform fit: Is Kubernetes already where your team runs and operates inference?
- Capacity: Is the required accelerator capacity available for a representative test?
- Measurement: Can you compare latency, throughput, cache behavior, and full infrastructure cost fairly?
- Operations: Who will own router health, model-server upgrades, alerting, and incident response?
- Rollback: Can you switch traffic back to the existing vLLM path if the pilot misses its objectives?
The key is to define a pass/fail criterion before deployment. For example, require a target improvement in p95 time-to-first-token under a real request mix, without exceeding a defined cost or error-rate budget. Pick thresholds that match your service; there is no universal number that makes llm-d worthwhile.
The takeaway
llm-d 1.0 makes the most sense as a fleet coordination layer for teams serving open models on Kubernetes. It adds routing, cache, flow-control, and operational capabilities around inference engines such as vLLM. It is not a drop-in speed switch, and the project's reported performance results are tied to particular hardware and workloads.
For a single model server, plain vLLM may remain the better engineering choice. For a multi-replica fleet with shared context, mixed priorities, and real operational requirements, llm-d is now a concrete stack to evaluate. Start with the official quickstart, measure your own workload, and only keep the extra layer if the results justify its complexity.
I write about practical AI infrastructure and developer tools. Subscribe for future implementation guides and evidence-based comparisons.
Have you operated a multi-replica vLLM deployment? What would make you add a routing layer: cache reuse, latency under bursts, or simpler operations?
Top comments (0)