DEV Community

James LIN
James LIN

Posted on

llm-d Is a Serious Kubernetes Inference Design, Not a Drop-In Gateway

#bash #kubernetes #vllm #inference

The interesting problem llm-d addresses is not simply “how do I run an LLM container?” Kubernetes can already do that. The harder problem is keeping inference performance predictable when requests have different prompt lengths, generation sizes, GPU profiles, and cache behavior.

From a gateway engineer’s perspective, llm-d treats inference as a scheduling problem rather than a reverse-proxy problem. That is the right direction. A basic load balancer sees pods and request counts. An inference-aware system needs to understand model readiness, accelerator capacity, prompt processing, and the cost of moving requests between replicas.

Under the Hood

The project builds around Kubernetes-native components and inference-serving engines such as vLLM. The request path can use Gateway API concepts while an inference-aware scheduler selects a suitable backend instead of blindly round-robining traffic.

That separation matters. The gateway handles connectivity and policy; the scheduler handles model-specific placement decisions. It also leaves room for techniques such as prefix-cache awareness and disaggregated serving without forcing every application team to implement those decisions themselves.

The trade-off is operational complexity. This is not one binary and one Deployment. You are introducing controllers, routing resources, model-serving pods, GPU scheduling, and additional observability requirements.

A Minimal Starting Point

The exact manifests should follow the repository’s current examples, but the workflow looks like this:

git clone https://github.com/llm-d/llm-d.git
cd llm-d

# Inspect the Kubernetes examples before applying anything
find . -maxdepth 3 -type f \( -name '*.yaml' -o -name '*.yml' \) | sort
Enter fullscreen mode Exit fullscreen mode

A typical deployment needs a GPU-capable node pool, a compatible Kubernetes Gateway implementation, and an inference backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-server
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:<pinned-version>
          resources:
            limits:
              nvidia.com/gpu: "1"
Enter fullscreen mode Exit fullscreen mode

Do not treat this as production-ready configuration. Pin images, restrict service exposure, define network policies, and verify that request logs do not capture prompts or generated content.

The Production Question

llm-d’s value is architectural: it creates a path toward accelerator-aware governance instead of hiding everything behind a generic proxy. Its cost is the number of moving parts and the expertise needed to debug scheduling, GPU utilization, model loading, and cache locality.

I would evaluate it on a real workload, not star count: sustained token throughput, tail latency, failure recovery, and whether the team can operate the control plane without turning every incident into a research project.

Top comments (0)