Deploying LLM Models on Kubernetes

#aiinfrastructure #oxlo #ai

Running large language models in production requires more than a GPU. Kubernetes is the default orchestration layer for teams that need control over inference latency, data residency, and model versioning. Deploying LLMs on a cluster introduces challenges that standard microservices rarely face: multi-GPU scheduling, massive container images, context-length-aware autoscaling, and expensive idle nodes.

Architecture and Node Provisioning

Start by isolating GPU workloads. Create dedicated node pools with taints so that only inference pods schedule on expensive GPU instances. Install the NVIDIA Device Plugin and GPU Feature Discovery to expose hardware topology to the scheduler. For models that require tensor parallelism across multiple GPUs, you need topology-aware scheduling or the Kubernetes Topology Aware Routing beta. Choose instance types with NVLink or high-bandwidth interconnects to minimize communication overhead between GPUs.

Model Serving Runtimes

Hand-rolling an inference server is rarely worth the effort. Production teams typically choose vLLM for its PagedAttention throughput, TensorRT-LLM for maximum NVIDIA performance, or HuggingFace Text Generation Inference for simpler integration. Most runtimes now expose an OpenAI-compatible HTTP server on a well-known port.

The following Deployment runs vLLM with tensor parallelism across two GPUs. It mounts a PersistentVolumeClaim that holds cached model weights so the pod does not re-download on restart.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama
  template:
    metadata:
      labels:
        app: vllm-llama
    spec:
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "meta-llama/Llama-3.3-70B-Instruct"
        - "--tensor-parallel-size"
        - "2"
        - "--port"
        - "8000"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 2
        volumeMounts:
        - name: model-cache
          mountPath: /models
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-pvc

Storage and Model Caching

Container images for inference runtimes are large, but the real problem is model weights. A 70B parameter model at BF16 can consume over 140 GB. Pulling that from an object store on every pod start adds minutes of latency. Use one of two patterns. First, a ReadWriteMany PVC backed by fast network storage that is mounted by every replica. Second, an init container that copies weights from S3 to an emptyDir volume backed by local NVMe. For multi-node clusters, consider a cluster-wide caching layer such as Dragonfly or a read-only NFS export to avoid redundant downloads.

Autoscaling GPU Workloads

Standard Horizontal Pod Autoscaler based on CPU is useless for LLM inference. You need custom metrics such as request queue depth, time-to-first-token, or GPU memory utilization. KEDA supports scaling Deployments based on Prometheus queries or message broker lag. Be careful with scale-to-zero. While it saves money, cold starts on GPU nodes can exceed thirty seconds while the runtime loads weights into VRAM. If your latency budget is tight, you must keep a minimum number of warm replicas, which means paying for idle GPUs.

Observability and Cost Control

Export hardware metrics with NVIDIA DCGM Exporter. Track GPU utilization, memory bandwidth, and power draw at the pod level. Combine this with Kubecost or OpenCost to attribute infrastructure spend to specific teams or models. Service-level metrics matter just as much. Measure time-to-first-token and inter-token latency, because high throughput does not guarantee a responsive user experience. Alert on OOMKilled pods immediately; they usually signal that your context length exceeded available VRAM.

Operational Tradeoffs and Managed Alternatives

Self-hosting gives you complete control over the stack, but the operational tax is significant. You become responsible for driver compatibility, CUDA upgrades, runtime bugs, and capacity planning. For teams that need a broad model catalog or run agentic workloads with long contexts, maintaining a dedicated GPU fleet for every model variant is economically inefficient.

Oxlo.ai offers a pragmatic alternative. It is a developer-first inference platform with flat per-request pricing, so cost does not scale with prompt length. That makes Oxlo.ai significantly cheaper than token-based providers for long-context and agentic workloads. The platform hosts 45+ open-source and proprietary models, including Llama 3.3 70B, DeepSeek R1 671B MoE, and Qwen 3 32B, and it is fully compatible with the OpenAI SDK. There are no cold starts on popular models, and the API base URL is https://api.oxlo.ai/v1.

A common hybrid pattern is to keep your Kubernetes cluster for fine-tuned or sensitive data workloads, and to route general-purpose chat, coding, vision, or overflow traffic to Oxlo.ai. Because Oxlo.ai is a drop-in replacement, you can switch endpoints with a single configuration change in your existing OpenAI client. For detailed pricing, see https://oxlo.ai/pricing.

Conclusion

Kubernetes remains a powerful platform for LLM inference at scale. With the right node pools, serving runtimes, and autoscaling policies, you can deliver low-latency predictions inside your own infrastructure. Just remember that owning the stack is not free. Evaluate hybrid strategies that combine self-hosted Kubernetes with managed APIs like Oxlo.ai to balance control, cost, and model variety.