Deploying LLM Models on Kubernetes Clusters with GPU Support: Best Practices

Running large language models in production requires more than just a GPU. Kubernetes has become the default orchestration layer for machine learning workloads, yet deploying inference servers at scale introduces real complexity around driver management, memory allocation, and autoscaling. This guide covers the core mechanics of deploying LLMs on GPU-backed Kubernetes clusters, and explains where a managed inference platform like Oxlo.ai removes the infrastructure burden without sacrificing control.

GPU Node Setup and Driver Configuration

Before scheduling model workloads, your cluster must expose NVIDIA GPUs as schedulable resources. The simplest path is the NVIDIA GPU Operator, which installs the device plugin, driver containers, container toolkit, and node feature discovery as a single Helm release.

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --wait

After installation, verify that your nodes advertise nvidia.com/gpu:

kubectl describe node <gpu-node> | grep nvidia.com/gpu

You should see allocatable GPU capacity. To prevent non-GPU workloads from consuming expensive GPU nodes, apply a taint to the node pool and add the matching toleration to your inference pods. Use node selectors or node affinity to pin LLM serving pods to the correct instance type.

Choosing a Model Serving Stack

Most production deployments use one of three engines:

• vLLM: Optimized for throughput via PagedAttention and continuous batching. Ideal for high-QPS chat endpoints.
• Text Generation Inference (TGI): HuggingFace compatible, with built-in watermarking and safety tooling.
• SGLang: Strong for structured generation and multi-modal workflows.

Your container image must match the CUDA version installed on the host. Mismatched driver or toolkit versions are the most common reason for pods stuck in CrashLoopBackOff. For multi-GPU models, pick an engine that supports tensor or pipeline parallelism natively.

Sample Deployment Manifest

The following manifest deploys a vLLM inference server with tensor parallelism across two GPUs. Adjust the model name and GPU count to fit your node shape.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama
  template:
    metadata:
      labels:
        app: vllm-llama
    spec:
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model
        - meta-llama/Llama-3.3-70B-Instruct
        - --tensor-parallel-size
        - "2"
        - --gpu-memory-utilization
        - "0.9"
        resources:
          limits:
            nvidia.com/gpu: "2"
            memory: "80Gi"
            cpu: "16"
        ports:
        - containerPort: 8000
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60

Expose the deployment through a ClusterIP or LoadBalancer service, or front it with an ingress controller that handles rate limiting and TLS termination. Set resource limits exactly equal to requests for GPU workloads to avoid oversubscription.

Autoscaling and Queue Management

Horizontal Pod Autoscaling based on CPU or memory does not work well for inference because GPU utilization is the real bottleneck. Instead, use KEDA with a Prometheus scaler or a custom metrics adapter that tracks queue depth, time-to-last-token, or request backlog.

The hardest part of autoscaling LLMs is the cold start. Pulling a 70B or 400B+ parameter model into GPU memory can take minutes, which is unacceptable for synchronous user requests. You either over-provision GPUs to absorb spikes, or you accept startup latency. Oxlo.ai removes this tradeoff entirely for supported models by offering no cold starts on popular models, letting you offload variable traffic without maintaining a warm buffer of expensive GPU nodes.

Storage Optimization for Model Loading

Downloading weights from HuggingFace on every pod restart wastes bandwidth and extends startup time. Use one of the following patterns:

• Node-local cache: Store weights on an NVMe-backed host path and mount it as a hostPath volume. This works best when your node pool is dedicated to a single model.
• Shared read-only PVC: Use a ReadWriteMany persistent volume backed by high-throughput network storage. An init container can clone the model once, and all pods mount the same volume.
• Container image layers: Bake the weights directly into the serving image. This increases image size but eliminates external dependencies at runtime.

For Mixture-of-Experts checkpoints like DeepSeek R1 671B or GLM 5, the total parameter count demands fast local storage. Slow I/O during weight loading becomes the bottleneck before inference even begins.

Observability and GPU Utilization

Deploy the NVIDIA DCGM Exporter in your cluster to stream GPU metrics into Prometheus. The most actionable gauges for LLM inference are:

• DCGM_FI_DEV_GPU_UTIL: Percentage of time the GPU compute engines are active. Sustained values below 40% often mean your batch size is too small.
• DCGM_FI_DEV_MEM_COPY_UTIL: Memory bandwidth utilization. High values with low compute utilization indicate memory-bound decoding, common with long-context prompts.
• DCGM_FI_DEV_FB_FREE: Free frame buffer memory. If this approaches zero, vLLM will begin evicting blocks and throughput will collapse.

Correlate these hardware metrics with application-level signals such as time-to-first-token and inter-token latency. If your GPUs are underutilized but latency is high, look at input padding inefficiency or suboptimal scheduling logic before adding more nodes.

When Managed Inference Makes Sense

Self-hosting gives you absolute control over weights, custom fine-tunes, and compliance boundaries. It also forces your team to manage CUDA upgrades, OS security patches, capacity planning, and 3 AM pages for stuck GPU nodes. If infrastructure maintenance is consuming engineering cycles that could ship product features, a managed platform is the pragmatic alternative.

Oxlo.ai is a developer-first AI inference platform with request-based pricing: one flat cost per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads. The platform hosts 45+ open-source and proprietary models across seven categories, including Llama 3.3 70B, DeepSeek R1 671B MoE, Kimi K2.6, Qwen 3 32B, and Oxlo.ai Coder Fast, all through a fully OpenAI SDK compatible API.

A practical hybrid architecture keeps sensitive or fine-tuned models on your own Kubernetes cluster while routing general chat, reasoning, coding, and image generation to Oxlo.ai. This minimizes idle GPU spend, eliminates autoscaling complexity, and removes cold start risk for bursty traffic. You can evaluate the flat request model against your current provisioned GPU costs at https://oxlo.ai/pricing.

Conclusion

Deploying LLMs on Kubernetes with GPU support is a solved problem, but it is not a free one. The operational surface area spans drivers, storage, autoscaling, and observability. Start with a clear serving stack, enforce strict node isolation, and measure time-to-first-token under load before you commit to a fixed fleet size. Where self-hosting becomes a maintenance tax, Oxlo.ai provides a flat-cost, request-based alternative with broad model coverage, OpenAI-compatible endpoints, and no cold starts on popular models.