Deploying LLM Models on Kubernetes with GPU Support and Autoscaling

Running large language models in production requires more than a GPU. It demands orchestration, scaling logic, and careful resource management. Kubernetes has become the default substrate for AI infrastructure because it unifies these concerns under a single control plane. Yet self-hosting inference at scale introduces operational overhead that can quickly outpace the benefits of control. This guide walks through a production-ready deployment of LLM inference on Kubernetes with NVIDIA GPU support and autoscaling, then explains where a managed platform like Oxlo.ai fits when operational complexity grows.

Cluster prerequisites and GPU node configuration

Before scheduling pods, your cluster must expose NVIDIA GPUs through the device plugin mechanism. Install the NVIDIA GPU Operator or the standalone device plugin, then verify that nvidia.com/gpu appears as an allocatable resource.

kubectl describe node gpu-node-01 | grep nvidia.com/gpu

Pin your GPU workloads to dedicated node pools using taints and tolerations. This prevents CPU-only microservices from landing on expensive GPU instances. A typical node pool configuration uses the nvidia.com/gpu.present=true taint with a matching toleration in your deployment spec.

Storage is equally important. Model weights range from tens to hundreds of gigabytes. Use node-local NVMe caching, a ReadWriteMany PVC backed by a high-throughput filesystem, or an object-store init container that pulls weights before the serving container starts. Cold starts from storage can add minutes to pod readiness, so pre-caching on the node or using a persistent volume is essential.

Serving stack and container images

Most teams choose vLLM or Hugging Face TGI as the inference engine. vLLM offers PagedAttention for high throughput, while TGI provides broader compatibility with custom model architectures. Both run as standard containers and expose an OpenAI-compatible HTTP interface.

Here is a minimal vLLM deployment serving Llama-3.3-70B on a single A100 node. The example assumes weights are pre-cached at /models on the host.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      nodeSelector:
        node-type: gpu-a100
      tolerations:
        - key: "nvidia.com/gpu.present"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "/models/Llama-3.3-70B"
            - "--tensor-parallel-size"
            - "1"
            - "--max-model-len"
            - "8192"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "80Gi"
              cpu: "8"
          volumeMounts:
            - name: model-cache
              mountPath: /models
      volumes:
        - name: model-cache
          hostPath:
            path: /opt/cache/models
            type: Directory

The resources.limits field requests one GPU. Kubernetes will schedule this pod only on nodes with sufficient allocatable GPUs. If you run larger models such as DeepSeek R1 671B MoE or GPT-Oss 120B, you will need multi-GPU tensor parallelism and significantly more node memory.

Autoscaling with GPU and request metrics

Horizontal Pod Autoscaler (HPA) based on CPU and memory is insufficient for LLM serving. GPU utilization is not exposed to HPA by default, and inference latency is driven by queue depth, KV cache pressure, and batch size rather than CPU load. You need custom metrics.

Install the Prometheus Adapter or KEDA to scale on application-level signals. Useful metrics include:

• Average queue depth per pod
• Time-to-first-token (TTFT) p99
• GPU memory utilization via DCGM exporter
• Request rate per replica

A KEDA ScaledObject using Prometheus might look like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
spec:
  scaleTargetRef:
    name: llm-inference
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: vllm_queue_length
        threshold: "5"
        query: avg(vllm_queue_length{job="llm-inference"})

Scaling pods is only half the problem. If your cluster lacks idle GPU nodes, the Cluster Autoscaler must provision a new VM, install drivers, join the node, and pull the container image. This process routinely exceeds five minutes. During that window, requests queue, latency spikes, and user experience degrades. Oxlo.ai avoids this entirely. The platform delivers inference with no cold starts on popular models, so traffic surges do not trigger minute-long provisioning delays.

Traffic routing and load balancing

Expose the deployment with