Deploying LLM Models on Kubernetes Clusters

Running large language models on Kubernetes gives you control over data residency, hardware, and request routing. For teams with strict compliance requirements or existing GPU investments, self-hosting is often a necessity. That said, operating GPUs at scale introduces real complexity: driver compatibility, model caching, autoscaling logic, and API compatibility all become your responsibility. If you need to ship today without building an infrastructure team, a managed platform like Oxlo.ai provides fully OpenAI-compatible inference with request-based pricing and no cold starts. This guide walks through a production-ready Kubernetes deployment for teams that must own the stack, while highlighting where Oxlo.ai can eliminate that overhead entirely.

Architecture Overview

A typical LLM serving stack on Kubernetes includes four layers:

• GPU-enabled worker nodes with the NVIDIA GPU Operator.
• A model inference engine such as vLLM, TGI, or llama.cpp, containerized and deployed as a StatefulSet or Deployment.
• Shared model storage via PVC (ReadWriteMany) or a node-local cache backed by NVMe.
• An API gateway or ingress layer that handles routing, load balancing, and authentication.

For production, treat the inference engine as a stateful workload. Models are multi-gigabyte artifacts, and pulling them on every pod restart creates unacceptable latency. Use a PersistentVolumeClaim backed by a high-throughput storage class, or preload images with the model weights baked into the container layer.

Prerequisites

• A Kubernetes cluster (1.28+) with NVIDIA GPUs (A100, H100, or L40S recommended).
• NVIDIA GPU Operator installed.
• Helm 3.x and kubectl configured.
• Sufficient node memory: plan for at least 1.5x the model size in RAM per replica.

Setting Up GPU Support

Install the NVIDIA GPU Operator if it is not already present.

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=true

Verify that your nodes expose the nvidia.com/gpu resource.

kubectl describe node <gpu-node> | grep nvidia.com/gpu

Label GPU nodes for workload targeting.

kubectl label nodes <gpu-node> node-type=gpu-l4

Deploying the Inference Engine

The following example deploys a vLLM-based Llama 3.3 70B service. Adjust the model name and GPU count for your target workload.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      nodeSelector:
        node-type: gpu-l4
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model
        - meta-llama/Llama-3.3-70B-Instruct
        - --tensor-parallel-size
        - "2"
        - --max-model-len
        - "8192"
        resources:
          limits:
            nvidia.com/gpu: "2"
            memory: "96Gi"
            cpu: "16"
        volumeMounts:
        - name: model-cache
          mountPath: /models
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: llm-inference-svc
  namespace: inference
spec:
  selector:
    app: llm-inference
  ports:
  - port: 8000
    targetPort: 8000
  type: ClusterIP

Apply the manifest.

kubectl apply -f llm-deployment.yaml

Exposing the API

Use an ingress controller with TLS termination to expose the inference service externally. If you standardize on the OpenAI API schema, your client code remains portable across self-hosted and managed providers.

import openai

client = openai.OpenAI(
    base_url="https://llm.yourdomain.com/v1",
    api_key="your-internal-api-key"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Explain Kubernetes rolling updates."}]
)

If maintaining this endpoint tier feels like overhead, Oxlo.ai exposes an OpenAI-compatible API at https://api.oxlo.ai/v1 with request-based pricing. You can switch the base_url and API key without changing your client logic, which makes Oxlo.ai a viable drop-in replacement for clusters that hit capacity or budget constraints.

Scaling Strategies

Horizontal autoscaling for LLMs is non-trivial. GPU metrics are not exposed by default, so install the NVIDIA Data Center GPU Manager (DCGM) exporter and configure the Prometheus Adapter for the Horizontal Pod Autoscaler.

Target metrics:

• GPU utilization above 75 percent for sustained windows.
• Request queue depth reported by the inference engine.
• Custom metrics for time-to-first-token (TTFT).

A conservative approach is to run a dedicated node pool with cluster-autoscaler and scale replicas based on custom metrics. Avoid scaling on CPU or memory alone, as these do not correlate with inference saturation.

Observability

Instrument your inference engine with Prometheus metrics. Key signals include:

• End-to-end request latency (p50, p95, p99).
• Time to first token and inter-token latency.
• GPU memory utilization and temperature.
• Batch size distribution.

Grafana dashboards built on these metrics will surface whether your tensor-parallelism configuration is correct or if you are leaving GPU memory unused.

Managed vs. Self-Hosted

Self-hosting gives you complete control, but it comes with a fixed cost: node maintenance, driver upgrades, model artifact management, and scaling logic all require engineering time. Token-based cloud providers can also introduce unpredictable costs for long-context prompts, which complicates budgeting for agentic workloads.

Oxlo.ai eliminates infrastructure management while preserving the developer experience. It offers request-based pricing, so a single API call costs the same regardless of prompt length, which makes it significantly cheaper for long-context and agentic use cases than token-based alternatives. With 45+ models across seven categories, fully OpenAI SDK-compatible endpoints, and no cold starts on popular models, Oxlo.ai lets you move from cluster administration back to product development. Compare plans at the Oxlo.ai pricing page.

Conclusion

Deploying LLMs on Kubernetes is a powerful pattern for organizations with existing infrastructure and compliance requirements. By using GPU operators, persistent model caches, and OpenAI-compatible serving engines, you can build a robust inference platform inside your own environment. Just remember that every layer you own is a layer you must maintain. When the operational burden outweighs the flexibility, Oxlo.ai provides a direct migration path with flat per-request pricing and full API compatibility.