Deploying LLM Models on a Kubernetes Cluster with GPU Support: A Tutorial

#engineering #oxlo #ai

In this tutorial we will build a Kubernetes-native LLM agent and deploy it to a GPU-enabled cluster. Instead of self-hosting multi-billion parameter models on expensive GPU nodes, our service calls Oxlo.ai for inference. This gives you elastic LLM capacity without managing terabyte-scale model weights or vLLM serving stacks.

What you'll need

A running Kubernetes cluster (v1.28+) with kubectl configured
NVIDIA GPU nodes and the Device Plugin installed (optional for this agent, but keeps the cluster ML-ready)
Docker and a container registry you can push to
Python 3.10+
pip install openai fastapi uvicorn pydantic
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Define the agent system prompt

Our agent acts as a site-reliability assistant that reads Kubernetes pod descriptions and proposes fixes. This prompt is injected into every chat completion sent to Oxlo.ai.

SYSTEM_PROMPT = """You are K8sGPT, a site-reliability assistant. You analyze Kubernetes pod descriptions and log snippets. Respond with a concise diagnosis and a kubectl command or YAML patch to fix the issue. If you need more data, ask a specific follow-up question. Keep responses under 120 words."""

Step 2: Write the FastAPI service

The application exposes a single POST endpoint. It uses the OpenAI SDK as a drop-in client for Oxlo.ai, routing all inference to Llama 3.3 70B for fast, general-purpose reasoning.

import os
from fastapi import FastAPI
from pydantic import BaseModel
from openai import OpenAI

SYSTEM_PROMPT = """You are K8sGPT, a site-reliability assistant. You analyze Kubernetes pod descriptions and log snippets. Respond with a concise diagnosis and a kubectl command or YAML patch to fix the issue. If you need more data, ask a specific follow-up question. Keep responses under 120 words."""

app = FastAPI()

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

class AnalyzeRequest(BaseModel):
    namespace: str
    pod_description: str

@app.post("/analyze")
async def analyze_pod(req: AnalyzeRequest):
    user_content = f"Namespace: {req.namespace}\n\nPod description:\n{req.pod_description}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_content},
        ],
    )

    return {
        "model": response.model,
        "diagnosis": response.choices[0].message.content
    }

Step 3: Containerize the application

We package the service with a slim Python image. Because Oxlo.ai handles inference, the container needs no CUDA runtime or GPU drivers, keeping the image small and the build fast.

FROM python:3.11-slim

WORKDIR /app

RUN pip install --no-cache-dir openai fastapi uvicorn pydantic

COPY main.py .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Step 4: Configure Kubernetes manifests

We deploy to CPU nodes and keep GPU nodes free for training or batch jobs. Oxlo.ai eliminates the need to reserve GPUs for LLM inference, which is usually the most expensive part of a self-hosted stack.

apiVersion: v1
kind: Secret
metadata:
  name: oxlo.ai-secret
type: Opaque
stringData:
  api-key: YOUR_OXLO_API_KEY
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8sgpt-agent
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8sgpt-agent
  template:
    metadata:
      labels:
        app: k8sgpt-agent
    spec:
      nodeSelector:
        node-type: cpu
      containers:
        - name: agent
          image: your-registry/k8sgpt-agent:v1
          ports:
            - containerPort: 8000
          env:
            - name: OXLO_API_KEY
              valueFrom:
                secretKeyRef:
                  name: oxlo.ai-secret
                  key: api-key
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: k8sgpt-service
spec:
  selector:
    app: k8sgpt-agent
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

Step 5: Deploy to the cluster

Apply the manifests and verify the rollout. The pods should start quickly because the container is lightweight and has no large model files to load.

kubectl apply -f secret.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

kubectl rollout status deployment/k8sgpt-agent
kubectl get pods -l app=k8sgpt-agent

Run it

Port-forward the service and send a pod description to test the agent against Oxlo.ai.

kubectl port-forward svc/k8sgpt-service 8080:80

curl -X POST http://localhost:8080/analyze \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "production",
    "pod_description": "Pod k8sgpt-agent-7d9f4b8c5-x2v9p is in CrashLoopBackOff. Last exit code 137. Memory limit 256Mi. No OOMKilled event, but usage graph shows spike to 262Mi before kill."
  }'

Example output:

{
  "model": "llama-3.3-70b",
  "diagnosis": "Exit code 137 indicates SIGKILL, likely from the kernel OOM killer or a memory limit breach. Your memory limit (256Mi) is too close to actual usage. Increase the limit to 512Mi and add a liveness probe to catch crashes early.\n\nkubectl set resources deployment k8sgpt-agent --limits=memory=512Mi --requests=memory=256Mi -n production"
}

Wrap-up

Add a HorizontalPodAutoscaler based on CPU or custom metrics to scale the agent during peak usage. You can also switch the Oxlo.ai model to DeepSeek R1 671B for deeper reasoning by changing a single string in the client call. For teams running long-context or agentic workloads, Oxlo.ai request-based pricing removes the cost scaling tied to prompt length. See https://oxlo.ai/pricing for plan details.

Another concrete next step is to wire the agent to the Kubernetes Events API so it streams real-time pod failures into the Oxlo.ai chat context, turning this service into an autonomous SRE assistant.

DEV Community