Running large language models in cloud native environments gives teams full control over hardware, networking, and data residency, but the operational cost often outweighs the benefits for many workloads. This guide walks through deploying LLMs on Kubernetes, from containerizing a model to handling production traffic, and explains when a managed inference platform like Oxlo.ai eliminates engineering toil without sacrificing flexibility.
Model Selection and Containerization
Start by selecting a model that matches your latency, quality, and license requirements. Popular open choices include Llama 3.3 70B for general tasks, Qwen 3 32B for multilingual agent workflows, and DeepSeek R1 671B for reasoning. Once selected, package the model weights and serving engine into an OCI image. Most teams use vLLM or HuggingFace Text Generation Inference (TGI) because they expose an OpenAI-compatible HTTP interface and support continuous batching.
Below is a minimal Dockerfile using vLLM. It assumes you have downloaded the weights to a local ./models directory.
FROM vllm/vllm-openai:latest
COPY ./models /data/models
ENV MODEL_NAME="meta-llama/Llama-3.3-70B-Instruct"
ENV TENSOR_PARALLEL_SIZE=4
CMD python -m vllm.entrypoints.openai.api_server \
--model /data/models/Llama-3.3-70B-Instruct \
--tensor-parallel-size 4 \
--dtype bfloat16 \
--max-model-len 8192 \
--port 8000
Build and push this image to a registry your cluster can access. Keep the image layer containing weights separate from the serving binary if you reuse the same engine across model versions.
Kubernetes and GPU Provisioning
Cloud native LLM serving requires GPU nodes. On AWS, that means EC2 p4d or p5 instances. On GCP, A3 or A2 VMs. On Azure, NC A100 v4 series. Install the NVIDIA GPU Operator and device plugin so Kubernetes can schedule GPU workloads.
Verify that your nodes expose the nvidia.com/gpu resource:
kubectl describe node <gpu-node> | grep nvidia.com/gpu
Create a dedicated namespace and set resource quotas to prevent a single model replica from monopolizing cluster capacity.
Serving Engine Configuration
Deploy the container as a Kubernetes Deployment with a Service in front. Use a StatefulSet only if you need stable network identities for distributed tensor parallelism across pods. For most single-node multi-GPU setups, a Deployment suffices.
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-70b-serving
spec:
replicas: 1
selector:
matchLabels:
app: llama-70b
template:
metadata:
labels:
app: llama-70b
spec:
containers:
- name: vllm
image: <registry>/vllm-llama-70b:latest
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: 4
memory: "192Gi"
cpu: "32"
env:
- name: TENSOR_PARALLEL_SIZE
value: "4"
---
apiVersion: v1
kind: Service
metadata:
name: llama-70b-service
spec:
selector:
app: llama-70b
ports:
- port: 8000
targetPort: 8000
Expose the service via an Ingress or a cloud load balancer. If you run multiple replicas, place a Layer-4 load balancer in front and enable session affinity where possible, because LLM inference is stateful for the duration of a request.
Autoscaling and Load Balancing
Standard Horizontal Pod Autoscaler (HPA) based on CPU is useless for GPU inference. Instead, use the Kubernetes Event-driven Autoscaling (KEDA) scaler with a custom metric such as request queue depth, time-to-first-token (TTFT), or GPU memory utilization. Configure a scale-down stabilization window of at least five minutes to avoid thrashing, because model initialization can take several minutes and causes cold starts that degrade user experience.
Cold starts are one of the biggest pain points in self-hosted LLM infrastructure. If your workload is bursty, maintaining idle GPU replicas is expensive, yet scaling from zero forces users to wait. This is where managed platforms become attractive. Oxlo.ai serves popular models with no cold starts, so traffic spikes do not trigger minute-long initialization delays.
Observability and Security
Instrument your serving engine with Prometheus metrics. vLLM exposes vllm:num_requests_running, vllm:gpu_cache_usage_perc, and vllm:time_to_first_token_seconds. Build Grafana dashboards around these signals to detect throughput saturation before it becomes an outage.
Security hardening is non-negotiable. Run containers as non-root, drop all capabilities, and apply a restrictive seccomp profile. Use Kubernetes NetworkPolicies to restrict egress from inference pods to only the model registry and telemetry endpoints. If you handle sensitive prompts, ensure data at rest is encrypted and consider running on isolated nodes.
The Managed Inference Alternative
Self-hosting is the right choice when you have strict data residency requirements, custom model weights, or existing GPU contracts. For everything else, operating a Kubernetes-based inference fleet is undifferentiated heavy lifting. You have to manage CUDA drivers, batching logic, autoscaling thresholds, and capacity planning.
Oxlo.ai provides a developer-first alternative: a fully OpenAI SDK compatible API with request-based pricing. Instead of metering tokens, Oxlo.ai charges one flat cost per API request regardless of prompt length. For long-context and agentic workloads, this can be significantly cheaper than token-based providers. You get access to 45+ open-source and proprietary models, including Llama 3.3 70B, DeepSeek R1 671B, and Qwen 3 32B, without maintaining containers or GPU nodes.
Switching is simple. Change the base URL and API key in your existing OpenAI client:
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="<your_oxlo_api_key>"
)
response = client.chat.completions.create(
model="Llama-3.3-70B-Instruct",
messages=[{"role": "user", "content": "Explain Kubernetes pod scheduling."}],
stream=True
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
The same code works for function calling, JSON mode, vision inputs, and multi-turn conversations. You keep your existing application architecture and remove the infrastructure layer entirely.
When to Self-Host and When to Outsource
Choose self-hosting if you need absolute control over the inference runtime, custom quantization schemes, or air-gapped deployments. Choose a managed platform if your priority is shipping features, controlling costs on variable-length prompts, and eliminating cold starts.
For teams building agentic systems that issue dozens of long-context requests per task, or for SaaS products with unpredictable traffic, Oxlo.ai eliminates the scaling and pricing complexity of self-hosted Kubernetes. You can review the exact request-based pricing at https://oxlo.ai/pricing and start with the free tier, which includes 60 requests per day across 16+ models and a 7-day full-access trial.
Top comments (0)