Running large language models in production usually means wrestling with CUDA drivers, multi-gigabyte model weights, and container orchestration before you ever send a prompt. For engineering teams that need control, containerized platforms like Kubernetes remain the default substrate, but the operational tax is steep and often underestimated. This article walks through a practical containerized deployment, then explains why many teams are shifting inference to managed APIs such as Oxlo.ai to eliminate that tax entirely.
The Containerized LLM Stack
A production LLM container stack typically includes the NVIDIA Container Toolkit, a GPU-aware runtime, a model serving engine, and an orchestration layer. The serving engine, often vLLM or Text Generation Inference, handles continuous batching, PagedAttention, and OpenAI-compatible HTTP endpoints. Below is a minimal but functional example using vLLM inside Docker.
A Minimal vLLM Deployment
# Dockerfile
FROM vllm/vllm-openai:latest
# Pre-download weights at build time to avoid cold-fetch on startup
ENV MODEL_NAME="meta-llama/Llama-3.3-70B-Instruct"
RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('${MODEL_NAME}')"
ENTRYPOINT ["python", "-m", "vllm.entrypoints.openai.api_server"]
CMD ["--model", "meta-llama/Llama-3.3-70B-Instruct", "--tensor-parallel-size", "2", "--dtype", "bfloat16"]
Build and run with GPU exposure:
docker build -t llm-serve .
docker run --gpus all -p 8000:8000 llm-serve
This gives you a local OpenAI-compatible API, but it assumes the host has drivers, CUDA libraries, and enough VRAM. For a 70B parameter model, that means multiple A100 or H100 GPUs and high-bandwidth interconnects.
Kubernetes Complexity
Moving from Docker to Kubernetes adds scheduling, autoscaling, and service discovery, but also complexity. You need the NVIDIA GPU Operator, device plugins, and likely a custom node pool for GPU instances. A typical deployment involves a StatefulSet or Deployment with resource limits for nvidia.com/gpu, a ClusterIP service, and a Horizontal Pod Autoscaler that reacts on GPU utilization or request queue depth.
The manifest below is simplified but shows the core requirements:
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-deployment
spec:
replicas: 1
selector:
matchLabels:
app: llm
template:
metadata:
labels:
app: llm
spec:
containers:
- name: vllm
image: llm-serve:latest
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: 2
Even with this in place, you still face model weight updates, security patching, GPU bin-packing, and cold starts when scaling from zero. Each new model version requires a new container image, another multi-gigabyte layer, and a rolling restart that can take minutes.
The Hidden Costs of Self-Hosting
Self-hosting gives you full control, but the cost model is deceptive. You pay for reserved GPU capacity whether the model is idle or processing a thousand tokens. For long-context workloads, where prompts can span tens of thousands of tokens, that reserved capacity becomes expensive fast. Token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale shift this to a pay-per-token model, but costs still scale linearly with input length. If you are building agentic workflows that iterate over large contexts repeatedly, both reserved capacity and token-based billing can strain budgets.
Managed Inference Without the Overhead
Oxlo.ai removes the container layer entirely. It is a developer-first AI inference platform that offers request-based pricing: one flat cost per API request regardless of prompt length. Unlike token-based providers, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads. In fact, request-based pricing can be 10-100x cheaper than token-based for long-context workloads. There are no cold starts on popular models, and the API is fully OpenAI SDK compatible.
The platform hosts 45+ open-source and proprietary models across 7 categories, including general-purpose LLMs such as Llama 3.3 70B and Qwen 3 32B, deep reasoning models such as DeepSeek R1 671B MoE and Kimi K2.6, code models such as Oxlo.ai Coder Fast, vision models, image generation, audio, embeddings, and object detection. Endpoints cover chat/completions, embeddings, images/generations, audio/transcriptions, and audio/speech, with support for streaming, function calling, JSON mode, and multi-turn conversations.
Calling Oxlo.ai From Your Existing Stack
Because Oxlo.ai is a drop-in replacement, you can route existing containerized services to it by changing a base URL and API key. No Dockerfile, no GPU nodes, no pod scheduling.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
response = client.chat.completions.create(
model="Llama-3.3-70B",
messages=[{"role": "user", "content": "Explain request-based pricing for LLMs."}],
stream=False
)
print(response.choices[0].message.content)
This same pattern works in Python, Node.js, or cURL. For teams already running containers for application logic, Oxlo.ai becomes an external inference backend that eliminates the need to host model servers inside the cluster.
When to Choose What
Self-hosting still makes sense for air-gapped environments, strict data residency requirements that cannot be met by a managed API, or ultra-low latency scenarios where every millisecond of network hop matters. For everything else, the maintenance burden rarely justifies the flexibility.
Oxlo.ai offers a Free plan at $0 per month with 60 requests per day across 16+ free models and a 7-day full-access trial. The Pro plan is $80 per month for 1,000 requests per day across all models, while Premium at $350 per month adds 5,000 requests per day and a priority queue. Enterprise plans provide custom pricing, unlimited requests, dedicated GPUs, and a guaranteed 30% savings versus your current provider. See https://oxlo.ai/pricing for current details.
Conclusion
Containerized LLM deployments are powerful, but they saddle teams with infrastructure work that has nothing to do with model performance. If your goal is to ship features, not manage GPU drivers and pod lifecycles, replacing your in-cluster model server with Oxlo.ai gives you the same open-source models without the operational overhead. The request-based pricing model is especially effective for long-context and agentic applications, turning unpredictable token costs
Top comments (0)