Running large language models in production requires more than a GPU instance. You need autoscaling that reacts to queue depth, orchestration that handles node failures, and an API layer that standardizes access across model families. Most teams start with raw cloud VMs, then discover that serving LLMs at scale introduces problems that standard HTTP load balancers do not solve.
The Challenge of Self-Hosted LLM Infrastructure
Self-hosting begins with a base image containing CUDA drivers, PyTorch, and a serving framework like vLLM or TGI. From there, you must configure tensor parallelism across multiple GPUs, set up a Kubernetes cluster with GPU-aware scheduling, and manage model weights in a high-throughput storage layer. Each step adds latency to your deployment pipeline and operational surface area to your maintenance burden.
Autoscaling is particularly difficult because GPU utilization does not map cleanly to request volume. A single long-context request can saturate memory while leaving compute underutilized. Standard CPU-based Horizontal Pod Autoscaler metrics miss this nuance, which leads to either cold starts for users or wasted GPU hours from over-provisioning.
Autoscaling Strategies for GPU Workloads
Effective autoscaling for LLMs requires custom metrics. Instead of CPU percentage, monitor request queue depth per model replica, GPU memory utilization, time-to-first-token latency, and inter-token latency for streaming responses. You can expose these through Prometheus and configure the Kubernetes Metrics Server or KEDA to scale replicas based on a composite threshold.
A common pattern is to maintain a small warm pool of one or two replicas, then scale out when queue depth exceeds a configurable limit. The example below shows a KEDA ScaledObject that targets a vLLM deployment:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: llm-autoscaler
spec:
scaleTargetRef:
name: vllm-deployment
minReplicaCount: 1
maxReplicaCount: 8
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus:9090
metricName: vllm_queue_depth
threshold: "5"
query: sum(vllm_queue_depth{model="llama-3.3-70b"})
Note that scaling up a GPU node pool takes minutes, not seconds. For traffic spikes, consider a two-tier architecture: a fixed warm pool on dedicated GPUs and a burst tier on serverless GPU instances where cold starts are acceptable. If cold starts are unacceptable, you must over-provision.
GPU Support and Resource Management
Cloud providers offer A100, H100, and L4 instances, but availability varies by region. When deploying across zones, ensure your cluster autoscaler is aware of GPU quotas and spot instance eviction rates.
Resource limits in Kubernetes must be precise. Requesting fractional GPUs is possible with NVIDIA time-slicing or MIG on A100 and H100, but these technologies add scheduling complexity. For multi-node inference, you need high-bandwidth inter-GPU networking. On AWS, this means EFA-equipped instances. On GCP, it means A3 VMs with GPUDirect RDMA.
Storage is another bottleneck. A 70B parameter model at FP16 requires roughly 140 GB of weights. Loading this from standard persistent volumes into GPU memory can take tens of minutes. Use node-local NVMe caches or container image layers preloaded onto the instance to reduce startup time.
API Design and SDK Compatibility
Once the serving layer is stable, you need a consistent API. The OpenAI SDK has become the de facto standard. Mapping your self-hosted endpoints to /v1/chat/completions, /v1/embeddings, and /v1/images/generations simplifies client migration and tool integration.
If you prefer not to manage this translation layer, managed inference platforms provide drop-in compatibility. Oxlo.ai exposes a fully OpenAI SDK-compatible API at https://api.oxlo.ai/v1. You can switch from a self-hosted stack to Oxlo.ai by changing the base URL and API key.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Explain autoscaling for LLMs."}],
stream=True
)
Oxlo.ai supports streaming responses, function calling, JSON mode, and vision inputs, so the migration does not require client-side refactoring.
Load Balancing and Request Routing
A self-hosted cluster needs intelligent routing. Long-context requests should not land on replicas that are already memory-constrained. Session affinity can hurt throughput if a single user sends many large prompts.
Implement prefix-aware routing if you run multiple model variants. Cache popular attention key-value pairs across requests with shared prefixes. vLLM offers automatic prefix caching, but you must ensure your load balancer sends prefix-sharing requests to the same replica.
For multi-modal workloads, route image inputs to vision-capable replicas and text-only queries to standard LLM replicas. This prevents vision token preprocessing from blocking text generation pipelines.
Observability and Cost Control
GPU cloud infrastructure is expensive. You need granular visibility into cost per request, cost per output token, GPU idle time between autoscaling events, and model cache hit rates. Tag all Kubernetes resources with model names and versions, export GPU metrics to a centralized dashboard, and set alerts for queue depths that indicate undersizing or for low utilization that indicates overspending.
If you find that token-based billing creates unpredictable costs for long-context workloads, consider platforms with alternative pricing models. Oxlo.ai uses request-based pricing with one flat cost per API request regardless of prompt length. For agentic workflows that send large context windows repeatedly, this model removes the pricing uncertainty associated with token counters.
When to Choose Managed Inference Over Self-Hosting
Self-hosting makes sense when you have strict data residency requirements, custom fine-tuned weights, or existing GPU hardware. It does not make sense when your team spends more time on CUDA drivers than on product features.
Managed platforms like Oxlo.ai abstract away node pools, GPU drivers, and autoscaling logic. Oxlo.ai offers 45+ open-source and proprietary models across seven categories, including chat and reasoning, code, vision, image generation, audio, embeddings, and object detection. There are no cold starts on popular models, which eliminates the over-provisioning penalty required by self-hosted clusters.
For teams comparing providers, Oxlo.ai differentiates itself from token-based competitors such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale through its flat per-request pricing. Because cost does not scale with input length, long-context and agentic workloads become significantly more predictable. See https://oxlo.ai/pricing for current plan details.
You can evaluate the platform through the free tier, which includes 60 requests per day across 16+ models, or explore the Pro and Premium plans for higher daily volumes and priority queue access. Enterprise plans offer dedicated GPUs and custom pricing.
Conclusion
Deploying LLMs on cloud platforms with autoscaling and GPU support is a solvable engineering problem, but it demands specialized knowledge in Kubernetes scheduling, GPU topology, and request routing. Most production teams eventually move to a managed inference layer to reduce operational overhead.
Whether you self-host or outsource, standardize on the OpenAI API contract and instrument your stack for cost per request. If you choose a managed provider, Oxlo.ai offers a compatible, request-priced alternative that removes the infrastructure burden while keeping long-context costs flat.
Top comments (0)