Deploying LLM Models on Cloud with Auto-Scaling

Running large language models in production requires more than provisioning GPU instances. Auto-scaling inference is uniquely difficult because LLMs are memory-bound, exhibit highly variable execution times, and carry expensive startup penalties when loading weights into GPU memory. This article walks through practical patterns for deploying self-hosted LLMs with auto-scaling on cloud Kubernetes clusters, then shows where managed inference platforms like Oxlo.ai can remove the entire operational burden without sacrificing control.

Architecture Patterns for LLM Auto-Scaling

Traditional auto-scaling strategies break down for LLMs. Web server metrics like CPU utilization are the wrong signals for memory-bound transformer inference. A useful control plane must account for GPU memory fragmentation, model weight load times, request queue depth, and KV-cache statefulness.

Most teams settle on one of two architectures:

• Cluster-proportional autoscaling: Maintain a base pool of GPU nodes and scale horizontally on custom inference metrics.
• Queue-driven event scaling: Use an external queue or metrics API to scale replicas only when backlog exceeds a threshold.

Both require a custom metrics pipeline because standard CPU or memory HPA does not map to LLM throughput.

Kubernetes and Custom Metrics

The Kubernetes Horizontal Pod Autoscaler can scale GPU workloads, but only if you feed it custom metrics. GPU utilization alone is misleading: a model can be at 100% GPU compute while throughput collapses due to long input sequences. A better signal is pending request count or time-to-first-token latency.

First, deploy your inference engine with a GPU resource limit. The example below uses a standard vLLM container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm
  template:
    metadata:
      labels:
        app: llm
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: "1"
        args:
        - --model
        - meta-llama/Llama-3.3-70B-Instruct
        - --tensor-parallel-size
        - "1"
        ports:
        - containerPort: 8000

Next, configure an HPA that references a custom metric exposed through the Prometheus Adapter:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-inference
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: pending_requests
      target:
        type: AverageValue
        averageValue: "5"

The Prometheus Adapter must be configured to map pending_requests to a query such as sum(vllm:num_requests_waiting{model="llm-inference"}). Without this translation layer, the HPA has no visibility into the inference queue.

Queue-Aware Scaling with KEDA

For stricter cost control, use Kubernetes Event-Driven Autoscaling (KEDA) to scale from zero or to react to queue depth faster than the HPA loop allows. KEDA can poll an external metrics endpoint and trigger replica changes within seconds.

A ScaledObject targeting a metrics API might look like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-keda
spec:
  scaleTargetRef:
    name: llm-inference
  pollingInterval: 5
  cooldownPeriod: 60
  minReplicaCount: 0
  maxReplicaCount: 12
  triggers:
  - type: metrics-api
    metadata:
      url: "http://prometheus.monitoring:9090/api/v1/query?query=sum(vllm:num_requests_waiting)"
      valueLocation: "data.result.0.value.1"
      targetValue: "3"

Scaling to zero saves idle GPU cost, but introduces a cold start penalty when the model weights reload. For user-facing applications, that latency is often unacceptable.

Load Balancing and Session Affinity

Even with the right replica count, routing matters. Continuous batching and prefix caching mean that sending consecutive requests from the same session to different pods wastes memory and recomputes attention. Configure your ingress or service mesh with session affinity based on a conversation ID or user token. If your inference engine supports chunked prefill or prefix caching, affinity becomes a throughput requirement, not just an optimization.

The Hidden Cost of Self-Hosted Auto-Scaling

Owning the full stack gives you control, but it also creates a long list of operational tax: node pool bin-packing, CUDA driver compatibility, model weight storage on fast disk, and real-time rebalancing across GPU types. You pay for GPU uptime even when traffic is idle, and scaling to zero introduces the cold start problem you were trying to avoid.

This is where Oxlo.ai becomes a relevant alternative. Oxlo.ai is a developer-first inference platform with flat per-request pricing. Unlike token-based providers, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads. The platform offers 45+ open-source and proprietary models, is fully OpenAI SDK compatible, and serves popular models with no cold starts. For teams that have built auto-scaling clusters, Oxlo.ai can absorb overflow traffic. For teams still designing their architecture, it can remove the need to run a GPU control plane entirely.

You can compare plans on the Oxlo.ai pricing page.

A Hybrid Approach: Oxlo.ai as a Drop-In Fallback

Because Oxlo.ai exposes an OpenAI-compatible API at https://api.oxlo.ai/v1, you can use it as a fallback or even a primary provider without rewriting client code. The Python SDK initialization is identical.

The snippet below tries a self-hosted endpoint first, then falls back to Oxlo.ai if the local cluster is saturated:

import os
import openai

primary = openai.OpenAI(
    base_url="http://internal-llm-gateway.cluster.local/v1",
    api_key="sk-internal"
)

fallback = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def complete(messages):
    try:
        return primary.chat.completions.create(
            model="Llama-3.3-70B",
            messages=messages,
            max_tokens=1024,
            stream=False
        )
    except Exception:
        return fallback.chat.completions.create(
            model="llama-3.3-70b",
            messages=messages,
            max_tokens=1024,
            stream=False
        )

Oxlo.ai supports the same features you would expect from a self-hosted stack: streaming responses, function calling, JSON mode, vision input, and multi-turn conversations. That means your fallback path does not require feature flags or response format changes.

Monitoring and SLOs

Whether you self-host, use Oxlo.ai, or run a hybrid, instrument for these signals:

• Time to First Token (TTFT): Measures queue and prefill latency.
• Time Between Tokens (TBT): Captures decode-phase throughput.
• Queue depth: The backlog waiting for an available replica.
• GPU memory utilization: Indicates how close you are to an out-of-memory crash.
• Cost per request: Normalize total infrastructure spend by successful completions.

If your self-hosted cost per request exceeds flat per-request pricing for your traffic pattern, shifting workload to Oxlo.ai is a straightforward way to cut spend without redesigning your application.

Conclusion

Auto-scaling LLMs on cloud Kubernetes is possible, but it demands custom metrics pipelines, careful queue management, and constant tuning of GPU node pools. For many engineering teams, the operational overhead outweighs the theoretical cost savings of self-hosting.

Oxlo.ai offers request-based pricing that can be 10-100x cheaper than token-based alternatives for long-context workloads, with no cold starts on popular models and full OpenAI SDK compatibility. Whether you need a resilient fallback for peak traffic or a complete replacement for your inference layer, Oxlo.ai is a strong, relevant option worth evaluating.