Deploying LLM Models on Cloud Infrastructure with Auto-Scaling

Deploying large language models on cloud infrastructure with auto-scaling requires more than provisioning GPU instances. You need to orchestrate model sharding, queue-aware scaling, and request routing while keeping cold start latency low. For teams running long-context or agentic workloads, the operational overhead can quickly exceed the cost savings of self-hosting. This guide walks through a production-ready architecture, then explains where a managed inference platform like Oxlo.ai eliminates that overhead entirely.

Architecture Overview for LLM Auto-Scaling

A production LLM serving stack typically consists of four layers: the inference engine, the orchestration plane, the auto-scaling control loop, and the observability stack. The inference engine, such as vLLM or Text Generation Inference (TGI), runs inside containers on GPU-enabled nodes. The orchestration plane, usually Kubernetes, handles pod scheduling and service discovery. The auto-scaling layer must scale both the pod replica count and the underlying GPU node pool. Finally, the observability stack tracks queue depth, GPU memory utilization, and time-to-first-token so scaling decisions are based on user-facing latency, not just CPU pressure.

Unlike standard web services, LLM inference pods are not interchangeable across hardware generations. Mixing NVIDIA A100 and H100 GPUs in the same tensor-parallel group will crash or severely degrade performance. This means your auto-scaling groups should be partitioned by instance family, and your routing layer must direct traffic to the correct partition.

GPU Selection and Model Sharding Strategies

Before writing any YAML, decide how your model weights are distributed. Dense models like Llama 3.3 70B fit on a single A100 80GB with quantization, but mixture-of-experts architectures such as DeepSeek R1 671B MoE or GLM 5 require multiple GPUs with tensor parallelism. If you plan to serve state-of-the-art open-source reasoning models, provision node pools with NVLink-connected GPUs and ensure your inference server launches with the correct --tensor-parallel-size and --pipeline-parallel-size flags.

Auto-scaling works best when each replica is homogeneous. Define separate Kubernetes deployments for each GPU class. For example, one deployment for g5.xlarge GPU instances running distilled code models, and another for p4d.24xlarge instances running your primary chat model. Keep the sharding configuration immutable inside each deployment so that new pods spin up with identical topology.

Implementing Auto-Scaling with Kubernetes and KEDA

The Horizontal Pod Autoscaler (HPA) works for CPU-bound services, but LLM inference is bottlenecked by GPU memory and request queue depth. The Kubernetes Event-driven Autoscaler (KEDA) is better suited because it can scale on custom Prometheus metrics. Below is a ScaledObject that targets a vLLM deployment, scaling out when the average queue depth exceeds five requests.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-scaler
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-llama-deployment
  pollingInterval: 10
  cooldownPeriod: 300
  minReplicaCount: 1
  maxReplicaCount: 8
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      metricName: vllm_num_requests_waiting
      threshold: '5'
      query: |
        avg(vllm_num_requests_waiting{deployment="vllm-llama-deployment"})

The cooldownPeriod of 300 seconds prevents thrashing. GPU pods take minutes to schedule and load weights, so aggressive scale-down will cause cold starts on the next traffic spike. You should also configure the Kubernetes Cluster Autoscaler with a GPU-specific node group that has a high maxNodesTotal but uses spot or preemptible instances to control cloud compute costs.

Load Balancing and Request Routing

Standard round-robin HTTP load balancing fails for LLMs because request duration varies from milliseconds to minutes depending on output length. Use a least-connection or request-queue-aware routing algorithm. For multi-turn conversations, you need session affinity so that subsequent messages in the same conversation hit the same replica, unless your architecture stores context in an external cache.

If you run multiple model families in the same cluster, deploy a lightweight gateway that routes /v1/chat/completions to the appropriate backend based on the model parameter in the request body. This gateway should also enforce rate limits and translate OpenAI-compatible request shapes into the native format expected by your inference engine. Oxlo.ai provides this routing layer out of the box with full OpenAI SDK compatibility, which removes the need to maintain a custom gateway for heterogeneous model fleets.

Monitoring and Cost Controls

Auto-scaling saves money only if it scales down. Track these three metrics in your Prometheus or Datadog setup: GPU memory utilization, request queue depth, and time-to-first-token. If time-to-first-token rises while GPU utilization is low, your batch size is too small or your inference engine is not efficiently scheduling prefill and decode phases. If GPU utilization is high but queue depth is flat, you are compute-bound and need more replicas, not larger GPUs.

Set billing alerts on your cloud provider for GPU node pools. A single H100 node left running over a weekend can cost more than a month of managed inference for moderate traffic. Implement node-level auto-scaling with a scale-down delay of at least 10 minutes to account for GPU weight loading times.

When Managed Inference Becomes the Better Architecture

Self-hosted auto-scaling is the right choice for large, steady-state workloads with predictable traffic and a dedicated infrastructure team. For most product engineering teams, the complexity of node pools, GPU drivers, topology scheduling, and queue-aware scaling is a distraction from core features. This is especially true for long-context and agentic workloads, where token-based cloud providers scale costs linearly with prompt length.

Oxlo.ai offers a developer-first alternative: a fully managed inference platform with request-based pricing. You pay one flat cost per API request regardless of prompt length, which makes it significantly cheaper than token-based providers for long-context and agentic use cases. The platform hosts 45+ open-source and proprietary models, including Llama 3.3 70B, DeepSeek R1 671B MoE, Kimi K2.6, and GLM 5, with no cold starts on popular models. Because the API is fully OpenAI SDK compatible, you can replace your self-hosted gateway with a single endpoint change to https://api.oxlo.ai/v1.

If you are currently managing Kubernetes GPU clusters primarily to avoid per-token billing surprises, evaluate Oxlo.ai's pricing model. Request-based pricing can be 10-100x cheaper than token-based billing for long-context workloads, and you eliminate the engineering overhead of maintaining auto-scaling infrastructure entirely. See the pricing page for plan details.

Conclusion

Auto-scaling LLMs on cloud infrastructure is a solved problem at the container level but remains difficult at the GPU economics level. You must align model sharding, queue metrics, and node pool autoscaling to avoid both cold starts and runaway compute bills. If your traffic is volatile or your team lacks dedicated ML infrastructure engineers, the total cost of ownership for self-hosting is often higher than it appears. In those scenarios, dropping in a managed platform like Oxlo.ai gives you elastic scale, predictable request-based pricing, and immediate access to state-of-the-art models without writing another YAML manifest.