Deploying LLM Models on Cloud with Auto-Scaling: A Technical Guide

Running large language models in production at scale means accepting a hard truth: a single GPU instance is rarely enough. Traffic patterns fluctuate, prompt sizes vary by orders of magnitude, and model replicas are expensive. Auto-scaling is the standard answer, yet doing it well for LLMs requires more than a simple CPU threshold. You need GPU-aware metrics, careful orchestration, and serving backends that can actually exploit added capacity. This guide walks through the architecture, tooling, and policy decisions required to auto-scale open-source LLMs on cloud infrastructure, and where managed alternatives fit.

Architecture Overview

A production LLM deployment separates the control plane from the data plane. The control plane manages model weights, configuration, and scaling policies. The data plane handles request ingress, queueing, and token generation.

At minimum, you need:

• A container orchestrator with GPU support. Kubernetes is the default.
• A GPU node pool with taints and tolerations so only model workloads schedule there.
• Object storage or a model cache for weights. Pulling a 70B parameter model from S3 on every pod start is too slow.
• An ingress layer with timeout and payload-size tuning.

If you lack the operational bandwidth to manage GPU clusters, managed inference platforms remove this stack entirely. Oxlo.ai, for example, provides fully managed endpoints for over 45 models across seven categories, including Llama 3.3 70B, DeepSeek R1 671B MoE, and Qwen 3 32B. It exposes a standard OpenAI-compatible API at https://api.oxlo.ai/v1, so you can route traffic there without maintaining a single GPU node pool.

Metrics That Matter

CPU and memory are poor signals for LLM scaling. You need metrics that reflect the actual user experience and GPU saturation.

Key signals include:

• GPU utilization: High utilization suggests saturation, but near-zero can hide queue bottlenecks.
• Request queue depth: The number of pending requests waiting for a slot in the batching engine.
• Time to first token (TTFT): High TTFT means users are waiting too long for generation to start.
• Time per output token (TPOT): Measures throughput once generation begins.
• Inter-token latency: Critical for streaming workloads.

Export these from your model server or sidecar. vLLM and TGI both expose Prometheus metrics. Feed them into the Kubernetes Prometheus Adapter so the Horizontal Pod Autoscaler (HPA) can react to custom metrics like vllm:num_requests_waiting.

Orchestrating GPU Pools

Standard cluster autoscalers treat all nodes equally. GPU instances require special handling because they are costly, limited by region availability, and take minutes to initialize.

We recommend Karpenter over the default Cluster Autoscaler for GPU workloads. Karpenter can directly provision node classes without waiting for managed node groups, and it consolidates workloads to reduce stranded capacity.

Example NodePool for GPU inference:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.xlarge", "g5.2xlarge", "p4d.24xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  limits:
    cpu: 1000
    memory: 4000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

The real enemy here is the cold start. Even with Karpenter, downloading container layers and model weights can keep users waiting for minutes. Oxlo.ai removes this variable entirely. Popular models are pre-warmed, so there are no cold starts when traffic spikes.

Model Serving Backends

Your auto-scaling logic is only as good as the software running inside the pods. The backend must support continuous batching and efficient memory management, otherwise adding replicas merely duplicates inefficiency.

vLLM with PagedAttention is a solid default for throughput. TensorRT-LLM offers lower latency on NVIDIA hardware but requires ONNX or TensorRT engine builds. Text Generation Inference (TGI) from Hugging Face provides good observability out of the box.

A minimal vLLM deployment might look like this:

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-3-3-70b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llama-3-3-70b
  template:
    metadata:
      labels:
        app: llama-3-3-70b
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - --model
            - meta-llama/Llama-3.3-70B-Instruct
            - --tensor-parallel-size
            - "2"
            - --max-model-len
            - "8192"
          resources:
            limits:
              nvidia.com/gpu: "2"
          ports:
            - containerPort: 8000