Running production LLM inference at scale requires more than a GPU cluster. Autoscaling keeps latency low during traffic spikes without burning budget during quiet periods, but the mechanics differ significantly from standard web services. Model weights are measured in gigabytes, initialization times are non-trivial, and the cost structure of token-based billing can distort how you think about capacity. Whether you are self-hosting with vLLM or TensorRT-LLM, or evaluating managed APIs, the goal is the same: maintain a consistent latency SLO while controlling infrastructure spend.
Design Autoscaling Around Inference Patterns
LLM inference is not stateless request processing. The prefill phase is compute-intensive, while decode is memory-bandwidth-bound. A Horizontal Pod Autoscaler that targets CPU utilization will miss the actual bottleneck, which is often GPU memory bandwidth or KV-cache capacity.
Build scaling policies around the inference lifecycle. For self-hosted deployments using vLLM or Text Generation Inference (TGI), expose custom metrics such as batch size, queue depth, and active slot utilization. Scale out when queue depth exceeds a threshold, not when GPU utilization crosses 80%. Increase batch size to improve throughput, but cap it based on your Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT) budgets. If your workload is dominated by long-context prefills, consider scaling on prefill latency rather than token throughput.
Choose Metrics That Reflect User Experience
Standard infrastructure metrics are insufficient for LLM serving. Track these directly:
- TTFT: Measures how quickly a user sees the first token. High TTFT means your batch or queue is too deep.
- TPOT / Inter-token latency: Captures streaming quality during decode.
- Queue depth: The number of requests waiting for a free slot. This is the cleanest signal for scaling.
- KV-cache utilization: Prevents out-of-memory crashes on long-context requests.
Export these from your inference engine into Prometheus, and use KEDA or a custom metrics pipeline to drive replica counts. Averaging over too long a window will delay scale-out, so use short evaluation windows or scale on both average and instantaneous queue depth.
Eliminate Cold Starts
A 70B parameter model can take several minutes to load from network storage onto GPU memory. In a serverless or aggressive downscaling setup, that latency is unacceptable for interactive applications. Mitigations include persistent local NVMe caching, pre-baked container images with model weights, or simply maintaining a minReplicaCount of one or more.
If cold starts are a hard constraint, managed inference platforms remove them entirely. Oxlo.ai serves popular models with no cold starts, so requests hit warm GPUs immediately without requiring pre-warming logic or over-provisioned node pools.
Optimize Cost for Context Length and Workload Type
Token-based pricing creates a coupling between prompt length and cost that self-hosted infrastructure does not share. A 100K token prompt consumes the same GPU time regardless of billing model, but on a token-based provider it can cost orders of magnitude more than a short prompt. This mismatch makes cost attribution difficult when autoscaling for agentic or retrieval-augmented generation workloads.
Oxlo.ai uses flat request-based pricing: one cost per API request regardless of prompt length. For long-context and agentic loops, this decouples your infrastructure scaling decisions from token-counting economics. Request-based pricing can be 10-100x cheaper than token-based for long-context workloads, and it makes capacity planning predictable. See https://oxlo.ai/pricing for current plan details.
Implement Traffic Splitting and Gradual Rollouts
When you upgrade a
Top comments (0)