From GPU Saturation to Autoscaling: Engineering a Capacity-Driven AI Inference Platform on Amazon EKS

Running an LLM in Kubernetes is not the difficult part.

The difficult part is knowing when one GPU-backed replica is actually full, which workload signal should add another expensive GPU, and whether scaling happens before users encounter a latency cliff.

I built an AI inference capacity engineering platform on Amazon EKS to investigate those questions through measurement rather than assumptions.

The complete implementation is available here:

Kelvinskell / ai-inference-capacity-engineering-platform

Production-grade AI inference optimization and capacity engineering on Amazon EKS with vLLM, KServe, Envoy AI Gateway, GPU autoscaling, LMCache, AMP, and performance-cost analysis.

AI Inference Capacity Engineering

A production-grade, capacity-driven AI inference platform on Amazon EKS, built for high-throughput LLM serving, GPU efficiency, benchmark-derived autoscaling, and full-stack observability with KServe, vLLM, Envoy AI Gateway, KEDA, Prometheus, and Grafana.

Mermaid source: docs/architecture/platform-architecture-diagram.md

What This Project Proves

This repository is more than a collection of deployment manifests. It connects infrastructure, serving, traffic control, observability, benchmarking, and autoscaling into one capacity-engineering loop:

1. Provision an EKS platform with Terraform, EKS Auto Mode, S3 model storage, and Spot-first GPU capacity.
2. Serve a pinned DeepSeek-R1-Distill-Qwen-14B-AWQ revision with KServe RawDeployment and vLLM.
3. Protect the OpenAI-compatible API with authentication, request limits, token quotas, and inference-aware timeouts.
4. Measure request latency, token throughput, scheduler pressure, KV-cache demand, and NVIDIA GPU telemetry.
5. Scale from benchmark-derived Prometheus signals instead of generic CPU utilization.

The result is a reproducible platform for answering the questions that matter in inference operations: Where is the throughput ceiling? When does…

View on GitHub

The platform serves DeepSeek-R1-Distill-Qwen-14B-AWQ with KServe and vLLM, controls the public request path with Envoy AI Gateway, provisions Spot-first GPU capacity with Karpenter, and uses Prometheus metrics to drive KEDA autoscaling.

Most importantly, its scaling thresholds come from a reproducible benchmark suite.

The System I Wanted to Build

I wanted the project to answer five operational questions:

1. What is the practical throughput of one GPU-backed predictor?
2. At what concurrency does throughput stop improving?
3. When does request queueing become sustained pressure?
4. How do prompt and output length change the capacity envelope?
5. Which metrics can scale the service before latency becomes unacceptable?

That led to an architecture with four distinct ownership layers:

• Terraform owns AWS infrastructure and platform controllers.
• Kubernetes manifests own model-specific workloads and traffic policies.
• Prometheus owns the shared operational evidence.
• The benchmark harness converts that evidence into capacity recommendations.

Platform Architecture

The public inference path is:

Client / Open WebUI
        |
        v
AWS Network Load Balancer
        |
        v
Envoy Gateway + Envoy AI Gateway
        |
        v
KServe InferenceService
        |
        v
vLLM OpenAI-compatible server
        |
        v
DeepSeek R1 14B AWQ on one NVIDIA GPU

The major components are:

Layer	Implementation	Responsibility
Cloud foundation	Amazon VPC, EKS, IAM, S3	Network, identity, cluster, and durable model storage
GPU capacity	Karpenter Spot and On-Demand NodePools	Provision GPU nodes as predictor demand changes
Model serving	KServe RawDeployment and vLLM	Run the OpenAI-compatible inference server
Traffic management	Envoy Gateway and Envoy AI Gateway	Authentication, routing, rate limits, quotas, and timeouts
Autoscaling	KEDA and HPA	Adjust predictor replicas from Prometheus queries
Observability	Prometheus, Grafana, Alertmanager, DCGM Exporter	Correlate inference behavior with GPU telemetry
Evidence	Async Python load generator and metrics collector	Measure the service and derive capacity guidance

Why Generic CPU Autoscaling Is Not Enough

CPU utilization does not describe the important bottlenecks of an LLM inference server.

A predictor can experience:

• requests waiting in the vLLM scheduler,
• increasing KV-cache occupancy,
• fully utilized GPU execution capacity,
• growing prefill cost from long prompts,
• growing decode time from long outputs,
• or rapidly increasing tail latency.

Those conditions can exist while CPU utilization remains a poor representation of user experience.

I therefore used three independent autoscaling signals:

1. Average waiting requests over a rolling one-minute window
2. The sum of KV-cache utilization across predictor replicas
3. p99 inference latency over five minutes

KEDA exposes each signal to the HPA. The HPA calculates a desired replica count for every trigger and applies the largest result.

Establishing a Capacity Envelope

The main concurrency benchmark used:

• one ready GPU-backed predictor,
• DeepSeek-R1-Distill-Qwen-14B-AWQ,
• 1,024 input tokens per request,
• 128 output tokens per request,
• non-streaming OpenAI-compatible completions,
• 30 seconds of warmup,
• 480 seconds of measurement per case,
• and the full authenticated Envoy Gateway path.

The load generator maintained real asynchronous concurrency. It also generated exact-length prompts from a deterministic corpus and added unique request fingerprints to avoid producing misleading shared prefix-cache benefits.

For every measurement window, the collector queried Prometheus for client latency, vLLM scheduler behavior, token throughput, KV-cache utilization, and NVIDIA DCGM metrics.

Benchmark Results

Concurrency	Successful requests/s	Total tokens/s	p95 latency	p99 latency	Avg waiting	Max waiting
1	0.33	381	3.1s	3.1s	0.000	0
5	0.98	1,134	5.1s	5.2s	0.242	2
10	1.32	1,518	8.0s	8.5s	1.152	5
20	1.54	1,778	13.9s	15.5s	1.424	7
40	1.63	1,879	25.9s	35.5s	2.000	25

The important result is not the highest throughput number. It is the shape of the curve.

Increasing concurrency from 20 to 40 added only 0.09 successful requests per second. During the same increase:

• p99 latency rose from 15.5s to 35.5s,
• average waiting requests rose from 1.424 to 2.000,
• maximum waiting requests rose from 7 to 25.

At concurrency 40, the service was doing more waiting for almost no useful throughput gain.

I therefore use approximately 1.5 successful requests per second as the practical planning capacity of one replica for this specific request shape. The saturated result of 1.63 requests per second is not a sensible operating target.

Average Queue Depth vs Maximum Queue Depth

This distinction matters.

Maximum queue depth is valuable for diagnosing bursts, but a single peak should not necessarily provision another GPU. The first benchmark version reported maximum waiting requests prominently, which made a threshold of two look like an early scale-out decision.

The average tells a different story.

At concurrency 5, the queue briefly reached 2, but its average was only 0.242. Average queue depth reached 2 only at concurrency 40, where p99 latency had already crossed 30 seconds.

For that reason, the KEDA query now averages the raw vLLM waiting-request gauge over one minute:

sum(
  avg_over_time(
    vllm:num_requests_waiting{
      namespace="llm-serving",
      pod=~"deepseek-r1-14b.*"
    }[1m]
  )
)

avg_over_time smooths each predictor's queue gauge. The outer sum produces cluster-wide queue pressure across all replicas.

The initial threshold remains 2, but it is important to describe it correctly: it is a saturation guardrail, not an early queue detector. Multi-replica scale testing may show that it should be lowered to add capacity sooner.

Why I Sum KV-Cache Utilization

Queue depth is a direct pressure signal, but it is not always persistent. When a new predictor becomes ready, waiting requests can immediately move onto the added capacity. The queue then falls, even though the active workload may still require both replicas. If queue depth is the only signal holding the extra replica, the HPA can eventually scale it down after the stabilization window, recreate the queue, and trigger another scale-up cycle.

KV-cache occupancy provides a more sustained view of the active inference working set. The query deliberately sums the raw utilization fractions across predictor replicas:

sum(
        vllm:kv_cache_usage_perc{
                namespace="llm-serving",
                pod=~"deepseek-r1-14b.*"
        }
)

The target is 0.75 per replica. KEDA and HPA divide the summed KV-cache utilization by 0.75 and round up to the next whole replica.

Summed KV-cache utilization, S	Desired replicas
0 <= S <= 0.75	1
0.75 < S <= 1.50	2
1.50 < S <= 2.25	3
2.25 < S <= 3.00	4
3.00 < S <= 3.75	5
S > 3.75	6, capped by maxReplicaCount

For example, one predictor must move above 0.75 before the cache signal requests two replicas:

pod-1 = 0.76
sum   = 0.76

ceil(0.76 / 0.75) = 2 replicas

After pod 2 becomes ready, active request contexts redistribute across both pods:

pod-1 = 0.38
pod-2 = 0.38
sum   = 0.76

ceil(0.76 / 0.75) = 2 replicas

Neither pod is individually above 0.75, but their sum remains above the threshold at 0.76. The cache signal therefore keeps two replicas even after the request queue drains.

For the service to scale from two replicas to three, aggregate KV-cache utilization must grow beyond 2 x 0.75 = 1.50:

pod-1 = 0.76
pod-2 = 0.75
sum   = 1.51

ceil(1.51 / 0.75) = 3 replicas

After pod 3 becomes ready, active request contexts redistribute again:

pod-1 = 0.51
pod-2 = 0.50
pod-3 = 0.50
sum   = 1.51

ceil(1.51 / 0.75) = 3 replicas

The total remains 1.51, so the cache signal continues to request three replicas. Averaging would produce 1.51 / 3 = 0.503 and make the same active workload appear smaller merely because another replica became available.

The HPA still applies its tolerance, stabilization windows, and scaling policies before changing the deployment.

This does not make KV-cache occupancy a perfect demand signal. A few long-context requests can consume substantial cache without producing high request throughput, and cache usage falls as sequences complete. I use it as a complementary capacity-retention signal: queue depth detects requests that cannot enter execution, summed KV-cache utilization reflects active state distributed across replicas, and p99 latency protects the user-facing objective.

The KEDA Scaling Policy

The service scales between one and six replicas:

spec:
  minReplicaCount: 1
  maxReplicaCount: 6
  pollingInterval: 15
  cooldownPeriod: 180

The three trigger targets are:

Signal	Threshold	Role
One-minute average waiting requests	2	Sustained scheduler-pressure guardrail
Sum of per-replica KV-cache utilization	0.75	Active working-set and capacity-retention signal
p99 inference latency	30s	User-experience guardrail

Scale-up can double the replica count every 30 seconds after a 60-second stabilization window. Scale-down removes at most 25% of replicas every 60 seconds after a five-minute stabilization window.

The service keeps one warm replica because GPU-node provisioning, S3 model mounting, weight loading, vLLM initialization, and KV-cache allocation make scale-from-zero unsuitable for this workload.

Request Shape Changes Everything

Request rate alone is not enough for LLM capacity planning.

At concurrency 20, I held the prompt at 1,024 tokens and increased output length:

Output tokens	Successful requests/s	Total tokens/s	p99 latency
128	1.54	1,772	15.8s
512	0.78	1,202	28.1s
1,024	0.45	928	48.8s

The largest output reduced completed-request throughput by more than 70% and tripled p99 latency.

Long prompts also increased prefill cost and KV-cache pressure. This means a capacity model based only on requests per second will fail when tenants submit materially different token shapes.

A production admission or quota system should consider at least:

• prompt tokens,
• requested output tokens,
• concurrent requests,
• queue pressure,
• and workload-specific latency objectives.

Spot-First GPU Capacity

Karpenter manages two GPU NodePools:

• gpu-spot has a higher scheduling weight and acts as the preferred elastic tier.
• gpu-on-demand remains available as the baseline fallback when Spot capacity is constrained.

Both pools use a dedicated GPU NodeClass, apply an NVIDIA GPU taint, and enforce aggregate GPU limits. Predictor pods request one GPU and tolerate only the intended inference nodes.

This separates replica scaling from node provisioning: KEDA decides how many predictor pods are needed, while Karpenter supplies the nodes required to schedule them.

Reproducible Model Delivery

The model is not embedded in the serving image.

A Kubernetes Job downloads pinned Hugging Face revisions and synchronizes them to a private S3 bucket. For each artifact, the uploader:

1. Calculates a SHA-256 hash
2. Skips matching objects already in S3
3. Uploads changed files with hash metadata
4. Writes _MANIFEST.json only after every file succeeds

Predictor pods mount the validated S3 prefix read-only through Mountpoint for Amazon S3 CSI.

This keeps serving images small and gives replacement pods a shared artifact source. It does not remove cold-start cost: pods still need to read model data, load weights onto the GPU, initialize vLLM, and allocate cache memory.

One Observability Source

Prometheus is used for more than dashboards.

It is simultaneously:

• the metric backend for KEDA,
• the source for benchmark measurement windows,
• the data source for Grafana,
• and the evaluator for GPU alerting rules.

vLLM exposes request, token, scheduler, queue, and KV-cache metrics. DCGM Exporter adds GPU utilization, framebuffer memory, power, temperature, and activity telemetry.

That shared data source makes a scaling event explainable. I can compare the signal that requested a replica with the latency, scheduler state, and GPU behavior visible during the same period.

What Is Validated and What Is Not

The current benchmark is intentionally scoped.

It validates one model revision, one ready predictor, one tested GPU path, and non-streaming requests. It does not establish a universal capacity number for DeepSeek, vLLM, or every GPU type.

The next important experiments are:

• end-to-end KEDA scale-out under sustained load,
• GPU-node provisioning and model-load recovery time,
• multi-replica queue behavior,
• streaming response capacity,
• GPU-memory configuration comparisons,
• and model-length configuration comparisons.

The current development gateway credential and default UI passwords must also be externalized before production use. Redis and the rate-limit service are single-replica components in the present development architecture.

Calling those boundaries out is part of capacity engineering. A benchmark result is useful only when its scope is explicit.

What I Learned

Three conclusions changed how I think about inference autoscaling:

1. Saturation is a curve, not a single utilization number

The GPU was fully utilized across the concurrency tests. The useful signal came from observing where throughput flattened while queueing and latency continued to rise.

2. Queue statistics need precise semantics

Maximum queue depth describes bursts, while a rolling average describes sustained pressure. Queue depth can disappear as soon as added capacity accepts the waiting work, so it needs a more persistent companion signal.

3. Aggregate cache demand can retain capacity after a queue drains

Summing per-replica KV-cache utilization preserves the active working set when requests are redistributed. Averaging the percentages would make the same workload look smaller merely because another replica became available.

4. Token shape belongs in capacity planning

Two requests are not equivalent when one generates 128 tokens and another generates 1,024. Request rate without token dimensions hides the actual compute and latency profile.

Explore the Project

The repository includes the complete Terraform environment, Kubernetes manifests, architecture decision records, benchmark harness, raw result schema, operational runbooks, dashboards, and benchmark report:

GitHub: Kelvinskell/ai-inference-capacity-engineering

Useful starting points:

• Platform architecture
• Benchmark report
• Benchmark-derived autoscaling decision
• KEDA ScaledObject
• Benchmark orchestrator

The project is built around a simple principle: GPU scaling decisions should be tied to measured workload behavior, not infrastructure defaults.