Most GPU clusters running LLM inference waste between 30% and 50% of their available compute, not because engineers are careless but because Kubernetes treats GPUs as atomic units with zero visibility into KV cache memory. That's the core problem with GPU scheduling for LLMs in 2026: the hourly rate you negotiated so hard barely matters if your orchestrator silently discards a third of the throughput you're paying for.
Silicon is abundant. What's scarce is effective utilization, and that's governed by scheduling software, not contract negotiation. The teams winning at infrastructure cost aren't the ones with the cheapest GPU contracts. They're the ones running cache-aware, fractional schedulers that extract maximum useful tokens from hardware they already have.
I've been looking at the data, and the pattern is consistent: orchestration is the primary cost lever now, and most teams are optimizing the wrong variable.
Why Does Kubernetes Waste So Much GPU Compute?
Kubernetes was built for stateless web services and batch training jobs where one pod genuinely needs one whole GPU. LLM inference breaks that model completely.
The root cause is a mismatch between how the scheduler thinks about resources and what inference actually requires. When you request nvidia.com/gpu: 1, you get an entire physical card — 80GB of HBM on an H100 — regardless of whether your 7B model needs 14GB. A cluster running three different 7B models with sporadic traffic will show each GPU as fully "allocated" while actually busy less than 15% of the time. The scheduler refuses to place new work on a card that's marked consumed.
Three specific blind spots drive the waste:
- No KV cache visibility. The scheduler can't see how full a running pod's KV cache (the attention state that determines concurrent request capacity) actually is. Routing a long-context request to a pod at 90% cache utilization produces dramatically different latency than one at 10%. Kubernetes treats them identically.
- Cold-start blindness. Spinning up a new inference pod takes 2–5 minutes for image pulls, weight deserialization, and warm-up passes. The scheduler happily places a pod on a node that will make users wait four minutes because it has no concept of model loading state.
- Broken autoscaling. Horizontal Pod Autoscaler triggers on CPU and memory metrics. At the moment LLM inference most needs to scale — a deep queue with requests waiting for token generation — memory utilization might sit at 30% and CPU flat. The system looks healthy while users experience unbounded queue delays.
The consequence is predictable: teams overprovision heavily, keeping warm instances for every model to avoid cold-start penalties. That's not a configuration problem — it's a fundamental architectural mismatch between the orchestrator's resource model and inference workload characteristics.
How Does Cache-Aware Routing Fix This?
If round-robin load balancing scatters requests across all replicas, recomputing the same system prompt on every pod, cache-aware routing does the opposite: it sends each request to the GPU that already has the relevant context cached.
The open source llm-d project implements this with a scheduling layer called the endpoint picker. Before forwarding each request, it checks which replica holds the matching prefix, factors in queue depth and real-time load, then routes accordingly. In upstream benchmarks on shared-prefix workloads, this approach cut time-to-first-token by more than 99% and more than doubled throughput — without changing hardware.
The mechanics are straightforward. LLM engines like vLLM cache key-value tensors computed during prefill. If the next request shares the same prefix, the engine skips that computation entirely. Intelligent scheduling computes it once and reuses it.
This is the same principle that makes cache-aware load balancing critical for LLM inference economics: matching requests to replicas holding relevant cached prefixes restores throughput instead of degrading it linearly as fleets grow.
Can Fractional GPU Sharing Actually Work in Production?
Whole-GPU allocation is the other silent killer. If your 7B model needs 14GB of VRAM and you're giving it an 80GB H100, you're paying for 66GB of idle memory. Fractional GPU sharing fixes this, and the benchmarks are now strong enough to take seriously.
NVIDIA Run:ai's fractional GPU allocation benchmarked with Nebius delivered 77% of full GPU throughput and 86% of full-GPU concurrent user capacity using only a 0.5 GPU fraction, with TTFT under one second. At 0.25 fraction, they saw up to 2x more concurrent users on smaller models. On mixed workloads — chat, reasoning, embeddings — they measured up to 3x more total system users on shared GPUs.
Kubernetes is finally catching up. Dynamic Resource Allocation (DRA) reached general availability in v1.34 and is enabled by default since v1.35, allowing pods to request fractional GPU memory natively. This absorbs the encoding work that HAMi (the CNCF incubating project) previously had to hack around with mutating webhooks and annotations. HAMi's container-level enforcement — limiting CUDA calls to allocated fractions — remains necessary because DRA was never designed for that. The two are complementary, not competitive.
Here's the comparison that matters:
| Approach | Pricing Impact | Key Feature | Target Audience |
|---|---|---|---|
| Whole-GPU pods (K8s native) | Baseline hourly rate, 30-50% wasted compute | Simple ops, no extra software | Teams with homogeneous, high-density workloads |
| Fractional GPU (Run:ai / HAMi + DRA) | 2x+ utilization on same hardware | Sub-GPU memory and compute allocation | Multi-model serving with variable traffic |
| Serverless GPU (Modal, RunPod) | Higher per-hour rate, zero idle cost | Scale-to-zero, per-second billing | Bursty inference under 30% utilization |
The serverless break-even point is worth noting: for bursty inference, serverless GPU compute wins against an always-on instance below roughly 30% utilization, despite the higher per-hour rate. Above that threshold, you're paying a premium for capacity you could use yourself.
What Scheduling Algorithms Actually Reduce Tail Latency?
Mean latency gets the headlines, but tail latency — P99 — dominates user experience. A system that averages 200ms but occasionally spikes to 8 seconds feels worse than one that consistently delivers 400ms. Several research efforts published at ICML 2026 tackle this directly.
A tail-aware scheduling framework from Microsoft Research takes a contrarian approach: it's prediction-free. Recent schedulers approximate Shortest Job First using predicted decode lengths, but those predictions are fragile under distribution shifts and bursty arrivals. This framework replaces explicit length prediction with soft priority boosting driven by lightweight statistical signals, co-optimizing scheduling with cache-aware preemption. The result: P99 time-to-last-token reduced by up to 35-50% relative to SRPT with perfect length knowledge, and TTFT reduced by 34-47% across reasoning-heavy and chat-heavy workloads.
MAPS (Memory-Aware Predictive Scheduling) takes a different angle, focusing on disaggregated serving where prefill and decode run on separate instances. MAPS performs device-assisted speculative output length prediction overlapped with cloud-side prefilling — the estimate is essentially free because it runs on the user's device while the server prepares. It then uses uncertainty-aware calibration to derive safe upper bounds.
For reasoning models specifically, PASCAL phase-aware scheduling distinguishes between the reasoning phase (extended CoT that delays user-visible output) and the answering phase. It prioritizes reasoning to reduce TTFT while using controlled preemption and token pacing during answering to preserve quality-of-experience. On DeepSeek-R1-Distill-Qwen-32B benchmarks, PASCAL reduced tail TTFT by up to 72% while maintaining answering phase SLO attainment.
The common thread: none of these require newer or more GPUs. They're pure scheduling improvements that extract better latency characteristics from existing hardware.
How Does KV Cache Quantization Change the Cost Equation?
GPU memory is the binding constraint for LLM inference, and the KV cache — not model weights — is usually what fills it first. Quantizing that cache from BF16 to FP8 halves its memory footprint, which directly increases how many concurrent requests you can serve.
FP8 KV cache doubled concurrent requests from 32 to 64 and reached 2,192 tokens per second — about 41% higher than BF16's peak — for roughly 30% less cost per token. At any single concurrency level, BF16 is actually a few percent faster per token. But BF16 runs out of memory at 32 concurrent requests while FP8 keeps going to 64. The throughput gain comes from concurrency, not raw speed.
This connects to the broader cost equation: if you're thinking about model serving architecture costs, KV cache quantization is one of the few levers that simultaneously increases throughput and reduces per-token cost without changing hardware. The tradeoff is a small accuracy risk — FP8 attention kernels add conversion overhead per token — but in production, the concurrency gain dwarfs the per-token overhead.
On the kernel side, vLLM's integration with Tencent HPC-Ops backends on 8× H20 cut TTFT by about 24% and time-per-output-token by about 17% versus default, with up to 2.95× decode speedup on mixed-length batches. The attention backend uses a per-step, load-balanced decode scheduler that avoids stalling on the longest request in a mixed batch. These are drop-in improvements — no source changes, no fork.
What Does the Tooling Landscape Look Like for Large-Scale Clusters?
Once you move past single-node serving, the scheduling problem shifts from request routing to job placement across thousands of GPUs with complex topology constraints.
Slurm's topology-aware job scheduling on NVIDIA GB200 NVL72 aligns jobs with NVLink domain boundaries, minimizing fragmentation. In simulations on a 5,000-node cluster, this achieved GPU occupancy within 1% of theoretical maximum. The key insight: larger job segment sizes (up to 18 nodes) benefit high-I/O workloads like mixture-of-experts training by keeping all communicating GPUs within a single NVLink domain, while smaller jobs use 2-8 node segments to avoid scheduler constraints.
On the AMD side, ROCm Spur is a new AI-native job scheduler written in Rust, designed for multi-thousand GPU clusters. It's drop-in compatible with Slurm's CLI, REST API, and C FFI, uses WireGuard mesh networking, and provides GPU-first scheduling with state that survives restarts. It's early, but the Slurm compatibility means teams can evaluate it without rewriting their orchestration layer.
UiPath offers a real-world example of the shared-fleet model. They shifted from isolated GPU clusters to a shared fleet managed across training and inference, using Google Cloud's Dynamic Workload Scheduler to reserve capacity ahead of time. During busy periods, the fleet handles real-time inference and latency-sensitive workloads. When demand eases, it switches to batch training and longer-running jobs. This eliminated the idle capacity that their previous per-instance elasticity model left during quiet periods.
One more infrastructure-level concern: power. Empromptu's Grid Guard software staggers GPU workloads by 50-200 milliseconds to reduce synchronized power spikes, with the vendor claiming an average 80% reduction in power volatility without significant performance impact. That claim isn't independently validated, but the underlying problem is real — increasingly dense GPU deployments create highly variable electrical demand that grids weren't designed for.
Which Scheduling Approach Should You Choose?
The decision framework is simpler than the tooling landscape suggests. It maps to three questions:
What's your traffic pattern? If you're running bursty inference below 30% utilization, serverless GPU with scale-to-zero wins despite the higher per-hour rate. If you have sustained traffic, fractional GPU sharing with cache-aware routing is the lever. If you're running homogeneous, high-density workloads that genuinely saturate whole GPUs, plain K8s pods are fine — but verify that's actually the case.
What's your model mix? Single-model deployments with uniform request sizes can get away with simple round-robin. Multi-model serving with variable traffic patterns needs KV-cache-aware routing — the llm-d endpoint picker or equivalent. Reasoning models with extended CoT phases benefit from phase-aware scheduling like PASCAL.
What's your cluster scale? Under a few hundred GPUs, Kubernetes with DRA for fractional allocation covers most needs.
The hourly rate is the wrong variable to optimize. By mid-2026, the winning edge in AI infra is scheduling software — not cheaper contracts or newer GPUs. Teams using cache-aware fractional schedulers cut effective cost 30-50% versus those optimizing only hourly rate. If your GPU utilization is below 70% and you're not using fractional allocation or cache-aware routing, that gap is your money sitting on the table.
The open question worth tracking: DRA just reached GA in Kubernetes v1.34, and HAMi is rebuilding on top of it. Will the native Kubernetes scheduling API absorb enough of the fractional-GPU problem to make dedicated schedulers like Run:ai unnecessary, or will the enforcement layer and cache-awareness always require purpose-built tooling? The answer determines whether GPU scheduling becomes infrastructure plumbing or remains a competitive moat.
Originally published at SaaS with Alex
Top comments (0)