This article was originally published at sivaro.in

GPU Utilization vs Admission Control Tradeoff

Slug: gpu-utilization-vs-admission-control-tradeoff

---

GPU Utilization vs Admission Control Tradeoff: A Practitioner's Buying Guide

Two weeks ago I watched a Series B company burn $84,000 in a month on H100s that sat at 31% utilization. Their LLM inference queue was backed up 40 seconds deep at peak. Sounds contradictory, right? It's not. And it's the entire reason this article exists.

Here's the deal. Every team running GPUs for LLM serving hits the same wall: you can chase maximum GPU utilization OR you can guarantee low queue latency. You usually can't have both without serious engineering. The gpu utilization vs admission control tradeoff is the single most expensive decision you'll make in production AI infrastructure, and most teams get it wrong because they optimize the wrong number first.

By the end of this, you'll know which side of the tradeoff fits your workload, what queue-based admission control for LLM serving actually looks like in code, how to squeeze both metrics without lying to yourself, and how to pick tooling. I've built these systems. I've watched them fail. Let's go.

---

What "Admission Control" Actually Means (And Why It's Not a Firewall)

Admission control is simple in concept: decide whether to accept a request before you put it in the queue. That's it.

Most people hear "admission control" and think rate limiting. Wrong. Rate limiting says "you, the client, get 100 requests per second." Admission control says "the system can handle this right now, so we're letting it in. If not, we're rejecting, deferring, or degrading it."

That distinction matters because your GPU isn't a fixed resource. A 7B model on an A100 might take 12ms for a 100-token completion and 340ms for a 4,000-token completion. Same hardware, wildly different cost per request. Static rate limits are trash for this. They can't see the shape of the load.

Queue-based admission control for LLM serving goes further: it looks at current queue depth, expected service time, deadline budget, and sometimes SLO tier before deciding. It's the difference between a bouncer with a clicker and a bouncer who knows the band is about to stop playing.

Kubernetes doesn't give you this out of the box. The scheduler places pods. It has no idea your vLLM replica is 8 requests deep on a 30-second P99 budget. You have to build the layer yourself or buy it.

---

The Tradeoff, Stated Brutally

Here's the core tension:

High utilization means you keep the GPU busy. Batched requests pile in. Throughput per dollar goes up. But queue depths grow, tail latencies explode, and a single slow request starves everything behind it.

Strict admission control means you reject early. Queues stay short. P99 latency stays predictable. But you're rejecting work you could've done, and your expensive GPUs idle during the valleys.

Most teams pick one by accident. They deploy vLLM, crank --max-num-seqs to 256, and wonder why p99 is 8 seconds at 95% utilization. Or they set aggressive timeouts and watch utilization crater to 40% because every burst gets rejected.

I've run both configurations in production. Neither is "correct." The correct answer depends on your traffic shape, your SLOs, and honestly whether your customers notice tail latency.

---

When to Optimize for Utilization

If your workload is throughput-bound and latency-tolerant, push utilization hard. This is the right call for:

• Offline batch jobs. Embedding generation, document classification, batch summarization. Nobody's waiting on the response. Maximize throughput per GPU-hour.
• Best-effort inference. Free tier, internal tools, async workloads. If p99 occasionally hits 30 seconds, nobody churns.
• Training-adjacent serving. Fine-tune evaluations, synthetic data generation. Latency is irrelevant.

Company like Anthropic-style batch APIs (they launched the pattern publicly in 2024, and everyone copied it) is the archetype. You tell the customer "results within 24 hours" and batch the hell out of it. Utilization north of 90% is achievable and correct.

But here's what people miss: high utilization doesn't mean good utilization. If you're keeping the GPU busy with tiny 20-token requests, you're spiking utilization and getting terrible throughput per FLOP. Batching modern LLM inference is about token-level efficiency. Continuous batching (the vLLM and TensorRT-LLM approach) gets you there. Naive request batching doesn't.

---

When Admission Control Wins

If customers are sitting in front of your product watching a spinner, admission control is not optional. It's the product.

Latency-sensitive workloads:

• Chat interfaces. Sub-second first-token matters. Users bounce at 3 seconds. That's a hard behavioral cliff.
• Agentic pipelines. A single user action might trigger 12 sequential LLM calls. If each one adds 200ms of queue time, you've added 2.4 seconds invisibly.
• Real-time coding assistants. Completion latency is felt directly in the editor. Anything past 400ms feels broken.
• Voice. Dead air is fatal. You have maybe 700ms of budget total.

For these, you cap the queue. You set request deadlines. You reject or route-to-cheaper-model when the deadline won't be met. You use queue-based admission control for LLM serving because the alternative is churning users.

The cost: your utilization graph will look worse. That's fine. Utilization is a means, not an end. The end is served requests that met their SLO.

---

The Three Architectures You'll Actually Choose Between

Option A: Static Replica Provisioning (The Default)

You run N vLLM replicas behind a Kubernetes Service. Traffic hits them round-robin. Hope for the best.

Pros: Dead simple. No extra infrastructure. Works fine up to maybe 30 requests/sec.
Cons: Zero visibility into queue depth. No admission control. Kubernetes autoscaling reacts to CPU, which is useless for GPU inference. You will blow your SLO during bursts.

If you're pre-PMF and traffic is tiny, this is fine. Don't over-engineer. But please, add basic metrics. At minimum, scrape vLLM's num_requests_waiting and time_to_first_token histograms.

Option B: Queue-Based Admission Control Layer

You put an explicit queue between your API gateway and the GPU workers. The queue has a controller that decides accept/reject/defer based on real-time state.

# Rough sketch: deadline-aware admission control for vLLM pods
import time
from dataclasses import dataclass

@dataclass
class AdmissionDecision:
    accept: bool
    reason: str

class QueueAdmissionController:
    def __init__(self, max_queue_depth: int, deadline_ms: int):
        self.max_queue_depth = max_queue_depth
        self.deadline_ms = deadline_ms

    def admit(self, request_tokens: int, queue_depth: int,
              p99_service_ms: float, deadline_budget_ms: int) -> AdmissionDecision:
        # Rule 1: hard cap on queue depth
        if queue_depth >= self.max_queue_depth:
            return AdmissionDecision(False, "queue_full")

        # Rule 2: can we hit the deadline given current p99?
        expected_wait = queue_depth * (p99_service_ms / max(1, self.worker_count))
        remaining = deadline_budget_ms - expected_wait
        if remaining <= 0:
            return AdmissionDecision(False, "deadline_miss_predicted")

        # Rule 3: shrink admission window for very long prompts
        token_penalty = request_tokens / 4096.0
        if remaining < (p99_service_ms * (1 + token_penalty)):
            return AdmissionDecision(False, "cost_exceeds_budget")

        return AdmissionDecision(True, "ok")

Pros: Predictable latency. You can enforce SLO tiers. Queue depth becomes a first-class metric you can autoscale on.
Cons: You're now running a stateful system. Failure modes multiply. You need to handle worker restarts, stale metrics, and the classic "controller says accept but worker is dead" race.

Option C: Utilization-First with Priority Preemption

You let everything in, but you tag requests by priority and preempt low-priority work when high-priority arrives.

# Priority class example for GPU inference workloads
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: llm-critical
value: 1000000
preemptionPolicy: PreemptLowerPriority
description: "Paid tier LLM inference — SLO bound"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: llm-batch
value: 100
preemptionPolicy: Never
description: "Best effort batch inference — preemptible"

Caveat: Kubernetes pod preemption is not the same as request preemption within a running inference server. You need vLLM's scheduling options or a custom scheduler inside the model server. This is where gpu queue latency optimization kubernetes gets hard, because the K8s scheduler and the in-process scheduler are two different animals that don't talk to each other.

Pros: Highest utilization while protecting critical latency.
Cons: Requires app-level cooperation. Preempting a KV-cache-heavy request mid-flight wastes real compute.

---

Comparison Table: Pick Your Poison

Dimension	Static Replicas	Queue-Based AC	Priority + Preemption
Utilization ceiling	High (85-95%)	Medium (55-70%)	High (80-90%)
p99 latency control	None	Strong	Strong for high-pri
Kubernetes fit	Native	Needs custom controller	Needs dual scheduler
Operational complexity	Low	Medium	High
Reject rate at burst	Implicit (timeouts)	Explicit, measurable	Low for high-pri
Best for	Low traffic, batch	Chat, real-time	Mixed-tier SaaS
Cost at steady-state	Low	Medium	Medium
Cost at burst	Catastrophic (SLO blowout)	Predictable	Mixed

I've shipped all three. My default recommendation for anything user-facing is queue-based admission control with a priority lane bolted on. Start with Option B. Add Option C's priority handling only when you have paying customers who'll churn over tail latency.

---

The Metrics That Actually Matter

Stop staring at nvidia-smi utilization percentage. It lies to you.

Here's what to instrument instead:

Duty cycle vs utilization. Utilization is "how much of the time was the GPU doing something." Duty cycle weighted by real work is what you want. A GPU waiting on KV-cache memory transfer shows 100% utilization and delivers 12 tokens/sec. Useless metric.

Queue depth histogram, not average. An average queue depth of 3 hides bimodal behavior where you're either at 0 or at 40. Use histograms.

TTFT vs TPOT separately. Time to first token and time per output token have different bottlenecks. TTFT is dominated by prefill and queueing. TPOT is dominated by decode efficiency and memory bandwidth. Instrument both.

Rejection rate as a first-class SLO. If your admission controller rejects 8% of requests at peak, that's a product decision, not a bug. Track it.

Effective throughput per dollar. Tokens served per GPU-hour. This is the number your CFO cares about. Optimizing utilization percentage without this metric is amateur hour.

Here's a Prometheus query I use constantly for gpu queue latency optimization kubernetes work:

# p99 queue wait time across all vLLM pods
histogram_quantile(0.99,
  sum(rate(vllm:request_queue_time_seconds_bucket[5m])) by (le, model)
)

# Admission rejection rate by reason
sum(rate(admission_rejections_total[5m])) by (reason)
  / sum(rate(admission_requests_total[5m]))

---

The Contrarian Take: Your Utilization Target Is Probably Too High

Most people think 90%+ GPU utilization is the goal. It's not.

Look at the math. If you run at 95% utilization, you have zero headroom for a traffic spike without blowing latency. Everything is queue. If you run at 70%, you absorb a 40% spike gracefully. The 30% "wasted" capacity is insurance.

At SIVARO, we target 65-75% sustained utilization for latency-sensitive workloads and 90%+ for batch. The delta between those numbers is the price of an SLO.

"But GPUs are expensive!" Yes. So is customer churn. Do the math: an H100 at $3/hr running 24/7 costs ~$2,200/month. If keeping it at 70% instead of 95% saves you 1% monthly churn on a $50k/month customer, you win by $500 minus the ~$500 in idle GPU time. Wash. Now add the second customer who stays because latency stayed low. You win.

Compute the SLO cost, not just the GPU cost.

---

Kernel-Level and Framework-Level Levers

Admission control is the macro decision. But you have micro levers that shift the entire curve:

Continuous batching. vLLM, TensorRT-LLM, and SGLang all do this. Without it, you're leaving 3-5x throughput on the table. This isn't optional in 2026.

PagedAttention / KV-cache paging. vLLM's signature technique. Cuts memory fragmentation dramatically, which raises the ceiling on concurrent batch size.

Chunked prefill. SGLang and newer vLLM versions split long prefills into chunks so decode doesn't stall. This is the single biggest p99 latency win I've seen in the last 18 months.

Speculative decoding. For latency-bound workloads, a small draft model generating candidates can cut TPOT significantly. Costs extra compute, so it's a utilization-for-latency trade you're buying with FLOPs.

# vLLM config knobs that actually matter for the tradeoff
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.3-70B-Instruct",
    tensor_parallel_size=4,
    max_num_seqs=128,              # concurrency ceiling — key lever
    max_num_batched_tokens=8192,   # prefill batching — tokens not requests
    enable_chunked_prefill=True,   # p99 latency win, slight throughput cost
    gpu_memory_utilization=0.92,   # leave headroom for KV cache growth
    swap_space=8,                  # GB of CPU swap — the overflow valve
)

That max_num_seqs number is where you feel the tradeoff most directly. Bump it up, utilization rises, tail latency rises. Bump it down, latency is crisp, throughput drops. Only your workload profile tells you the right value.

---

Buying Guide: What to Actually Deploy

If you're under 10 req/sec: Static replicas with basic monitoring. Don't buy anything. Write 200 lines of Python if you need a queue.

If you're 10-100 req/sec, one model, latency-sensitive: Buy or build a queue-based admission controller. Options in 2026: Ray Serve (has admission hooks), KServe with custom transformers, BentoML's adaptive batching, or Roll your own on top of Redis Streams. We use Redis Streams + a custom controller at SIVARO because nothing off-the-shelf understood our deadline math.

If you're 100+ req/sec, multiple models, tiered customers: You need a real inference gateway. Look at llm-d (the Kubernetes-native effort that's gained serious adoption this year), NVIDIA Dynamo, or a custom layer. Expect to spend 3-6 engineer-months.

If you're doing batch only: Skip admission control. Use Ray Data or a simple job queue. Crank max_num_seqs and sleep well.

Don't buy enterprise inference platforms just for admission control. Every vendor has it now. Buy them for the operational leverage — canary deploys, model versioning, multi-region failover. Admission control is table stakes.

---

FAQ

Q: Does high GPU utilization always mean lower LLM quality?
No. Utilization and output quality are unrelated. The tradeoff is between utilization and latency. Quality is a function of model choice and sampling parameters, not how full your batch is.

Q: Can I get 90% utilization AND sub-500ms TTFT?
Yes, if you engineer for it. You need continuous batching, chunked prefill, and a workload with relatively uniform prompt lengths. With highly variable prompts, the variance alone will force you to pick one.

Q: Is Kubernetes admission control the same thing as LLM admission control?
No. Kubernetes admission control validates pod specs at the API server. LLM admission control decides whether to accept inference requests. Different layers entirely. If a vendor conflates them, be suspicious.

Q: What's a reasonable rejection rate at peak?
For free tier, 10-20% is normal and healthy. For paid tiers, aim under 2%. Over 5% on paid is a churn generator.

Q: How do I know if I need queue-based admission control for LLM serving?
Measure your p99 TTFT during peak. If it's more than 2x your median, you have a queueing problem and you need admission control. If it's within 1.5x, you're fine with simpler approaches.

Q: Does vLLM have built-in admission control?
vLLM has a max_num_seqs cap, which is a crude form of admission control — it just queues everything else silently. For real admission control, you need a layer in front that sees the whole fleet.

Q: Should I buy an inference gateway or build my own?
Build if your requirements are unusual. Buy if they're standard. My rule: if you're spending more than 20% of an engineer's time on it, buy. That threshold hits fast.

---

The Bottom Line on the GPU Utilization vs Admission Control Tradeoff

There's no universal right answer to the gpu utilization vs admission control tradeoff. The right configuration depends on your SLO, your traffic shape, and what your customers will tolerate. But there is a universal wrong answer: pretending you can max out both without doing the engineering.

Pick your SLO first. Instrument the metrics that matter. Then tune utilization to meet the SLO, not the other way around. Batch workloads can chase 90%+ utilization. Latency-sensitive workloads should sit at 65-75% and treat the gap as SLO insurance. Everything else lives in between, and the only way to find your spot is to measure your actual queue behavior under real load.

Kubernetes won't save you here. It schedules pods, not requests. The admission layer is yours to build or buy. Build it deliberately, or it'll build itself across your customers' annoyed p99s.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.