This article was originally published at sivaro.in

Admission Control vs Request Prioritization LLM

Posted by Nishaant Dixit on September 11, 2026

A client called me last Tuesday, slightly panicked. Their vLLM cluster was falling over under a traffic spike from a new feature launch. They'd tried bumping --max-num-seqs from 128 to 256, then 512. It just made things worse. Latency went from 400ms p99 to 4 seconds, then requests started timing out entirely.

The fix wasn't more concurrency. It was the opposite. We installed an admission control layer that started shedding traffic at 60% GPU utilization, and p99 latency dropped to 180ms within an hour. They served fewer requests per second, but they served the right ones, and they stopped crashing.

This is the difference between admission control vs request prioritization llm serving that nobody explains clearly. Most teams think they're the same thing. They're not. And choosing wrong is why your inference cluster keeps dying.

The Real Problem You're Solving

You have finite GPU memory. A single H100 80GB running Llama 3.1 70B in FP8 takes roughly 70GB for weights. That leaves you about 10GB for KV cache, which at 8K context is maybe 40-60 concurrent sequences depending on batch shape. Push past that and the scheduler either preempts, swaps, or crashes.

Admission control decides whether to accept a request at all. Request prioritization decides which accepted request gets GPU time next. They're two different mechanisms solving two different failure modes.

If you conflate them, you end up with the classic bug I see in a dozen startups a year: a priority queue with no admission gate. High-priority traffic starves low-priority traffic, the low-priority queue grows unbounded, memory blows up on the queue metadata, and the whole thing topples. I watched a fintech do this in March 2026 during earnings season. Their priority queue OOM'd the API gateway. Two hours of downtime.

Admission Control vs Max Concurrency LLM Serving: The Distinction That Matters

Here's what throws people. In vLLM, you set --max-num-seqs 256 and think you've done admission control. You haven't. You've set a concurrency ceiling. The scheduler will still accept every request into the queue and then thrash trying to serve them.

Max concurrency is a static number. Admission control is a policy.

# What most people do: static concurrency cap
engine_args = {
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "max_num_seqs": 256,       # ceiling, not admission control
    "gpu_memory_utilization": 0.90,
}
# Result: queue grows to 10K requests, p99 explodes

# What actually controls admission: rejection at the gate
def admit(request, gpu_state):
    if gpu_state.kv_cache_used_pct > 0.75:
        return REJECT_503
    if gpu_state.queue_depth > 2 * gpu_state.optimal_batch:
        return REJECT_503
    if request.estimated_tokens > gpu_state.free_kv_tokens:
        return REJECT_503
    return ACCEPT

The static cap tells the scheduler how big a batch to build. The admission function decides whether a request ever reaches the scheduler.

I've run both. Static caps alone work fine up to about 60% utilization. Past that, you need a real admission gate.

Admission Control for HuggingFace TGI Inference

TGI added proper admission control hooks in the 3.x line. The --max-batch-prefill-tokens and --max-concurrent-requests flags are the concurrency knobs, but the interesting one is --waiting-served-ratio combined with the router's queue policy.

What TGI does well: it exposes a /metrics endpoint with tgi_queue_size and tgi_batch_current_size. You can write an external admission controller that polls these and returns 503s from your ingress before requests ever hit the TGI server.

# Envoy rate limit + TGI queue-aware admission
- name: tgi_admission
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
    domain: tgi
    descriptors:
      - key: gpu_utilization
        value: "high"
    # Combined with a stats sink reading tgi_queue_size
    # Reject when queue > 4 * max_batch_total_tokens

The gotcha with TGI: it doesn't natively support priority classes the way vLLM does with its --scheduling-policy flag. If you need request prioritization llm semantics inside TGI, you're building it yourself at the router layer. I usually recommend running TGI behind a custom FastAPI gateway that tags requests with priority and routes to separate TGI instances per priority tier. Costs more GPU, but it's simpler than fighting the scheduler.

We did this at a healthcare client in July 2026. Separate pools for interactive chat (high priority, small model) and batch summarization (low priority, big model). They run on the same node, different GPUs. Throughput dropped maybe 15%. Their SLA compliance went from 71% to 99.4%.

Request Prioritization in vLLM: What Actually Works

vLLM 0.6.x and later support --scheduling-policy=priority with per-request priority values. This is real prioritization, not the fake kind where you just sort the queue.

from vllm import SamplingParams

# Priority: lower number = higher priority in vLLM
high = SamplingParams(temperature=0.7, priority=0)
low = SamplingParams(temperature=0.7, priority=10)

# Scheduler will preempt low-priority sequences to make room for high-priority
# But ONCE ADMITTED, not at the gate

Here's the dirty secret. vLLM's priority scheduling preempts running sequences. When a high-priority request arrives, it can evict a low-priority sequence mid-generation, swap its KV cache to CPU, and restart it later. That swap is expensive. On a 70B model with 8K context, a full KV swap costs you 200-400ms.

So priority scheduling without admission control creates a thrashing pattern. High-priority arrives, evicts low, low finishes, gets re-admitted, high-priority arrives again, evicts low. Your GPU spends more time swapping KV cache than generating tokens.

The fix is brutal and simple: don't admit low-priority traffic above a hard ceiling. Reserve 30% of your KV cache budget for high-priority only.

# Admission gate that protects priority headroom
HIGH_PRIORITY_RESERVE = 0.30

def admit(request, state):
    if request.priority == HIGH:
        # High priority can use the reserve
        return ACCEPT if state.kv_used_pct < 0.95 else REJECT
    else:
        # Low priority capped at 70% of KV cache
        usable = 1.0 - HIGH_PRIORITY_RESERVE
        return ACCEPT if state.kv_used_pct < usable else REJECT

This single change eliminated the thrash pattern for a customer in August 2026. Their p50 stayed flat, but p99 for high-priority dropped from 3.1s to 340ms.

When You Need Both

Most production systems need both admission control and request prioritization. Here's my rule of thumb after building these at four companies:

• Admission control alone is enough when all traffic has the same SLA and you just need to survive overload.
• Prioritization alone is enough when you have headroom and want to shape latency distribution among tiers.
• Both are required when you have multiple SLAs and you're running hot (>70% steady-state utilization).

The combination looks like this in a real ingress:

import time
from fastapi import FastAPI, HTTPException

app = FastAPI()

class Scheduler:
    def __init__(self):
        self.kv_used = 0.0
        self.queue_depth = 0
        self.high_budget = 0.30

    def admit(self, req):
        # Stage 1: hard rejection
        if self.kv_used > 0.95:
            return False, "gpu_saturated"
        if self.queue_depth > 500:
            return False, "queue_full"

        # Stage 2: priority-aware admission
        if req.priority == "low":
            if self.kv_used > (1.0 - self.high_budget):
                return False, "reserved_for_high_priority"

        return True, "ok"

@app.post("/v1/completions")
async def generate(req):
    ok, reason = scheduler.admit(req)
    if not ok:
        raise HTTPException(503, detail=reason)
    # forward to vLLM/TGI

Two stages. First protects the system. Second protects your SLAs.

The Benchmark That Changed My Mind

I used to think admission control was a band-aid. Just buy more GPUs, right? Then in February 2026 we ran a load test comparing four configs on identical hardware (4x H100, Llama 3.1 70B FP8, 8K context, 2000 concurrent users):

Config	Throughput (tok/s)	p50 latency	p99 latency	Error rate
No control	8,400	2.1s	14.7s	8.3%
Max concurrency cap	9,100	1.8s	9.2s	2.1%
Admission control	8,900	620ms	2.3s	0.1%
Admission + priority	8,200	480ms (high)	1.1s (high)	0.0%

Read that carefully. The best-performing config on raw throughput was the concurrency cap. But it had 2.1% errors and 9-second p99. The admission+priority config served 9% fewer tokens per second but delivered 1.1 second p99 for high-priority traffic and zero errors.

For a customer-facing product, that trade is obviously correct. For batch processing where you don't care about latency, the concurrency cap wins.

The Cost Angle Nobody Talks About

Admission control lets you run your GPU fleet hotter. That's the real economic argument.

Say you're running 8x H100 for inference. At $3/hr per H100 on-demand, that's $24/hr, roughly $17,500/month. If admission control lets you go from 55% to 80% safe utilization, you either serve 45% more traffic on the same hardware or shut down 2-3 GPUs. That's $5-6K/month in savings, or the equivalent capacity uplift.

I've seen teams spend six figures on observability for inference and skip the admission layer entirely. Backwards. You can't observe your way out of a saturated GPU. You have to say no to some traffic.

FAQ

Is admission control just rate limiting?

No. Rate limiting is per-client, time-windowed, and oblivious to system state. Admission control reads current GPU KV cache, queue depth, and batch size, and rejects requests that would harm in-flight work. A rate limiter that allows 100 rps will happily kill your cluster if your cluster can only serve 60 rps at the current context length.

Can I do admission control without modifying vLLM or TGI?

Yes. Put an Envoy or nginx layer in front that reads /metrics from the server and rejects with 503 when queue depth or KV utilization crosses thresholds. It's less precise than in-process admission but works in 30 minutes.

Does priority scheduling in vLLM preempt running requests?

Yes, and this is the main reason priority alone isn't enough. Preemption causes KV cache swaps that cost 200-400ms on large models. You want admission control reserving headroom so preemption is rare, not routine.

What's the right KV cache utilization threshold to start rejecting?

For interactive SLAs, start rejecting at 70%. For batch, 90% is fine. The threshold depends on how much your tail latency can tolerate. There's no universal answer, and anyone claiming one is guessing.

How does this change with 1M token context models?

Dramatically. Long-context requests blow up KV cache. A single 200K-token request on a 70B model consumes more memory than 40 short requests. You need token-aware admission, not request-count admission. Estimate KV footprint per request before admitting.

Does admission control work with disaggregated prefill/decode?

It works differently. You need separate admission gates for prefill and decode pools, and the prefill gate should be much more aggressive because prefill is compute-bound. I'll write a separate piece on this. It's a whole topic.

What about speculative decoding and admission control?

Speculative decoding changes your compute/memory trade. Draft model KV cache eats into the same budget. Reduce your admission thresholds by the draft model's footprint or your acceptance rate collapses under load.

How do I know if my current setup needs admission control?

Graph p99 latency against request rate. If the curve has a knee, and past the knee p99 grows superlinearly while throughput stays flat, you're saturated. Add admission control. If p99 grows linearly with load, you're just under-provisioned, and admission control won't save you.

My Recommendation

If you're running any LLM inference in production with user-facing SLAs, build admission control first. It's cheap, it's a few hundred lines of code, and it prevents the failure mode where your cluster dies under load.

Add request prioritization second, and only when you actually have differential SLAs. Priority without admission is a footgun. Priority without reserved headroom is a thrash machine.

And if you're choosing between 2 more H100s versus an admission control layer: build the admission layer first. You'll squeeze more useful throughput out of the GPUs you have than out of the ones you don't. I've watched this decision play out at six companies now. The teams that built admission control first survived their traffic spikes. The ones that bought more GPUs first bought more GPUs again three months later, and their p99 was still garbage.

The uncomfortable truth of admission control vs request prioritization llm serving is that they're not competitors. They're layers. The question isn't which one to buy. It's which one to build first. Build admission. Then build priority. Your on-call rotation will thank you.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.