This article was originally published at sivaro.in

Admission Control vs Max Concurrency LLM Serving

How to stop your GPUs from melting when traffic spikes — and why the queue you don't manage will manage you.

I got paged at 2:47 AM on a Tuesday in August 2025 because a client's inference cluster had gone from 60% utilization to fully wedged in under four minutes. Nobody changed anything. A retry storm from an upstream service did it. Their config had max_concurrency: 256 on a vLLM deployment across eight A100s, and the moment a downstream dependency started timing out, the client's frontend retried every request three times, the queue ballooned, KV cache filled, and every single request — including the health checks — started timing out at 30 seconds. We restarted pods for two hours. That night is why I care about admission control vs max concurrency in LLM serving the way other people care about their college football team.

Here's the thing most teams learn the hard way: max concurrency is a knob. Admission control is a policy. One sets a ceiling. The other decides who gets through the door when the ceiling is the least of your problems.

This article is a buying guide — not for a product exactly, but for an architectural decision you're going to make whether you realize it or not. I'll walk through what each approach actually does under load, where max concurrency alone falls over, how admission control fits into Hugging Face TGI inference, how it relates to request prioritization, and what I'd pick (and have picked) for different workloads. You'll come away knowing which one to ship, which to add later, and which combination has cost me the least sleep.

What max concurrency actually controls (and what it doesn't)

Max concurrency in an LLM server — vLLM, TGI, TensorRT-LLM, SGLang — is a cap on how many requests can be in flight at once. Simple. When the server hits that number, new requests either wait in an internal queue or get rejected with a 429, depending on config.

That sounds like backpressure. It isn't. Not really.

Why? Because concurrency and latency aren't linear. On a single A100 running Llama 3 70B with tensor parallelism across two GPUs, going from 16 to 32 concurrent requests might cost you 30% more per-token latency. Going from 64 to 128 might triple it. The relationship is a curve, and past some point it's a cliff. Max concurrency is a flat number that ignores the shape of that curve entirely.

# vLLM-style config — the naive version
engine:
  max_num_seqs: 256        # max concurrent sequences
  max_num_batched_tokens: 8192
  gpu_memory_utilization: 0.90
  kv_cache_dtype: fp8

Set max_num_seqs to 256 on a 70B model and you've told the scheduler "please let 256 requests fight for KV cache." When memory pressure hits, vLLM starts preempting sequences — swapping them to CPU or recomputing them. Throughput goes up. Tail latency goes to Mars.

Max concurrency answers one question: how many requests can the box hold without crashing? It does not answer: should this particular request be here right now?

And that's the whole ballgame when you're serving production traffic.

What admission control adds on top

Admission control is the practice of deciding whether to accept a request at all — before it enters the serving system. It's the bouncer, not the room capacity sign.

A real admission controller looks at more than concurrency. It looks at:

• Current queue depth and estimated wait time
• Request priority class (interactive vs batch vs eval)
• Token budget (a 4K-token generation is not the same as a 20-token completion)
• Tenant identity and fair-share quotas
• Whether the request can be shed or delayed without business damage

Then it does one of four things: admits, queues with a deadline, rejects with a specific error, or degrades (shorter max_tokens, smaller model, cached response).

Here's the contrarian bit, and I'll say it plainly: most teams don't need a sophisticated admission controller on day one. They need a boring one. max_concurrency plus a queue-depth-based reject. That gets you to maybe 50 requests per second per node with predictable p99. The teams that get in trouble are the ones who crank max_concurrency to 512 because benchmarks look great, then wonder why their p99 is 40 seconds during a marketing push.

Admission control isn't about squeezing more throughput. It's about protecting the latency SLO for the traffic you've decided matters.

Admission control for Hugging Face TGI inference

TGI has a slightly different model than vLLM, and it's worth knowing the specifics because a lot of people run TGI behind a router and assume the router is doing admission control. It usually isn't.

TGI exposes a --max-concurrent-requests flag and a --max-batch-total-tokens limit. It also has a client-side queue managed by the router in the text-generation-inference repo — but the router is not a general-purpose admission controller. It's a load balancer with health checks and a simple queue.

# TGI launch — this is a concurrency cap, not a policy
text-generation-launcher \
  --model-id meta-llama/Llama-3.3-70B-Instruct \
  --max-concurrent-requests 128 \
  --max-batch-total-tokens 16384 \
  --max-input-length 4096 \
  --max-total-tokens 8192

Notice what's missing: no notion of priority, no tenant quotas, no per-request deadlines, no adaptive shedding. If you send 128 requests from a batch job and one interactive request, the interactive request gets no preference. It waits behind 127 batch tokens-generations.

So admission control for Hugging Face TGI inference almost always lives in front of TGI, not inside it. In practice that means one of three things:

A gateway layer (Envoy, LiteLLM, a custom FastAPI shim) that enforces token-bucket rate limits per tenant and tags requests with priority before forwarding.

A queue service (Redis + a worker pool, or something like Celery) that holds requests, sorts by priority and deadline, and dispatches to TGI below its concurrency limit.

Or, if you're already running on Kubernetes with a service mesh, an admission webhook that rejects at the mesh layer using custom metrics exported by TGI's Prometheus endpoint.

The last one is my default. TGI exports tgi_queue_size and tgi_batch_current_size. You can wire those into a KEDA scaler and into a rejecting admission layer simultaneously. When tgi_queue_size > 2 * max_concurrent_requests, reject at the edge with a 503 and a Retry-After. Don't let it hit TGI.

That's the trick. The cheap, ugly, effective trick.

Admission control vs request prioritization in LLM serving

People conflate these two constantly. They're different layers solving different problems.

Admission control decides whether a request enters the system. Request prioritization decides when an admitted request gets served relative to others.

You can have admission control without prioritization: reject when full, FIFO otherwise. That's fine for single-tenant workloads.

You can have prioritization without admission control: accept everything, but serve premium traffic first. This is a disaster. Your queue grows unbounded, low-priority requests consume memory, and eventually the scheduler's own bookkeeping becomes the bottleneck. I've watched a Postgres-backed priority queue fall over at 40K pending items because the dequeue query started doing table scans.

The combination that actually works, and that I've deployed for three clients now, is admission control at the edge with priority-class tagging, and preemptive scheduling inside the inference server using those tags.

vLLM has a priority field on RequestOutput when you use the async engine, but it's coarse. For real multi-tenant priority, you either run separate deployments per priority class (interactive pool + batch pool) or you build a small dispatcher that owns the KV cache budget directly.

Two pools is boring and it works. One pool with clever scheduling is elegant and it will page you at 3 AM.

I'll pick boring.

# Minimal priority-aware admission gate in front of TGI/vLLM
import asyncio, time
from dataclasses import dataclass

@dataclass
class Request:
    tenant: str
    priority: int        # 0 = interactive, 1 = standard, 2 = batch
    deadline: float      # unix ts
    tokens_estimate: int

class AdmissionGate:
    def __init__(self, max_inflight: int, max_queue: int):
        self.max_inflight = max_inflight
        self.max_queue = max_queue
        self.inflight = 0
        self.queue = asyncio.PriorityQueue()

    async def submit(self, req: Request):
        if self.inflight < self.max_inflight:
            self.inflight += 1
            return "ADMIT"
        if self.queue.qsize() >= self.max_queue:
            return "REJECT"
        if req.deadline < time.time() + 2.0:
            return "REJECT"  # can't meet deadline, don't queue it
        await self.queue.put((req.priority, time.time(), req))
        return "QUEUED"

That's the shape. About 30 lines. Not a framework. It handles 90% of what teams actually need.

Notice the deadline check. That one line — "if we can't serve you in time, don't accept you" — is the single most valuable thing admission control does that max concurrency can't. It converts a future timeout into an immediate, honest rejection. Your client's retry logic gets a clear signal instead of ambiguity. That matters more than any throughput number.

When max concurrency alone is fine

I want to be fair to the simple approach, because I've also watched teams over-engineer admission control and ship nothing.

Max concurrency alone is fine when:

You have one workload type and one tenant. Internal tooling, a demo, a research endpoint. If everyone hitting the box is doing the same thing, priority is meaningless.

Your clients handle 429s well. If downstream has exponential backoff and a real queue, letting the server reject is a valid backpressure signal.

Your traffic is smooth. If you're serving 5 requests per second steady with occasional bursts to 20, a generous concurrency cap plus autoscaling handles it. Don't build a bouncer for a party that never gets crowded.

Your SLO is throughput, not latency. Batch inference jobs — offline eval, embedding generation, dataset scoring — don't care about p99. They care about tokens per dollar. Set max concurrency high, let the scheduler preempt, run at 90%+ GPU utilization, done.

I've shipped that config for a data-labeling pipeline handling 40M generations a month. It's fine. It would be wrong for a customer-facing chat product.

The mistake isn't choosing max concurrency. It's choosing max concurrency because you didn't know there was another option.

What I'd actually buy (or build)

If you're standing up LLM serving in September 2026, here's my honest recommendation matrix, based on what I've run and what I've watched fail.

For a customer-facing chat or agent product: Admission gate at the edge with priority classes, deadline-based rejection, and separate GPU pools per priority tier. vLLM for throughput, not TGI, unless you specifically need TGI's token streaming guarantees. Expect to spend a week building the gate and a week tuning the deadline heuristic.

For RAG over a large corpus with a handful of enterprise tenants: LiteLLM or a similar gateway for per-tenant rate limits and cost tracking, TGI behind it for the generation, and a Redis-backed admission queue with tenant fair-share. This is what I'd call tier-2 complexity.

For batch or offline: Max concurrency, nothing else. Turn off priority. Use the cheapest GPUs that fit the model. Ship it.

For research or internal: One deployment, max concurrency set to whatever makes the box stable, no gate. Move on with your life.

The pricing angle nobody talks about: admission control pays for itself through GPU reduction. When I added a proper gate to a client's 24-GPU cluster in early 2026, they dropped to 16 GPUs and their p95 got better, because the scheduler stopped thrashing on KV cache eviction. Two months of GPU spend covered the engineering. That's the actual ROI story, and it has nothing to do with features on a comparison sheet.

One more thing, and I'll say it directly since I've been burned: don't buy a commercial "LLM gateway" that markets admission control as a checkbox and then runs a single global semaphore. I've evaluated three of them in the last year. Two were rate limiters with a new dashboard. The third worked, but only after we rewrote their default config. Ask specifically: does it do per-request deadline awareness? Does it tag priority? Does it export queue-depth metrics you can alert on? If the answer to any is no, it's a max concurrency knob with a marketing budget.

FAQ

What's the difference between admission control vs max concurrency in one sentence?
Max concurrency caps how many requests the server holds; admission control decides which requests deserve to be held.

Can vLLM do admission control natively?
Not really. It has max_num_seqs and preemption, which is capacity management. Priority scheduling exists but is coarse. Real admission control usually lives in front of it.

Is admission control for Hugging Face TGI inference built into the router?
No. The TGI router balances and health-checks. It doesn't enforce priority, deadlines, or tenant quotas. You add those upstream.

How is admission control vs request prioritization LLM different in practice?
Admission says yes or no. Prioritization says "you first." You need admission to make prioritization safe — otherwise the queue grows without bound.

What's a good starting deadline heuristic?
Set it roughly at your p99 SLO minus 20%. If p99 target is 5 seconds and the model takes 3 seconds at median load, reject anything queued past 1 second that can't finish in time.

Will admission control reduce my throughput?
Under light load, no. Under heavy load, yes — you'll shed requests. That's the point. Total bytes served goes down, useful requests served goes up.

Do I need Kubernetes for this?
No. You can run an admission gate as a sidecar, a Lambda, or a small Go service. Kubernetes helps with scaling, not with the policy itself.

What metrics should I alert on?
Queue depth relative to max in-flight, rejection rate by priority class, and p99 time-to-first-token. If rejection rate on interactive traffic crosses 2%, something's wrong upstream or you need capacity.

Is there an off-the-shelf product that does this well?
As of September 2026, the honest answer is: a couple of gateways get close (LiteLLM, BentoML's newer routing, some of the Ray Serve patterns), but the last mile of deadline-aware, priority-tagged admission is still custom work at most shops. Budget for it.

The final call

If you take one thing from this: admission control vs max concurrency LLM serving isn't a rivalry. Max concurrency is a parameter. Admission control is a policy. You will always have the parameter. You might not have the policy, and if you don't, a retry storm at 2:47 AM will write it for you — badly.

Build the gate. Make it boring. Reject early, reject honestly, and give your clients a Retry-After they can actually use. Then measure p99 instead of throughput, because that's what your users feel. The GPU bill will follow.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.