This article was originally published at sivaro.in

Token Bucket vs Queue Based Admission Control LLM

Most teams get this wrong the first time.

They spin up vLLM behind a load balancer, point their app at it, and watch latency fall apart the moment real traffic hits. I've seen it at three companies now. The model is fine. The GPUs are fine. What's broken is admission control — the layer that decides which requests get in, which wait, and which get rejected outright.

If you're running LLM inference in production, you need to understand token bucket vs queue based admission control llm at a mechanical level. Not the Wikipedia version. The version that decides whether your p99 latency is 2 seconds or 40.

Here's what I'll cover: how each approach actually works under GPU constraints, when to pick one over the other, how Kubernetes interacts with both (spoiler: badly by default), and the hybrid patterns I've shipped to real customers.

Let's get into it.

Why LLM Admission Control Isn't Like Normal API Rate Limiting

Traditional rate limiting assumes requests are cheap and uniform. An HTTP GET costs roughly the same whether you send 100 or 100,000. LLM inference doesn't work that way.

A 200-token completion and a 4,000-token completion both count as "one request." But one occupies an H100 for 300ms. The other for 12 seconds. If your admission layer counts requests, you're counting the wrong thing.

This is why the industry settled on tokens as the unit of work. OpenAI, Anthropic, and every serious self-hosted stack price and limit on tokens for exactly this reason.

The second wrinkle: GPUs don't queue linearly. Thanks to continuous batching in engines like vLLM and TensorRT-LLM, throughput actually increases as you add concurrent requests — up to a point. Then KV cache fills, preemption kicks in, and things collapse. Admission control is about finding that cliff and staying just behind it.

Third wrinkle: the cost of saying "no" is different. Drop a normal API request, user retries. Drop an LLM request mid-generation after 8 seconds of compute, you've burned money for nothing.

Token Bucket: The Classic, and Why It Still Works

If you've used AWS API Gateway or nginx's limit_req, you've used a token bucket. The mechanics are simple: a bucket holds N tokens, refills at rate R per second. Each request takes tokens. Empty bucket, request rejected.

For LLM serving, you tune two numbers: token capacity and refill rate. Capacity = burst tolerance. Refill = sustained throughput.

Here's a minimal implementation in Python:

import time
from dataclasses import dataclass

@dataclass
class TokenBucket:
    capacity: int
    refill_rate: float  # tokens per second
    tokens: float = 0.0
    last_refill: float = 0.0

    def __post_init__(self):
        self.tokens = self.capacity
        self.last_refill = time.monotonic()

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now

    def try_consume(self, amount: int) -> bool:
        self._refill()
        if self.tokens >= amount:
            self.tokens -= amount
            return True
        return False

The critical detail for LLMs: amount should be estimated total tokens (prompt + expected completion), not request count. If you don't know the completion length upfront, use a conservative estimate from your historical p50, or require the client to declare a max_tokens and bill against that.

What token buckets do well:

• Burst absorption. A client can spike to 2x sustained for a few seconds without rejection.
• O(1) memory per client. No queue state, no backpressure tracking.
• Predictable fairness. Every client gets the same refill rate, guaranteed.
• Cheap to implement. Every major API gateway does it. You can run it at the edge before traffic hits your GPU nodes.

What they do badly:

• No visibility into what's waiting. A rejected request vanishes. The client sees a 429 and has to decide what to do.
• Binary. There's no "you're third in line, ETA 4 seconds." It's yes or no.
• Doesn't account for GPU state. Bucket says you have tokens, but the GPU's KV cache might be 92% full. Bucket doesn't know.
• Refill rate is a guess. Set it too high, you overload GPUs. Too low, you leave capacity idle during quiet periods.

I've shipped token buckets for early-stage products and they work fine up to maybe 50 requests per second per GPU. Past that, the lack of queue visibility starts to hurt.

Queue-Based Admission Control: Where the Real Complexity Lives

Queue-based admission control says: instead of rejecting immediately, put the request in a bounded queue. A worker pulls from the queue when GPU capacity is available. If the queue is full, then reject.

Sounds simple. It isn't.

The core question is: what's the queue discipline? FIFO? Priority? Shortest-job-first? And what's the queue depth before you start dropping?

I've found that shortest-job-first (SJF) with priority aging is the right default for LLM serving. Here's the reasoning, and it surprised me when I first measured it.

If you have 8 concurrent GPU slots and 20 waiting requests — 5 short (100 tokens out), 15 long (2,000 tokens out) — FIFO takes 25+ minutes to drain. SJF drains the short ones in ~30 seconds, then the long ones. Average latency drops by 60-70%.

The catch: SJF starves long requests. A single 4,000-token completion can wait indefinitely if short requests keep arriving. That's why you add aging — every 10 seconds in queue, a request's effective priority jumps.

A working queue implementation:

import asyncio
import heapq
from dataclasses import dataclass, field

@dataclass(order=True)
class InferenceRequest:
    priority: float
    enqueued_at: float = field(compare=False)
    payload: dict = field(compare=False)

class AdmissionQueue:
    def __init__(self, max_depth: int, aging_seconds: float = 10.0):
        self.max_depth = max_depth
        self.aging_seconds = aging_seconds
        self.heap: list[InferenceRequest] = []
        self.lock = asyncio.Lock()

    def _effective_priority(self, req: InferenceRequest, now: float) -> float:
        waited = now - req.enqueued_at
        # Lower priority number = served sooner
        return req.priority - (waited / self.aging_seconds)

    async def enqueue(self, req: InferenceRequest) -> bool:
        async with self.lock:
            if len(self.heap) >= self.max_depth:
                return False
            # Re-heapify with aging applied
            heapq.heappush(self.heap, req)
            self._reage()
            return True

    def _reage(self):
        now = asyncio.get_event_loop().time()
        self.heap = [InferenceRequest(
            priority=self._effective_priority(r, now),
            enqueued_at=r.enqueued_at,
            payload=r.payload,
        ) for r in self.heap]
        heapq.heapify(self.heap)

    async def dequeue(self) -> InferenceRequest | None:
        async with self.lock:
            if not self.heap:
                return None
            return heapq.heappop(self.heap)

Notice the tradeoffs I baked in:

• Bounded queue depth. Unbounded queues are how you get OOM kills on the GPU node.
• Aging every dequeue. Without it, starvation is guaranteed under load.
• Priority is a float, not an int. Lets you compose multiple signals (tenant tier, estimated cost, deadline).

Queue theory matters here. For LLM serving capacity planning, the key insight from queueing theory is Little's Law: L = λW (average number in system = arrival rate × average wait time). If you know your arrival rate and target latency, you can compute the queue depth you need. But Little's Law assumes a stable system — and LLM inference systems are not stable during traffic spikes. That's why bounded queues with explicit rejection are safer than theoretical sizing.

For deeper reading on the theory, Leonard Kleinrock's Queueing Systems is still the canonical reference, and the Google SRE Book has a solid practical chapter on handling overload with admission control.

How Admission Control Actually Works in Kubernetes for GPU Inference

This is where most teams get ambushed.

Kubernetes doesn't have native GPU-aware admission control. The scheduler sees GPUs as opaque resources (nvidia.com/gpu: 1). It has no idea about KV cache pressure, batch size, or memory fragmentation.

What you get out of the box:

• Pod scheduling ensures a GPU node isn't oversubscribed at the pod level.
• ResourceQuota limits pods per namespace.
• Horizontal Pod Autoscaler scales replicas based on metrics — but GPU scaling lags by minutes, not seconds.

What you don't get:

• Request-level admission control inside a model server
• Backpressure signaling from the GPU to the ingress
• Token-aware rate limiting at the edge

The way I've seen this solved in production: two-layer admission.

Layer 1: an Envoy or NGINX sidecar at the pod level doing token bucket rate limiting. This stops egregious traffic patterns before they hit anything expensive.

Layer 2: an application-level queue inside the inference server (vLLM has one built in, or you build your own with the pattern above). This handles GPU-aware admission.

You can also use KEDA to scale GPU replicas based on queue depth rather than CPU — this is the pattern that actually works for LLM workloads. HPA with CPU metrics on a GPU pod is useless; the CPU is idle while the GPU is pegged.

Here's a KEDA ScaledObject for queue-depth-based GPU scaling:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 2
  maxReplicaCount: 16
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: vllm_num_requests_waiting
        query: |
          avg(vllm:num_requests_waiting{namespace="inference"})
        threshold: "8"

The threshold of 8 is deliberate. If each replica handles ~16 concurrent requests comfortably and the queue is single-digit, you're under-saturated. If queue depth exceeds 8 per replica sustained for 60 seconds, scale up. This isn't theoretical — it's the number we landed on at a customer after a week of tuning.

Token Bucket vs Queue Based Admission Control LLM: The Real Comparison

Here's the honest table. Not marketing.

Dimension	Token Bucket	Queue-Based
Rejection timing	Immediate	After queue fills
Burst tolerance	High (up to capacity)	Low (fills queue)
Latency predictability	Poor under load	Better with SJF
Fairness across tenants	Strong	Weaker (needs priority)
Memory overhead	O(clients)	O(queue depth)
GPU-state awareness	None	Possible with custom impl
Implementation cost	Low	Medium-high
Debuggability	Easy	Harder (hidden state)

Most teams should run both. Token bucket at the edge for tenant-level fairness and abuse prevention. Queue-based admission at the model server for GPU-aware scheduling.

But if you have to pick one — and you do, because half-measures fail — pick based on traffic shape.

Pick token bucket when: traffic is spiky but predictable, you have many tenants, latency SLAs are relaxed (sub-10s acceptable), and you don't want to build queueing infrastructure.

Pick queue-based when: you're running a single high-value workload, latency SLAs are tight, you need to squeeze every GPU-dollar, and your traffic pattern rewards job-size-aware scheduling.

What I've Actually Shipped

At SIVARO, we built a hybrid for a customer running customer-support LLM inference for a fintech. Volume: 400 req/sec peak, 40 req/sec baseline. SLA: p99 under 4 seconds for tier-1 tenants.

The first version was pure token bucket. It worked at baseline, catastrophically failed at peak. Tier-1 customers got 429s because tier-3 traffic had consumed the shared bucket.

Second version: per-tenant token bucket at the edge + global queue at the server. Better, but tier-1 customers were still waiting behind tier-3 requests once they got into the queue.

Third version (current): per-tenant token bucket, per-tenant priority queue, SJF within priority. Tier-1 gets 10x priority weight, short requests (under 200 output tokens) get 3x weight. Aging kicks in at 15 seconds.

Results: p99 for tier-1 dropped from 8.2s to 2.9s. GPU utilization went from 61% to 84%. Total GPU cost down 22% because we could drop a replica.

That 22% is the honest case for spending engineering time on admission control. The token bucket is easy. The queue is where the leverage is. (Yes, I said leverage. Fight me.)

Practical Sizing: How Many Tokens, How Deep a Queue

Here's the cheat sheet I use.

Token bucket parameters:

• Refill rate = (p95 sustainable tokens/sec per GPU) × (number of GPUs) × 0.7
• Capacity = refill rate × 4 (four-second burst window)

The 0.7 is headroom. If you set refill to the theoretical max, you'll see queue backup even with tokens available because the bucket is optimistic.

Queue depth:

• Target p99 latency L, average service time S per request
• Depth = (L / S) × concurrency_limit × 1.5

For a customer with S=800ms, L=4s, concurrency=32, that's a depth of ~240. We set it at 200 and reject beyond. Rejections at that depth were 0.02% of peak traffic.

The 1.5 multiplier is because latency distributions in LLM serving are long-tailed. Prompt lengths vary, generation lengths vary, and preemption adds jitter. You want cushion.

The Continuous Batching Interaction

One thing that nobody tells you: continuous batching changes the admission calculus.

In vLLM or TensorRT-LLM, requests join the running batch mid-flight. A new request can enter the GPU the moment any other request finishes — not at a batch boundary. This means the effective "queue" is shorter than a classic batch system because slots free up continuously.

But it also means your queue depth math is wrong if you use static batching assumptions. A queue of 200 with continuous batching behaves more like a queue of 60-80 in a static system, because drain rate is smoother.

I spent a week at a customer (before I understood this) tuning queue depth down from 200 to 40, and their p99 went from 6s to 1.8s. That was the moment I stopped treating queue theory as a black box and started re-deriving it for the actual serving stack.

Current Landscape (September 2026)

The ecosystem has moved. As of mid-2026:

• vLLM 0.9+ ships with a production-grade admission controller built in. It handles priority queues natively; you just plug in your tenant mapping.
• Ray Serve now has GPU-aware autoscaling with queue-depth signals as a first-class concept.
• KServe added a gateway-level token bucket that understands OpenAI-style token accounting.
• NVIDIA Dynamo (released late 2025) has a dedicated frontend gateway that handles both patterns and lets you compose them declaratively.

If you're starting fresh today, you probably shouldn't build the token bucket from scratch — use Envoy's local_ratelimit filter or KServe's built-in. Build the queue logic, because that's where your workload-specific tuning lives.

FAQ

What's the difference between token bucket and queue based admission control for LLM serving?

Token bucket is a rate limiter that rejects requests when a client exceeds their allowed token rate. Queue-based admission control accepts requests into a bounded queue and rejects only when the queue is full, using scheduling policy (FIFO, priority, SJF) to decide who runs next. The first is about fairness and abuse prevention; the second is about GPU utilization and latency shaping.

How does admission control work in Kubernetes for GPU inference?

Kubernetes itself doesn't provide request-level admission control for GPU workloads. You layer it: token bucket rate limiting at the ingress or sidecar (Envoy, NGINX), queue-based admission inside the inference server (vLLM's built-in, or custom), and KEDA or custom autoscaling based on queue depth rather than CPU. The default HPA using CPU metrics is essentially useless for GPU inference workloads.

Which one should I pick for my first LLM deployment?

Start with a token bucket at the edge. It's simple, cheap, and stops obvious problems. Add queue-based admission once you're seeing GPU utilization above 60% and latency variance that you can't explain. Don't build both on day one.

Can I use both together?

Yes, and you should at scale. Token bucket at the edge for tenant-level fairness and abuse prevention. Queue-based admission inside the model server for GPU-aware scheduling and priority handling. This is the pattern most production stacks converge on.

How do I size my token bucket refill rate?

Measure your GPU's sustained tokens/sec under realistic load (not benchmarks), multiply by 0.7 for headroom, and multiply by the number of GPUs. Set capacity to 4x refill rate for burst tolerance. Re-measure quarterly; model changes and prompt drift will move this number.

What happens if the queue is unbounded?

You'll OOM the GPU node. KV cache fills, the inference engine starts swapping or preempting aggressively, latency goes to the moon, and eventually the process dies. Always bound your queue and reject beyond the bound. A fast rejection is better than a slow failure.

Is SJF always the right queue discipline?

No. SJF optimizes for average latency but starves long requests. If your workload has equal-length requests, FIFO is fine. If you have strict fairness requirements across tenants, priority queues with aging are better. If you have a mix, SJF with aging is the pragmatic default.

What about token bucket vs queue based admission control LLM when latency SLAs are strict?

Queue-based wins. Token bucket rejects outright, which means clients see 429s even when the system has capacity. Queue-based admits temporarily and drains in order, giving you a smoother latency distribution. For strict SLAs, combine queue-based admission with strict priority tiers.

Conclusion

Token bucket vs queue based admission control LLM isn't a religious war. It's a question of what problem you're actually solving.

Token bucket solves fairness and abuse. Queue-based admission solves GPU utilization and latency shaping. Most production systems need both, and the teams that skip the queue spend 30-40% more on GPUs than they should because they can't push utilization safely.

Start with a token bucket. Measure. When you cross 60% GPU utilization, add the queue. Tune aging and priority weights with real traffic, not benchmarks. And don't trust Kubernetes defaults to solve any of this for you — they won't.

The layer between your load balancer and your GPU is where the cost savings live. Few teams invest there. That's the opportunity.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.