Your load test looks fine at 40 concurrent requests. At 80 it looks fine. At 120, output tokens/sec drops below what you measured at 80, the queue is not empty, and p99 time-to-first-token goes from 400 ms to 9 seconds. GPU utilization still reads 95%. Nothing crashed.
What you are watching is KV cache preemption: vLLM admitted more sequences than it can hold KV blocks for, ran out mid-decode, and started evicting in-flight requests to make room. The evicted ones go back to the queue and get re-run from scratch. At that point the GPU is busy doing the same work twice, and adding load makes it worse, not better.
TL;DR
- KV cache preemption happens when vLLM's scheduler admits a request based on the KV blocks it needs right now, then runs out of blocks as that request decodes and its KV grows one block every ~16 tokens.
- vLLM's response is to evict running sequences (default:
RECOMPUTE) and push them to the front of the waiting queue. They get re-admitted, hit the same wall, and thrash. - The trigger is almost always
max_num_seqsset far above the KV capacity at your p95 total sequence length — not a lack of GPU FLOPs. - Size admission from KV bytes:
kv_bytes_per_token = 2 × layers × kv_heads × head_dim × dtype_bytes. For Llama-3.1-8B in fp16 that is 128 KiB/token, so ~50 GiB of KV holds ~400k tokens — about 68 concurrent requests at 6k tokens each, not the default 256. - Watch
vllm:num_preemptions_totalandvllm:gpu_cache_usage_perc. Any sustained nonzero preemption rate means you are paying for tokens twice.
What is KV cache preemption in vLLM?
KV cache preemption is vLLM evicting a running sequence's KV blocks because the block pool is exhausted, then restarting that sequence later. PagedAttention stores KV in fixed-size blocks (typically 16 tokens per block per layer). A request holds ceil(len / 16) blocks and acquires a new one every 16 decoded tokens.
The scheduler is greedy. It admits a waiting request if there are enough free blocks to run its prefill plus a small reserve (roughly 1% of the pool, the watermark). It does not know, and does not try to predict, how many more blocks that request will need over its lifetime. Neither do you — max_tokens is an upper bound, not an estimate.
So the failure is structural: admission control with no length model. Every running sequence is a slowly growing claim on a fixed pool, and the scheduler keeps signing new claims until the pool is empty. When the pool empties, something has to give, and the thing that gives is a request that already did work.
Why does throughput collapse instead of plateauing?
Because preemption is positively fed back into the queue. In vLLM's scheduler, preempted sequence groups go to the front of the waiting queue so they are retried first, and preemption picks victims from the end of the running batch. Under sustained overload the same population of long-running requests gets evicted, re-admitted, and evicted again.
Each cycle burns prefill FLOPs on tokens that were already generated. Recompute itself is not catastrophic — prefill is compute-bound and processes thousands of tokens per forward pass, while the decode steps it replaces were memory-bound and did one token each. Re-prefilling 2,000 tokens is cheap next to generating them. The damage is elsewhere:
- Duplicated work scales with generated length. A request preempted at token 3,000 re-prefills 3,000 tokens. Preempted twice, 6,000.
- The re-prefill competes with decode. With chunked prefill on, those recompute chunks share the batch budget with everyone else's decode steps, so inter-token latency spikes across the board.
- Latency variance explodes. A preempted request's clock keeps running. Its TTFT is already spent; now it pays queue time again mid-stream, which streaming clients see as a multi-second stall.
- Goodput diverges from throughput. Your tokens/sec counter may look acceptable while a growing share of those tokens are recomputes of tokens you already paid for.
Reasoning-style workloads make all of this much worse. Long chain-of-thought outputs have a heavy-tailed length distribution, so a handful of requests quietly grow to 8k–32k tokens of KV while the scheduler keeps admitting short ones on top of them.
How much KV cache does one request actually need?
Compute it; do not guess. Per token, per layer, you store one K and one V vector of size kv_heads × head_dim:
DTYPE_BYTES = {"fp16": 2, "bf16": 2, "fp8": 1}
def kv_bytes_per_token(layers, kv_heads, head_dim, dtype="fp16"):
# 2 = K and V
return 2 * layers * kv_heads * head_dim * DTYPE_BYTES[dtype]
def max_concurrent(kv_gib, seq_len, **model):
per_token = kv_bytes_per_token(**model)
total_tokens = int(kv_gib * 1024**3 // per_token)
return total_tokens, total_tokens // seq_len
# Llama-3.1-8B: 32 layers, 8 KV heads (GQA), head_dim 128
tokens, seqs = max_concurrent(50, 6000, layers=32, kv_heads=8, head_dim=128)
print(kv_bytes_per_token(32, 8, 128) // 1024, "KiB/token") # 128 KiB/token
print(tokens, "tokens ->", seqs, "concurrent @ 6k") # ~409600 -> 68
On an 80 GB H100 at gpu_memory_utilization=0.90, an 8B model in bf16 takes ~16 GB of weights, activations and CUDA graphs take a few more, and you are left with roughly 50 GiB of KV pool. That is ~400k tokens.
If your p95 request is 2k prompt + 4k output = 6k tokens, the honest concurrency limit is 68. vLLM's default max_num_seqs (256 in many builds, higher in some V1 configurations) over-admits by nearly 4×. Everything is fine until enough of those 256 sequences have grown long simultaneously — which is exactly what happens when load rises and the batch stops draining.
Note how the GQA config dominates. Llama-3.1-70B is 80 layers × 8 KV heads × 128 = 320 KiB/token, 2.5× the 8B model, on a card that also has to hold 140 GB of weights across the tensor-parallel group. A model with full multi-head attention at the same size would be 8× worse.
How do I size max_num_seqs so preemption never happens?
Pick the concurrency your KV pool can hold at p95 length, set max_num_seqs to it, and push the overflow outside the engine where you can see it and shed it.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 8192 \ # bound the worst case; reject, don't thrash
--max-num-seqs 64 \ # from the capacity math above, not the default
--gpu-memory-utilization 0.92 \
--enable-prefix-caching \
--kv-cache-dtype fp8 \ # 2x the KV pool; validate quality first
--max-num-batched-tokens 8192
Four things are doing work here:
--max-num-seqs 64 is the real fix. A smaller running batch means each admitted request has room to grow to its full length. Counter-intuitively this usually raises sustained throughput, because decode at batch 64 is already deep into the memory-bandwidth-bound regime — batch 256 was not buying you 4× the tokens, it was buying you preemptions.
--max-model-len 8192 bounds the pathological tail. A request that needs more gets a clean 400 instead of squatting on 200 blocks.
--kv-cache-dtype fp8 halves KV bytes per token, doubling concurrency at the same memory. This is a real accuracy trade-off — measure it on your eval set before shipping.
--enable-prefix-caching does not prevent preemption, but it softens recompute: shared prompt prefixes and, depending on version and eviction pressure, some of the preempted sequence's own blocks may still be resident, so the re-prefill hits cached blocks instead of recomputing them.
Then cap concurrency at the gateway with a semaphore sized to the same number. Queueing in your own service is strictly better than queueing inside the engine: you can return 429, you can prioritize, and you can autoscale on queue depth.
Should I use swap or recompute?
Recompute, in almost all cases — which is why it is the default for single-sequence requests.
PreemptionMode.SWAP copies the victim's KV blocks to pinned CPU memory and copies them back on resume. That traffic crosses PCIe at tens of GB/s against HBM at roughly 3 TB/s, and it burns bandwidth in both directions while the rest of the batch is trying to decode. Recompute re-runs a prefill, which is the operation your GPU is best at. Swap only pays off for very long generated suffixes where the recompute cost genuinely exceeds the transfer — and it also pins host memory you may want for something else. Both modes are symptoms; neither is a fix.
How do I detect KV cache preemption in production?
Scrape /metrics and alert on the preemption counter directly. It should be flat at zero.
- alert: VLLMKVCachePreemption
expr: rate(vllm:num_preemptions_total[5m]) > 0
for: 10m
annotations:
summary: "vLLM is preempting requests — max_num_seqs exceeds KV capacity"
- alert: VLLMKVCacheNearFull
expr: avg_over_time(vllm:gpu_cache_usage_perc[5m]) > 0.85
for: 15m
annotations:
summary: "KV pool >85% — preemption is imminent under any load spike"
vllm:gpu_cache_usage_perc is the leading indicator; the preemption counter is the confirmation. Pair them with vllm:num_requests_waiting and per-request inter-token latency. The diagnostic signature is unmistakable once you know it: cache usage pinned near 1.0, preemptions climbing, waiting queue nonzero, and tokens/sec decreasing as you add load. Older vLLM builds also log a warning naming PreemptionMode.RECOMPUTE and "not enough KV cache space" — grep your logs for it; the V1 engine reports it through the metrics path rather than that exact string, so do not rely on the log line alone.
One more habit worth building: track goodput (tokens delivered to clients per second) separately from engine throughput. Recomputed tokens inflate the second and not the first, and the gap between them is your preemption tax in one number.
Direct answer
vLLM throughput collapses under load because of KV cache preemption: the scheduler admits requests based on the KV blocks they need at admission time, but each decoding sequence claims a new block every ~16 tokens, so a batch that fit at admission stops fitting mid-flight. vLLM then evicts running sequences, discards their KV, and re-queues them at the front — so the GPU re-prefills tokens it already generated, and under sustained load the same requests are evicted repeatedly. It is an admission-control bug, not a compute shortage. Compute 2 × layers × kv_heads × head_dim × dtype_bytes per token, divide your KV pool by your p95 sequence length, set max_num_seqs to that number, bound max_model_len, queue the overflow at your gateway, and alert on vllm:num_preemptions_total staying at zero.
Top comments (0)