vLLM Preemption: Why 1 in 50 Requests Restarts From Scratch

Our chat endpoint had a mean latency of 1.9s and a p99 of 11.4s.

Same model. Same GPU. Same prompt template. nvidia-smi showed 96% utilization and no memory pressure worth mentioning. Nothing crashed. Nothing retried. But roughly one request in fifty would stream a few tokens, freeze for six seconds mid-sentence, then finish normally like nothing happened.

That freeze has a name: vLLM preemption. The scheduler evicted a request that was already 80% done, threw away every KV block it had built up, and put it back in the queue to be prefilled again from token zero. My users paid for those tokens twice. So did my GPU.

(Numbers in this post are from my own staging box, a single 48 GB card serving Llama-3.1-8B-Instruct in fp16. They are not published benchmarks, and yours will differ.)

TL;DR

• vLLM preemption happens when the KV cache pool runs out of free blocks while requests are still generating. The scheduler evicts in-flight requests to keep the batch moving.
• Default mode is RECOMPUTE: the evicted request's KV cache is discarded entirely and re-prefilled later. Work already done is gone.
• It shows up as p99 latency spikes and mid-stream stalls, never as an error. Mean latency and GPU utilization both look healthy.
• Root cause is almost always max_num_seqs (default 256) admitting far more concurrent sequences than your KV pool can hold at your real p95 output length.
• Fix: watch vllm:num_preemptions_total, cap max_num_seqs to what the pool actually sustains, and bound max_tokens per request.

What does vLLM preemption actually mean?

vLLM preemption is the scheduler kicking a running request out of the batch because there is no free KV cache block left to append its next token.

This is not admission queueing. Admission queueing is fine and fair: requests sit in the waiting queue, you see them in num_requests_waiting, first in first out. Preemption is the opposite. It punishes requests that already made it in, already burned prefill compute, already streamed tokens to a human who is watching.

Here is the mechanism, in order:

1. PagedAttention stores KV in fixed-size blocks (16 tokens each by default), like pages in virtual memory.
2. Every running sequence needs a fresh block roughly every 16 generated tokens.
3. The pool is finite. It is whatever GPU memory is left after weights and activation profiling, governed by gpu_memory_utilization (default 0.9).
4. When a running sequence needs a block and none is free, the scheduler picks a victim from the running batch and preempts it.
5. In RECOMPUTE mode (the default), the victim's blocks are freed and the request goes back to the front of the waiting queue. Its generated text is kept; its KV cache is not. To resume, vLLM must prefill the original prompt plus everything it already generated.

Step 5 is the expensive part. A request that had produced 900 tokens does not resume at token 900. It re-prefills ~900 extra tokens of context before it can emit token 901.

vLLM does tell you. The log line reads roughly like: sequence group is preempted because there is not enough KV cache space, followed by advice to raise gpu_memory_utilization or tensor parallelism. If you are running with default log levels and only alerting on 5xx, you will never see it.

Why does the KV cache run out when GPU memory looks fine?

Because KV cache pressure is a function of live tokens, not request count, and live tokens grow the whole time a request is generating.

A request holding 4,000 tokens of context occupies 4,000 tokens' worth of KV. Ten of those cost the same as forty requests at 1,000 tokens. Your load test with fixed 128-token outputs told you nothing about the request that asks for a 2,000-token summary at 3pm.

The math is simple enough to do on a napkin:

# Llama-3.1-8B-Instruct, fp16, GQA
layers, kv_heads, head_dim, dtype_bytes = 32, 8, 128, 2

bytes_per_token = 2 * layers * kv_heads * head_dim * dtype_bytes
print(bytes_per_token)            # 131072  == 128 KiB per token

kv_pool_gib = 24                  # what was left after weights on my box
max_live_tokens = kv_pool_gib * 1024**3 // bytes_per_token
print(max_live_tokens)            # 196,608   (~8,192 tokens per GiB)

# my real traffic: ~2,000 prompt tokens, p95 output ~1,500
live_per_request = 2000 + 1500
print(max_live_tokens // live_per_request)   # 56

Fifty-six. That is how many sequences my pool could hold at p95 shape.

max_num_seqs defaults to 256.

So the scheduler cheerfully admitted up to 256 concurrent sequences into a pool that supports 56 of them once they grow up. Everything looked great for the first few hundred generated tokens, then the pool hit 100% and the evictions started. And because a preempted request comes back, re-prefills, and immediately competes for blocks again, it can get preempted a second time. That is the thrash.

Why does vLLM preemption wreck p99 instead of mean latency?

Because preemption is rare and catastrophic, which is exactly the shape that hides in an average.

Ninety-eight percent of requests never get chosen as a victim and run at normal speed. The two percent that do pay: the stall while they sit in the waiting queue, plus a full re-prefill of prompt + generated-so-far, plus whatever queueing they hit on the way back in. On my box those requests landed 4-6x their normal latency. Averaged over the whole window, the mean barely moved.

GPU utilization actively lies to you here. Recompute is dense prefill work. The GPU is extremely busy doing it. Your dashboard reads "96% utilized, great throughput" while a meaningful slice of that throughput is tokens you already computed once.

The metric that does not lie is the counter:

curl -s localhost:8000/metrics | grep -E 'preemption|cache_usage|num_requests'
# vllm:num_preemptions_total{...}      <- should be flat at 0
# vllm:gpu_cache_usage_perc{...}       <- if this parks near 1.0, you are about to preempt
# vllm:num_requests_running{...}
# vllm:num_requests_waiting{...}

vllm:num_preemptions_total is a counter. Alert on its rate, not its value. Any sustained nonzero rate in production means you are paying for prefill twice.

And watch gpu_cache_usage_perc. If it sits pinned above ~0.9 during peak, you are one long request away from thrashing.

How do you stop vLLM preemption?

Cap concurrency at what the KV pool can actually sustain, and make queueing happen at the door instead of mid-generation.

The config that fixed mine:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 48 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

In priority order:

1. Lower max_num_seqs. This is the big one. Compute your sustainable concurrency from the napkin math above using your p95 live-token count, then set max_num_seqs a bit under it. Excess load waits in the admission queue, which is fair, observable, and cheap. Yes, your num_requests_waiting gauge will go up. That is the point: visible queueing beats invisible recompute.
2. Bound max_tokens per request. An unbounded max_tokens from a client is a request that can grow until it destabilizes everyone else's batch. Clamp it server-side. Most endpoints do not need 4,000 tokens of output.
3. Nudge gpu_memory_utilization up. Going from 0.90 to 0.92-0.94 buys real blocks. Do it in small steps and actually load-test, because the headroom left over is what absorbs activation spikes.
4. Keep prefix caching on. It is on by default in the V1 engine. When a preempted request comes back, freed blocks that have not been overwritten can still be hit on the recompute path, which softens the worst case. It reduces the cost of preemption; it does not prevent it.
5. Load-test with your real output-length distribution. Fixed-length synthetic load is the reason this bug ships. Replay actual production prompt and completion lengths, or you are testing a workload that does not exist.

Swapping to CPU (--swap-space) moves blocks over PCIe instead of recomputing. It trades one cost for another and it is a band-aid, not a fix. Reach for concurrency limits first.

So why does one request in fifty restart from scratch?

Because vLLM's scheduler will admit more concurrent sequences than your KV cache can hold once those sequences grow, and when the block pool runs dry it preempts a running request, discards its entire KV cache, and re-prefills it from the beginning later. The default max_num_seqs of 256 is far above what most single-GPU deployments can sustain at real output lengths, so the failure only appears under load, only affects a small fraction of requests, and never raises an error. Check vllm:num_preemptions_total. If it is climbing, cap max_num_seqs to your measured sustainable concurrency and clamp max_tokens per request. Your mean latency will barely change. Your p99 will fall off a cliff, in the good direction.

---

Written by the developer behind Preterview, an interview prep platform.