DEV Community

Cover image for vLLM 0.34 Stops Reserving 8% of Your GPU: Extensible KV Cache, the Block Size 64 Default, and What to Delete From Your Launch Flags
AI Tech News
AI Tech News

Posted on

vLLM 0.34 Stops Reserving 8% of Your GPU: Extensible KV Cache, the Block Size 64 Default, and What to Delete From Your Launch Flags

TL;DR

vLLM v0.34.0 (GitHub release 2026-11-16) changes three defaults that touch every vllm serve command line: the KV cache is sized from memory measured after warmup and CUDA graph capture, gpu_memory_utilization defaults to 1.0 when you do not set it, and --block-size defaults to 64 instead of 16. I read the release notes and the four pull requests behind them (#56492, #63231, #66704, #67071), checked metric names against the v0.34.0 source, and ran a small simulation of my own for the block-size effect. I did not run vLLM on a GPU for this article.

  • The old 0.92 default was a guess buffer. It left 8% of the device idle, about 15 GiB on a 192 GiB GPU, to absorb estimation error. The PR measured 4 to 10% more KV cache tokens across eight models on GB200 when that buffer is removed.
  • Your old flag now costs you. With the new sizing, an explicit --gpu-memory-utilization 0.92 gives a cache 0.6 to 3.7% smaller than the old engine produced, because the new code subtracts the graph and warmup memory it actually measured. Carrying 0.9 forward is the most likely way to lose capacity on this upgrade.
  • Startup is faster. Engine init without weight loading fell 21 to 34% in the PR's runs, because the throwaway graph-capture pass used for estimation is gone.
  • Block size 64 is a bookkeeping choice, not a measured win. The PR itself says a 16-versus-64 serving benchmark was still missing. My model says short contexts and short shared prefixes pay for it, long ones barely notice.
  • Two release-note warnings matter operationally: decode latency per token can rise 6 to 18% on some H200 models at concurrency 128, and GPU memory dashboards will now sit near full by design.

What does vLLM 0.34 actually change about KV cache sizing?

Until this release, vLLM sized the KV cache before CUDA graph capture. The reason is mechanical: captured graphs bake in tensor addresses, so the cache has to exist first. That forced a profiling run, an estimate of everything allocated later (graph memory, attention workspaces, sampler and logprobs buffers, speculative drafts), and a safety slack. The slack was gpu_memory_utilization=0.92.

PR #56492, merged 2026-11-02, replaces the estimate with a measurement. Per its description:

  1. The full cache capacity is reserved as virtual address space, with no physical memory behind it, using the driver's virtual memory API (CUDA cuMem*, HIP hipMem*).
  2. Only the blocks that capture and warmup touch get physical pages.
  3. After warmup, each worker measures what is really free and the transient peaks it really saw, then the engine commits the final block count in one step by mapping more physical memory into the same reserved range.

Addresses never move, so the layer views and the already-captured graphs stay valid. The sizing rule is: budget, minus the measured transient peak (the larger of the profiling peak and the warmup peak), minus a margin of max(1 GiB, 0.75 x transient peak). The engine takes the minimum across workers and across data-parallel engines, since they step in lockstep. Warmup also gained a full-budget prefill step and a prompt-logprobs chunk so those peaks are measured instead of guessed.

The release notes list it as on by default for CUDA and ROCm, with the V2 model runner, which is now the only GPU runner. To restore the old behavior you pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.

How much more KV cache do you get, and when do you get less?

The PR's table is the best data available because it uses the same hardware and process for every row: GB200 with 189 GiB per GPU, one unmeasured warm-up start per variant, then three interleaved off/on pairs, medians reported.

Model TP KV cache, old sizing KV cache, new default Token gain New code at explicit 0.92
Qwen3-8B 1 150.2 GiB 161.6 GiB +7.5% -0.6%
Qwen3-32B 1 103.1 GiB 113.0 GiB +9.5% -1.9%
gpt-oss-20b 1 150.5 GiB 161.0 GiB +7.0% -0.9%
Qwen3.6-35B-A3B 1 97.6 GiB 106.2 GiB +8.8% -2.3%
Qwen3-235B-A22B-FP8 4 108.1 GiB 117.1 GiB +8.2% -1.5%
DeepSeek-V4-Flash 4 119.9 GiB 124.9 GiB +4.1% -3.7%

KV cache change per model, default versus explicit 0.92

Two things stand out. First, the gain comes from the removed slack: 8% of 189 GiB is 15.1 GiB, and the measured differences are 4.9 to 11.3 GiB because the new code now pays for graph and warmup memory out of that same pool. DeepSeek-V4-Flash gains least because its graphs and workspaces need about 15 GiB per GPU after warmup, so the old slack was mostly in use already. Second, the right-hand column is the trap. Anyone who keeps an explicit 0.92 gets a smaller cache than before.

The follow-up PR #66704 (merged 2026-11-12) puts a number on it with a small run, Qwen3.5-0.8B on GB200 with max_model_len=4096: explicit 0.9 gave 23,920 blocks (157.68 GiB) and unset gave 26,200 blocks (172.71 GiB). The startup warning says the cap "keeps 15.03 GiB of free device memory out of the KV cache", and 15.03 GiB is exactly the difference between those two caches. That warning fires when an explicit value above 0.8 and below 1.0 withholds at least 1 GiB.

The release notes quote a bigger headline: MiniMax-M3 on H200 going from 433k to 672k cache tokens. That is +55%, far above the GB200 table. I would not expect it on your model. It comes from a different GPU and model, and the notes do not break it down, so the likely explanation (a much larger share of the device was sitting in the old slack or estimate error) is my guess.

What does a 7 to 10% cache gain mean in sequences?

A quick planning calculation. Take a model with 256 KiB of KV per token (64 layers x 8 KV heads x 128 head dim x 2 for K and V x 2 bytes in BF16 works out to exactly that, but substitute your own number). With the 103.1 GiB to 113.0 GiB change from the table:

def kv_tokens(cache_gib: float, kib_per_token: float) -> int:
    return int(cache_gib * 2**30 / (kib_per_token * 1024))

for label, gib in (("old sizing", 103.1), ("new default", 113.0)):
    t = kv_tokens(gib, 256)
    print(f"{label}: {t:,} tokens, ~{t // 4096} sequences at 4k context")
# old sizing: 422,297 tokens, ~103 sequences at 4k context
# new default: 462,848 tokens, ~113 sequences at 4k context
Enter fullscreen mode Exit fullscreen mode

Ten more concurrent 4k sequences per GPU for deleting a flag. Across a fleet that is a small but free capacity bump, not a re-plan.

Why can per-token latency rise with a bigger cache?

The release notes state the trade-off directly: "per-token decode latency can rise on some models (TPOT +6 to 18% on some H200 models at concurrency 128)". They do not state the mechanism, so what follows is my reading and not vLLM's explanation. A larger cache lets the scheduler keep more sequences running at once. A bigger running set means a bigger decode batch, and every decode step does more work. Throughput goes up and TTFT goes down under high concurrency because fewer requests wait, while each token takes slightly longer.

If you run a hard TPOT objective, the knob that bounds that effect directly is --max-num-seqs, which caps the running set no matter how large the cache is. That is my suggestion and I have not tested it against 0.34, so benchmark it with your own traffic before trusting it. The older habit of lowering gpu_memory_utilization to protect latency is now the wrong lever: it shrinks the cache and also hides free memory, and the startup warning will tell you so.

Does faster startup matter, and how much faster is it?

The same PR measured startup. "Engine init" is the logged time for profile, create KV cache and warm up, which excludes weight loading.

Model TP Engine init, old Engine init, new End to end, old End to end, new
Qwen3-8B 1 17.7 s 12.5 s 50.0 s 40.9 s
Qwen3-32B 1 30.7 s 21.6 s 80.9 s 71.4 s
Qwen3-235B-A22B-FP8 4 76.1 s 50.1 s 173.8 s 144.2 s
DeepSeek-V4-Flash 4 112.2 s 77.9 s 192.2 s 174.2 s

Engine init seconds, old sizing versus extensible KV cache

Most of the saving is the throwaway capture pass that the old sizing used to estimate graph memory; the real capture takes the same time. Two caveats from the PR: weights loaded from a warm page cache, so a cold start has a bigger fixed cost and a smaller percentage gain, and the TP4 rows used lazy checkpoint loading, so MiniMax-M2.7's end-to-end comparison mixes two loading modes. For autoscaling, the practical reading is 5 to 34 seconds off engine init per replica, which matters for scale-out latency more than for steady state.

What changes when --block-size goes from 16 to 64?

PR #63231, merged 2026-11-05, changes the default through CacheConfig.DEFAULT_BLOCK_SIZE. An explicit --block-size still wins, and backends with a preferred block size still override the default. ROCm paged attention keeps 16, because its native kernel supports only 16 and 32 and anything else falls back to a slower Triton path.

The stated motivation is bookkeeping: fewer block-table entries, less allocator metadata, fewer prefix-cache hash operations per token. The PR text also contains a sentence worth quoting: "Still missing: a 16-vs-64 throughput/memory benchmark on a real serving workload." I did not find one in the release notes either, so treat the default as a reasonable engineering bet and measure it on your workload.

Its first CI run is also instructive. It failed where 16-token assumptions were baked in: offloading-connector chunk sizes, shared-storage connector hash-count expectations, and a 25-minute hang in the ROCm attention backend. All were fixed, but if you maintain an out-of-tree KV connector or a custom cache test, grep for the number 16.

What does block size 64 cost in wasted slots and prefix hits?

Two mechanical effects follow from the size of a block, independent of vLLM's implementation. A sequence occupies whole blocks, so the unused tail of its last block is stranded. And prefix caching reuses whole blocks, so a shared prefix is only reusable up to the last full block boundary. I modeled both. This is illustrative arithmetic, not a measurement of vLLM.

import math, random
random.seed(7)

def unused_share(median, sigma, block, n=200_000):
    used = alloc = 0
    for _ in range(n):
        length = max(1, int(random.lognormvariate(math.log(median), sigma)))
        blocks = -(-length // block)
        alloc += blocks * block
        used += length
    return 100 * (alloc - used) / alloc

for med in (200, 1000, 4000, 16000):
    print(med, round(unused_share(med, 0.8, 16), 2), round(unused_share(med, 0.8, 64), 2))
Enter fullscreen mode Exit fullscreen mode
Median context (tokens) Unused slots, block 16 Unused slots, block 64
200 2.66% 10.24%
1,000 0.55% 2.24%
4,000 0.14% 0.57%
16,000 0.03% 0.14%

Unused KV slots in the last block, block size 16 versus 64

The expected stranded tail is about (block - 1) / 2 tokens per sequence, so 7.5 tokens at 16 and 31.5 at 64. That is invisible at 4k contexts and about a tenth of the cache at 200-token chat turns. Prefix reuse is a sharper edge: a 40-token system prompt gets 32 cached tokens at block size 16 and zero at 64, a 100-token prompt gets 96 versus 64, and a 1,000-token prefix gets 992 versus 960. If your traffic is many short requests behind a short shared prompt, this is where block 64 can hurt, and where --block-size 16 is worth testing.

To measure it, v0.34.0 exports vllm:prefix_cache_queries and vllm:prefix_cache_hits, both counted in tokens (Prometheus appends _total to counters):

rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])
Enter fullscreen mode Exit fullscreen mode

Replay the same traffic against block 16 and block 64 and compare this ratio together with vllm:kv_cache_usage_perc and vllm:num_preemptions.

Which launch flags should you delete, and which should you keep?

PR #67071 (merged 2026-11-12) cleaned explicit utilization values out of 89 recipe files, 14 examples and several CI configs, and added a docs section on when to set the option. It lists these legitimate reasons to keep an explicit value:

  • other processes share the GPU, including ones that allocate later;
  • unified-memory devices such as DGX Spark;
  • weight-transfer buffers for RL-style weight updates;
  • KV connectors with their own device buffers.

Everything else is a habit from estimate-based sizing. A starting checklist:

# before (0.33 habits)
vllm serve $MODEL --gpu-memory-utilization 0.92 --block-size 16

# after, dedicated GPU, nothing else allocating later
vllm serve $MODEL
# only if you measured a prefix-hit loss at block 64:
vllm serve $MODEL --block-size 16
# only if you need the old sizing back during rollout:
vllm serve $MODEL --no-enable-extensible-kv-cache
Enter fullscreen mode Exit fullscreen mode

Other items to check before rolling out:

  • Co-located engines. The new cache takes what is free after warmup. Two engines on one GPU need explicit fractions, as before. Values at or below 0.8 do not trigger the warning.
  • Late allocators. A sidecar or a second job that grabs memory after warmup now competes with a cache sized to nearly the whole device, minus the measured margin.
  • Memory alerts. A rule like "GPU memory above 95%" will fire on healthy replicas. Alert on vllm:kv_cache_usage_perc and on preemptions instead.
  • KV connectors. Support moved during the cycle. The release notes list fixes for NIXL same-host NVLink, LMCache and out-of-tree connectors with the feature, while the docs PR still names NIXL, Mooncake and MoRIIO prefill/decode setups among cases where the extensible cache stays off. I did not resolve which applies to your connector version, so confirm on a canary by comparing the reported KV cache size at startup with and without --no-enable-extensible-kv-cache.
  • Fixed-size overrides. Setting kv_cache_memory_bytes or num_gpu_blocks_override turns the feature off, and requesting it explicitly with them is an error.
  • Other 0.34 breakers in the same table. Model Runner V1 is removed (VLLM_USE_V2_MODEL_RUNNER=0 now fails at startup), huggingface_hub >= 2.0.0 is required, ROCm 7.2 images are gone, and Marlin W4A8 and QuTLASS fp_quant are removed.

FAQ

Is the extensible KV cache safe to leave on?

The PR validated greedy outputs matching with the feature on and off, and before and after sleep and wake, and the release notes list a string of follow-up fixes for allocator fragmentation and startup out-of-memory cases (#67296, #67485 and others). Both facts argue for a canary rollout rather than a switch-over, because the fixes are recent.

What does gpu_memory_utilization=1.0 mean now?

With the feature on and the option unset, the engine treats the whole device as the budget and keeps a measured reserve for transients plus the margin described above. An explicit value is a hard cap on everything the engine uses, so a value just under 1.0 leaves free memory unused. The startup warning exists to say exactly how much.

Do I need to change my Kubernetes memory requests or GPU sharing?

If each pod owns a whole GPU, no. If you slice or share GPUs, keep explicit per-engine fractions, because an engine left at the default will size to everything that is free when warmup finishes.

Should I keep block size 16?

Only if your traffic is dominated by short requests behind a short shared prefix and your measured prefix hit ratio drops at 64. The default is a bookkeeping optimization that the PR does not benchmark, so the evidence for either side is yours to collect.

Where do the numbers come from?

Capacity and startup tables are the PR author's GB200 measurements in vllm-project/vllm #56492, with medians of three interleaved pairs. The 15.03 GiB example is from #66704. The block-size tables are my own illustrative simulation, and the planning snippet uses an assumed 256 KiB per token.

Links

Further reading

Top comments (0)