DEV Community

Cover image for KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
AI Tech News
AI Tech News

Posted on

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

#vllm #ai #machinelearning #llm

TL;DR

  • Loading a cached KV prefix from NVMe only beats recomputing it when the drive can deliver more bytes per second than the break-even bandwidth for that prompt length, model, and GPU.
  • In a recent measurement study (arXiv 2609.11744, Llama 3.2 3B on an H100 node), the break-even bandwidth fell from 23.2 GB/s at 1k tokens to 10.4 GB/s at 8k and 3.5 GB/s at 80k tokens. A drive that sustains 13.5 GB/s clears the bar from 8k up and misses it at 1k.
  • By a first-order model, bigger models are easier to offload: KV bytes per token grow slowly with model size, while prefill FLOPs per token grow with parameter count. A 70B-class model needs about 1 GB/s in the illustrative calculation below, and a 3B model needs several times that.
  • The same study found a trace replay on an H100 where the average request sat below the SSD break-even point (6,203 tokens for Qwen3 4B) and external caching gave no TTFT gain over GPU prefix caching alone.
  • Treat external KV caching as an admission decision made per prefix, not a switch you turn on for a cluster.

KV offload break-even bandwidth versus prompt length

What question does a KV offload tier actually answer?

Prefix caching stores the key and value tensors computed for a prompt prefix so a later request with the same prefix can skip that part of the prefill. When the GPU cache is too small to hold every prefix you care about, an offload tier (CPU DRAM, then NVMe) keeps more of them. The question is never "is a bigger cache better". It is whether, for one specific hit, this inequality holds:

t_fixed + (prefix_tokens * kv_bytes_per_token) / effective_bandwidth  <  t_prefill(prefix_tokens)
Enter fullscreen mode Exit fullscreen mode

The left side is the cost of a hit: a fixed scheduling and copy overhead plus the transfer. The right side is what the GPU would spend recomputing the same prefix. If the left side is larger, the cache made the request slower. The py-kvcache study states it directly: for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache.

Two terms in that inequality scale differently. KV bytes grow linearly with prefix length. Prefill time grows faster than linearly, because attention cost adds a term quadratic in length on top of the linear matrix multiplications. So longer prefixes are better candidates for offload, and the bandwidth you need goes down as prompts get longer.

How much bandwidth does break-even need?

The py-kvcache authors computed the bandwidth a cache path must sustain for a fully cached prompt to tie with recomputation. For Llama 3.2 3B, which stores 114,688 bytes of KV per token in BF16, on their H100 node (arXiv 2609.11744, section 5):

Prompt length Break-even bandwidth
1k tokens 23.2 GB/s
8k tokens 10.4 GB/s
80k tokens 3.5 GB/s

Measured break-even bandwidth at 1k, 8k and 80k tokens

Compare that with what the hardware delivered in the same study. A GPU to CPU copy sustained 54 to 56 GB/s on the PCIe 5.0 node, so a DRAM tier clears the requirement at every length tested. The llm-d disk path moved 9.0 GB/s at 1k tokens and 11.2 to 12.0 GB/s at larger sizes, which leaves it below the requirement at 1k and 2k tokens and above it from 8k onward. Their fio reference was 13.5 GB/s for the PCIe 5.0 drive and 6 GB/s for a PCIe 4.0 drive.

Two consequences follow. Storage bandwidth sets a minimum useful prompt length: on that setup, an 8k prompt needed 77.8% of its tokens to be a cached prefix to break even, while at 80k tokens a prefix of only 7.8% was enough. And a PCIe 4.0 drive at 6 GB/s fails the bar for this model until prompts are well past 8k tokens, so the "NVMe offload" label hides a factor of two between drive generations.

Why do larger models make offload easier?

Take the prefill cost per token for a dense model as about 2 * P FLOPs, where P is the parameter count. KV bytes per token are layers * 2 * kv_heads * head_dim * dtype_bytes. Dividing the two gives the bandwidth needed to keep pace with the GPU at long context, ignoring attention and overhead:

break_even_bandwidth  ~=  kv_bytes_per_token * effective_FLOPs / (2 * P)
Enter fullscreen mode Exit fullscreen mode

Grouped-query attention keeps kv_heads at 8 for many current open models, so KV bytes per token barely move between a 3B and a 70B model, while P grows more than twentyfold. I read the configs for three open models from their Hugging Face repositories and took the Llama 3.2 3B value from the paper:

KV cache KiB per token for four models

Model Layers KV heads Head dim KV bytes per token
Llama 3.2 3B 28 8 128 114,688 (112 KiB)
Qwen3 4B Instruct 2507 36 8 128 147,456 (144 KiB)
Qwen3 32B 64 8 128 262,144 (256 KiB)
Llama 3.3 70B Instruct 80 8 128 327,680 (320 KiB)

Here is the calculation I used, which you can adapt to your own GPU and model:

FL = 989e12 * 0.50   # H100 SXM BF16 dense peak x 0.50 achievable (illustrative)

def t_prefill(P, layers, q_heads, head_dim, n_tokens):
    matmul = 2 * P * n_tokens
    attn = layers * 2 * n_tokens**2 * q_heads * head_dim   # causal QK^T and AV
    return (matmul + attn) / FL

def required_bw(kv_bytes_per_token, P, layers, q_heads, head_dim, n_tokens):
    return kv_bytes_per_token * n_tokens / t_prefill(P, layers, q_heads, head_dim, n_tokens)

# Llama 3.3 70B at 32,768 tokens
print(required_bw(327_680, 70.55e9, 80, 64, 128, 32_768) / 1e9)   # about 0.88 GB/s
Enter fullscreen mode Exit fullscreen mode

The results of that model, for prefixes of 8k and 32k tokens:

Model 8k tokens 32k tokens
Llama 3.2 3B 7.24 GB/s 4.70 GB/s
Qwen3 4B 6.97 GB/s 4.12 GB/s
Qwen3 32B 1.75 GB/s 1.30 GB/s
Llama 3.3 70B 1.07 GB/s 0.88 GB/s

Illustrative lower bound on break-even bandwidth by model

How to read this honestly: these are illustrative numbers from the stated assumptions (H100 SXM class compute at 50% utilization, no fixed overhead), not measurements. They are a lower bound. For Llama 3.2 3B the model gives 7.24 GB/s at 8k and 2.76 GB/s at 80k, against the measured 10.4 and 3.5 GB/s. The gap is the fixed per-hit overhead that the formula ignores, and it matters most for short prompts. The paper also ran on a different H100 variant than the one my constants describe, so do not compare absolute values too closely. What carries over is the shape: the requirement falls with prompt length and falls sharply with model size.

Two practical readings. A 70B-class model has plenty of headroom even on a PCIe 4.0 drive (about 1 GB/s needed against 6 GB/s available), so offload economics look much better for large dense models. And anything that shrinks KV bytes per token lowers the requirement in proportion: an FP8 KV cache halves it relative to BF16, which ties into the earlier post on KV cache quantization linked below. With tensor parallelism, KV bytes and FLOPs both shard across GPUs, so to first order the per-GPU requirement is unchanged, provided each GPU has its own path to storage.

Does the spec sheet bandwidth match what you actually get?

Not by default. The py-kvcache study found several places where the delivered number fell away from the device number:

  • Data amplification. In the llm-d connector, a request holding 4.4 GB of useful KV data caused roughly 10 GB to be copied and read or written, because of a minimum staging allocation. The authors reported it upstream and it was fixed in llm-d pull request 589. After removing the minimum allocation, TTFT fell about 50% below LMCache in that test.
  • Asymmetric reads and writes. The connector's reads reached more than 12 GB/s while writes stayed around 2 GB/s. A workload that writes many new prefixes will see a different picture than one that mostly reads.
  • Request size. Block tracing showed the filesystem issuing mostly 512 KiB requests. In a standalone benchmark, I/O engines differed widely at 4 KiB, but at 1 MiB with one thread every engine clustered near the 6 GB/s that fio measured on that drive. Swapping the I/O engine did not improve TTFT. KV blocks are far larger than 1 MiB (one 256-token block of Llama 3.2 3B is about 28 MiB), so the workload is limited by bandwidth and transfer placement, not by small-I/O rate.
  • Scheduling. Where a transfer enters the request schedule matters. The authors' py-kvcache connector starts disk reads while a request is still queued, and preloading contributed a 1.34x improvement at 80k tokens.

For capacity planning, measure end-to-end TTFT with your connector on your drives and treat the fio number as an upper bound.

When does external KV caching not pay off?

The most useful result in the study is a negative one. The authors replayed three production-style traces from Alibaba Cloud Bailian (a coder trace, an interactive trace, and an API-driven trace) on two machines.

  • On the smaller GPU (an RTX 4000 Ada with 20 GiB), py-kvcache was 1.12x to 1.79x faster than GPU prefix caching alone and 1.10x to 1.25x faster than LMCache.
  • On the H100 node, the average request length was below the measured 6,203-token SSD break-even point for Qwen3 4B, and the larger GPU memory kept a large share of the working set resident. TTFT with GPU prefix caching alone was identical to py-kvcache, and vLLM's native offload implementation was 0.2 s slower.

The same traces flipped from "offload helps" to "offload costs you" depending on the GPU. The authors conclude that external KV caching is a setup-specific admission decision, and that recomputation is a valid fallback rather than a cache miss.

Where it does pay, the numbers are large. At 80k tokens, loading from disk with py-kvcache was 2.0x faster than LMCache. With GPU, CPU, and disk tiers all enabled it was 1.23x faster than LMCache and within about 4% of vLLM's native KV offload implementation.

How do you configure KV offload in vLLM, and how do you gate it?

In the vLLM v0.32.0 source (released October 18, 2026), CacheConfig exposes two offload options. kv_offloading_size is the buffer size in GiB (summed across tensor-parallel ranks), and offloading is only active when it is set. kv_offloading_backend accepts native or lmcache, and the docstring describes the offload target as CPU. If you want an NVMe tier, the study's comparisons use LMCache's disk support and an external connector, so check what your chosen connector supports.

# CPU DRAM tier with the native backend (GiB, summed across TP ranks)
vllm serve <model> --kv-offloading-size 64 --kv-offloading-backend native
Enter fullscreen mode Exit fullscreen mode

The admission rule is separate from the tier. The py-kvcache design runs an offline calibration once per node and model, writes a small file with the break-even prefix length per source tier, and at lookup time declines a load that is predicted to cost more than recomputation. You can approximate the same policy:

def should_load(prefix_tokens, kv_bytes_per_token, measured_bw_Bps,
                t_recompute_s, t_fixed_s):
    t_load = t_fixed_s + prefix_tokens * kv_bytes_per_token / measured_bw_Bps
    return t_load < t_recompute_s
Enter fullscreen mode Exit fullscreen mode

Fill the inputs from your own sweep, not from a data sheet:

  1. Send prompts at several lengths (for example 1k, 4k, 8k, 16k, 32k, 64k) with all caches cold and record TTFT. That is t_recompute_s.
  2. Send the same prompts again with the prefix resident only in the offload tier, and record TTFT. Subtract the non-transfer part to get the transfer time and the effective bandwidth.
  3. Repeat with prefix fractions of 10%, 50%, and 90% of the prompt, since the break-even frontier depends on both length and fraction.
  4. Store the break-even prefix length per tier and re-run the sweep whenever the GPU, model, KV dtype, or drive changes.

One caveat from the same paper: a static threshold on prefix length and bandwidth cannot tell a demand read from a preload that queueing may hide, so a stronger policy would also track staging capacity and per-tier transfer rates.

A separate benchmark, PrefixBench-H100 (arXiv 2609.19657), reports that cache effectiveness is largely insensitive to concurrency and output length on a single H100. That supports calibrating the break-even point offline.

FAQ

Is NVMe KV cache offloading faster than recomputing the prefill?
Only above a break-even bandwidth that depends on prompt length, model, and GPU. In the measured study, a 13.5 GB/s drive beat recomputation for Llama 3.2 3B from 8k tokens up, but a 1k-token prompt needed 23.2 GB/s.

How do I calculate the break-even bandwidth for my model?
Divide the bytes of KV for the cached prefix by the time budget left after subtracting non-transfer overhead from the recompute time. A first-order estimate is kv_bytes_per_token * effective_FLOPs / (2 * parameters). Measure the fixed overhead on your own stack, because it dominates short prompts.

Why do larger models need less offload bandwidth?
With grouped-query attention, KV bytes per token change slowly with model size, while prefill FLOPs per token scale with parameter count. More compute per byte of KV means recomputation is slower, so a slower drive is enough to beat it.

Should I enable KV offload on an H100?
It depends on your prefix length distribution. In the study's trace replay on an H100 node, the average request was below the SSD break-even point and GPU prefix caching alone matched the offload connector. Measure your own trace before committing storage hardware.

Does vLLM offload KV cache to disk by default?
No. In v0.32.0 offloading is off unless kv_offloading_size is set, and the native backend targets CPU memory per the config docstring. Disk tiers come from connectors such as LMCache or external ones.

Further reading

Sources

Top comments (0)