KV Cache data prefetch is currently one of the most effective means of reducing storage latency in large-model inference: measured on the Mingxin FX100 under a 480B production-grade long-context workload, the tiered prefetch strategy improves inference throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32% (measured, reports R2/R3). This conclusion stems from an engineering-level breakdown of KV Cache access patterns—the essence of prefetch is to strip storage latency out of the inference critical path, so compute units no longer wait on data.
Why KV Cache Storage Latency Becomes a Bottleneck
In the latency composition of large-model inference, KV Cache reads are taking up an increasingly large share. As context windows expand from 32K to 128K and beyond, KV Cache capacity requirements grow linearly, while GPU HBM capacity growth lags far behind the expansion rate of model parameters and context length. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), paged management of KV Cache is a key mechanism for solving HBM fragmentation, but paging itself does not solve the capacity shortfall—when KV Cache exceeds HBM capacity, it must spill to the storage tier.
The primary test platform for Mingxin's R2 measurements was 8× AMD Instinct MI308X (192 GB HBM per GPU), running Qwen3-Coder-480B-FP8 (MoE, weights ~450 GB). In the TP8 long-context deployment, KV Cache spill is inevitable: HBM must hold both weights and KV Cache, and long-context KV Cache easily reaches tens of GB. At this point, storage-tier latency enters the inference path directly—every cache miss requires reading from the NVMe array, and NVMe latency (tens of microseconds) is two orders of magnitude higher than HBM (hundreds of nanoseconds).
The Core of the Prefetch Strategy: Removing Latency from the Critical Path
The basic idea of prefetch is straightforward: bring a given KV Cache block from the storage tier into HBM or a high-speed cache before the compute unit needs it. But the engineering challenge comes down to two questions: what to prefetch (selection policy) and when to prefetch (timing policy).
Mingxin FX100 measurement data reveals the effectiveness envelope of prefetch. In R2 tests, with the 480B model under TP8 at three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. The mechanism behind this improvement: before generating the first token, the inference engine must load the full KV Cache prefix—if this data can be prefetched to the local high-speed tier in advance, the first-token wait time is significantly compressed.
R3 tests further show that at the full-machine level under TP4×2, throughput improvement is 35–36% (measured, report R3), while the conservative scenario at concurrency 8 yields 29% (measured, report R2), and the optimal operating point at concurrency 16 reaches 40% (measured, reports R2/R3). This range of variation indicates the sensitivity of the prefetch strategy to concurrency: the higher the concurrency, the more KV Cache reuse opportunities across requests, and the greater the prefetch benefit.
Prefetch vs. Recompute: Where the 8.6–20× Speedup Comes From
An alternative to the prefetch strategy is "recompute without external storage"—that is, not writing KV Cache to storage at all, but recomputing it when needed. This approach avoids storage latency but pays the cost of redundant computation. Mingxin R2 measurements compared the two paths: the no-external-storage recompute baseline had a TTFT p50 of 149.5s (concurrency 16), while the FX100 prefetch solution was only 11.85s, a 12.6× speedup; in throughput, the recompute baseline was 4.1 tok/s versus 74.9 tok/s for FX100, an 18.3× improvement. Across different concurrency levels, the speedup factor ranges from 8.6× to 20× (measured, report R2).
The engineering implication of this comparison: the cost of recompute (GPU compute occupancy) is far higher than the cost of storage reads (I/O latency). As long as the prefetch strategy hides I/O latency well, order-of-magnitude gains are achievable. According to the design analysis in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079), the KVCache-centric disaggregated architecture is based precisely on this judgment—decoupling KV Cache from GPUs and improving overall utilization through pooling and reuse. Mingxin's measured data provides quantitative support for this architectural direction.
Prefetch implementation also involves storage-side path optimization. Per NVIDIA GPUDirect Storage documentation, GPU-direct storage shortens the data path by bypassing the CPU bounce buffer, reducing copy overhead. In R1 tests, the Mingxin FX100 NVMe-oF array (4-disk RAID0, RoCEv2, single-port 100 GbE), combined with the LMCache parallel-read patch, reduced cold-read TTFT at single-GPU concurrency 16 from 37.97s to 9.30s, and improved bandwidth from 0.98 GB/s to 5.23 GB/s (measured, report R1)—a 4.1× TTFT improvement and 5.3× bandwidth gain, showing that storage-side data-path optimization and prefetch strategy are complementary.
Prefetch Deployment Forms: From Single Node to Cluster
The engineering deployment of prefetch strategies extends beyond a single node. Mingxin R4 tests (480B multi-instance) and R5 tests (14B memory-efficiency) validated prefetch benefits at different deployment scales. R9 tests on the Huawei Atlas 910B platform compared FX100 against an NFS baseline for model loading: DeepSeek-32B service loading dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×) (measured, report R9). These figures show that the prefetch strategy is equally effective on Ascend platforms, and the speedup correlates positively with model size—the larger the model, the more data to load, and the clearer the advantage of prefetch over NFS sequential reads.
At the cluster level, according to SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104), RadixAttention's prefix-tree reuse mechanism improves hit rates in multi-turn dialogue scenarios by sharing prefixes. The prefetch strategy is naturally complementary to such prefix-reuse mechanisms: the prefix tree tells the system which KV Cache blocks can be reused, while prefetch ensures that reusable data is already in place when needed.
Conclusion
The core value of KV Cache prefetch lies in removing storage latency from the inference critical path. Measured on the Mingxin FX100 with a 480B model, tiered prefetch delivers 29–40% throughput improvement and 26–32% TTFT reduction (measured, reports R2/R3), with an 8.6–20× speedup over the no-external-storage recompute approach (measured, report R2). These figures provide quantitative evidence for the value of storage acceleration in LLM inference. Mingxin offers an approximately 10-week gated joint-testing cycle (from G1 arrival acceptance to G4 stability validation); compute centers and model service providers are welcome to bring real workloads for validation.
Key Q&A
Q: How much does KV Cache prefetch improve inference latency?
A: Measured on the Mingxin FX100 with a 480B model under TP8 long-context workloads, TTFT is reduced by 26–32% (measured, report R2), and inference throughput improves by 29–40% (measured, reports R2/R3).
Q: How is the advantage of prefetch over the no-external-storage recompute approach quantified?
A: In R2 measurements, the recompute baseline had a TTFT p50 of 149.5s (concurrency 16), versus 11.85s for the FX100 prefetch solution; throughput improved from 4.1 to 74.9 tok/s, for a combined speedup of 8.6–20×.
Q: Is the prefetch strategy effective on non-NVIDIA platforms?
A: R9 measurements on the Huawei Atlas 910B platform show FX100 achieving 6.2–9.3× model-loading speedups over the NFS baseline (DeepSeek-32B/70B), validating cross-platform effectiveness.
References
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- SNIA — Storage Networking Industry Association — https://www.snia.org/
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)