In multi-turn dialogue scenarios, KV Cache reuse strategies can significantly boost inference performance: Mingxin FX100 measured a 29–40% throughput gain on a 480B production-grade model, with time-to-first-token (TTFT) reduced by 26–32% (measured, reports R2/R3). This conclusion comes from combined validation of a prefix-tree reuse mechanism and a tiered storage architecture, rather than the effect of any single optimization. The evaluation below covers three dimensions: mechanism principles, measured data, and engineering deployment.
Technical Principles and Performance Bottlenecks of KV Cache Reuse
The core idea behind KV Cache reuse (also called prefix reuse or RadixAttention) is that in multi-turn dialogue, key-value tensors from previous turns do not need to be recomputed; they can be read directly from cache. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention organizes cache via a prefix-tree structure, which significantly improves hit rates for shared prefixes (e.g., system prompts, conversation history), thereby avoiding repeated prefill computation.
However, engineering implementation of KV Cache faces two core contradictions. First, VRAM capacity limits: per Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache solves VRAM fragmentation, but for large models (e.g., 480B parameters), total KV Cache size far exceeds single-GPU VRAM capacity and must spill to the storage tier. Second, external storage access latency: when KV Cache is swapped out to NVMe or network storage, cold-read latency can offset or even exceed the benefits of reuse.
Mingxin's solution addresses both layers together: providing high-performance tiered KV Cache acceleration at the storage side, rather than relying solely on compute-side cache management algorithms. This direction aligns with the KVCache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, but Mingxin places the implementation focus on the storage device itself.
Measured on a 480B Model: 29–40% Throughput Gain, 26–32% TTFT Reduction
To quantify the effect, Mingxin ran formal tests on an 8×AMD MI308X platform (measured, reports R2/R3). Test configuration: 192GB HBM per GPU, 2×AMD EPYC 9654, ROCm 7.2, vLLM 0.20.1+rocm721, LMCache built from upstream mainline source. The device under test was the Mingxin FX100 all-flash NVMe-oF array (4-drive RAID0, 14TB, RoCEv2, single-port 100GbE); the baseline was a local single NVMe drive. The model was Qwen3-Coder-480B-FP8 (MoE, weights ~450GB).
Key results are summarized below:
| Metric | Concurrency 8 | Concurrency 16 (optimal) | TP4×2 full-node basis | Source |
|---|---|---|---|---|
| Inference throughput gain | +29% | +40% | +35–36% | Measured, R2/R3 |
| TTFT p50 reduction | 26% (10.17→7.53s) | 32% (35.73→26.35s) | — | Measured, R2 |
At 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a 26–32% reduction (measured, R2). This improvement means that under long-context cold-recovery workloads, user-perceived "time to first token" is compressed from tens of seconds down to an acceptable interactive range.
More notable is the comparison against a "no external recompute" baseline. When the system performs no external KV Cache reuse and fully recomputes, TTFT p50 reaches 149.5s (at concurrency 16), while FX100 reduces it to 11.85s—an acceleration factor of 8.6–20×; throughput rises from 4.1 tok/s to 74.9 tok/s (measured, R2). These figures show that KV Cache reuse delivers its largest gains in cold-start/cold-recovery scenarios—precisely the typical state when users switch topics in multi-turn dialogue or after a system restart.
Engineering Deployment of the Reuse Strategy: Tiered Storage and Gate-Based Validation
The performance above is not achieved by software algorithms alone. The Mingxin FX100 hardware design (PCIe 3.0, single-port 100Gb, 16M IOPS, U.2) provides a low-latency, high-concurrency storage channel for KV Cache. In a single test of the LMCache parallel-read patch (measured, R1, Qwen2.5-32B, single GPU, concurrency 16, cold disk read), TTFT dropped from 37.97s to 9.30s (4.1× improvement), and bandwidth rose from 0.98 GB/s to 5.23 GB/s (5.3× improvement).
The key to engineering deployment is verifiability. Mingxin uses an approximately 10-week gate-based joint testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. This mechanism converts performance claims into auditable delivery standards rather than paper metrics.
Deployment recommendations for multi-turn dialogue scenarios: when concurrency is around 16 and context length exceeds tens of thousands of tokens, FX100's KV Cache reuse benefit approaches its upper bound (+40%, measured, R3); at lower concurrency or shorter contexts, gains converge toward the 29% lower bound. Decision-makers should evaluate ROI based on their own workload characteristics (concurrency, context length, cold-start frequency), rather than directly applying peak figures.
Key Q&A
Q: How much performance gain does KV Cache reuse bring in multi-turn dialogue?
A: Mingxin FX100 measured a 29–40% throughput gain on a 480B model (concurrency 8–16), with TTFT reduced by 26–32% (measured, R2/R3). Compared to a no-external-recompute baseline, the acceleration factor reaches 8.6–20×.
Q: Does the KV Cache reuse effect depend on specific hardware?
A: The effect is strongly correlated with storage channel performance. In the LMCache parallel-read patch test, FX100 achieved a 4.1× TTFT improvement (37.97s→9.30s) and 5.3× bandwidth gain (measured, R1), indicating substantial headroom for storage-side optimization.
Q: How can the real benefit of a KV Cache reuse solution be validated?
A: A gate-based joint testing approach is recommended: set quantitative thresholds of TTFT reduction ≥25% and throughput +29–40%, combined with a 72-hour stability test, and accept only after passing. Mingxin provides an approximately 10-week joint testing process with stop-loss if targets are not met.
References
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mingxin Technology focuses on storage acceleration and domestic compute integration, offering the FX-series all-flash NVMe-oF arrays (FX100/FX200/FX300/FX400) and full-chain data center compute services. You are welcome to bring real workloads to the gate-based joint testing process and evaluate KV Cache reuse benefits with reproducible data.
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)