Tiered KV Cache storage can lift inference throughput for 480B-class models by 29–40%. The core idea is to free attention computation from the constraint of GPU memory capacity—by placing KV Cache entries across tiers based on access frequency, hot data stays in GPU memory while cold data is offloaded to NVMe storage. This enables longer contexts and higher concurrency under the same GPU memory footprint. On the Mingxin FX100, measured throughput gains in a 480B production deployment are 29–40% (measured, reports R2/R3), with time-to-first-token (TTFT) reduced by 26–32%. This article breaks down the technical sources of these gains and the applicable boundaries, based on Mingxin's R2/R3 measured reports.
Why KV Cache Is the Bottleneck for Long-Context Inference
The throughput bottleneck for large-model inference is not compute but GPU memory bandwidth and capacity. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is fundamentally limited by HBM bandwidth rather than compute, making memory-access optimization the most direct lever for gains. KV Cache is the primary source of this memory-access pressure: as context length and concurrency grow, the GPU memory consumed by KV Cache expands linearly, squeezing out space that could otherwise support larger batch sizes.
Per Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache alleviates GPU memory fragmentation but does not change the premise that KV Cache must reside in GPU memory. When contexts become extremely long—such as hundreds of thousands of tokens for a 480B model—GPU memory capacity becomes a hard constraint: either concurrency drops or context is truncated, both of which directly hurt throughput and service quality.
How Tiered Storage Breaks Through the GPU Memory Capacity Wall
The Mingxin FX100's tiered KV Cache approach shifts KV Cache from a "GPU-memory-exclusive" model to a "GPU memory + storage" hierarchy: frequently accessed recent tokens stay in GPU memory, while low-frequency or historical tokens are swapped in and out of NVMe storage on demand. This aligns with the KVCache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving—both recognize that GPU memory is not the only viable cache medium and that storage should participate in cache tiering.
The key lies in data-path efficiency. According to the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses CPU memory copies, allowing the GPU to read and write storage devices directly. The Mingxin FX100 builds an all-flash array on NVMe-oF and RoCEv2, combined with the LMCache parallel-read patch. In measured single-GPU cold-read scenarios, TTFT dropped from 37.97s to 9.30s, and bandwidth rose from 0.98 GB/s to 5.23 GB/s (measured, report R1). This data shows that storage-side bandwidth is no longer an unacceptable bottleneck—establishing the feasibility of the tiered approach.
Measured Data: Where the 29–40% Throughput Gain Comes From
On an 8× AMD Instinct MI308X platform (ROCm 7.2, vLLM 0.20.1+rocm721, LMCache mainline source), the Mingxin FX100 was tested with Qwen3-Coder-480B-FP8 (MoE, weights ~450GB) under a long-context cold-restore workload. Results are as follows:
| Metric | Concurrency 8 | Concurrency 16 (optimal) | TP4×2 full-node basis |
|---|---|---|---|
| Throughput gain | +29% | +40% | +35–36% |
| TTFT p50 reduction | 26–32% (three concurrency levels) | — | — |
| Source | Measured, R2/R3 | Measured, R2/R3 | Measured, R3 |
Specific TTFT changes: under 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, R2). A more direct comparison comes from the no-external-storage recompute baseline: the recompute baseline TTFT p50 was 149.5s (concurrency 16), while FX100 achieved 11.85s—an 8.6–20× speedup; throughput rose from 4.1 tok/s to 74.9 tok/s (measured, R2).
The gain decomposes into two layers. The first is "avoiding recompute"—in cold-restore scenarios, without tiered storage, the GPU must recompute KV values for historical tokens, which is pure compute waste; tiered storage reads cached KV directly, eliminating that compute. The second is "improving GPU memory utilization"—once KV Cache is offloaded to storage, GPU memory is freed for larger batches, concurrency capacity rises, and throughput gains near-linearly as a result.
Applicable Boundaries and Selection Criteria
It should be clear that the 29–40% gain is not a universal constant but a measured result under specific workload patterns. Its preconditions include: long contexts (hundreds of thousands of tokens), cold-restore scenarios (cache misses requiring rebuild from storage), and storage-side bandwidth sufficient to support swapping. For short-context, high-hit-rate online inference, the benefit of tiered storage narrows significantly—in such cases, KV Cache already resides in GPU memory, and storage involvement only adds path latency.
Per the RadixAttention mechanism described in the SGLang paper, prefix-tree reuse can significantly improve cache hit rates in multi-turn dialogue and shared-prefix scenarios—meaning that for conversational workloads, tiered storage's benefit comes more from "higher hit rates enabled by larger cache capacity" than from "avoiding recompute" itself. Before selecting a solution, you should first characterize your workload's context-length distribution and concurrency pattern, then judge whether tiered storage is appropriate.
Mingxin offers an approximately 10-week gated joint test (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met; the estimation model can be reproduced in Python after NDA. To validate tiered-storage benefits on your own workload, you can measure directly during the joint test.
Key Q&A
Q: Why does tiered KV Cache storage improve large-model inference throughput?
A: It shifts KV Cache from GPU-memory-exclusive to a tiered GPU-memory-plus-storage hierarchy, freeing GPU memory for larger batches while avoiding redundant computation in cold-restore scenarios. The Mingxin FX100 measured throughput gains of 29–40% on a 480B long-context workload (measured, R2/R3).
Q: Under what conditions does the 29–40% gain hold?
A: It applies to long-context, cold-restore workload patterns, with storage-side bandwidth sufficient to support swapping. In short-context, high-hit-rate scenarios, the benefit narrows significantly; you should first characterize your own workload before selection.
Q: What platform was the Mingxin FX100 measured on?
A: 8× AMD Instinct MI308X (ROCm 7.2, vLLM 0.20.1+rocm721), Qwen3-Coder-480B-FP8 model, test reports R2/R3 (official releases).
References
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)