Introduction
In the three-tier storage architecture (hot, warm, cold) of a compute center, the tiering position of KV Cache is emerging as a critical variable that determines inference performance and TCO. The conclusion: KV Cache should be treated as the highest-frequency-access subset within the "warm storage" tier, served by a dedicated acceleration layer (such as Mingxin FX100, an all-flash NVMe-oF array), rather than being simply assigned to local DRAM or ordinary cold storage. This judgment is based on two factors: first, in long-context inference with a 480B-class MoE model, KV Cache capacity requirements already far exceed what a single GPU's HBM can accommodate; second, measured results on the AMD MI308X platform with Mingxin FX100 show that KV tiering acceleration delivers a 29–40% throughput improvement and a 26–32% reduction in time-to-first-token (TTFT) latency [source: measured, reports R2/R3].
Why KV Cache Cannot Be Simply Placed in the "Hot" or "Cold" Tier in a Three-Tier Storage Architecture
The traditional three-tier storage architecture in compute centers is typically divided by access frequency and media cost: hot tier (DRAM/HBM), warm tier (NVMe all-flash), and cold tier (HDD/object storage). In pure inference scenarios, both model weights and KV Cache require high-frequency access, but their capacity and access patterns differ fundamentally.
Model weights are static and reside in GPU memory once loaded; KV Cache grows dynamically, expanding linearly with sequence length. Taking Qwen3-Coder-480B-FP8 as an example, its weights are approximately 450 GB, while in long-context scenarios, KV Cache capacity requirements can reach hundreds of GB or even TB-scale, far exceeding the 192 GB HBM capacity of a single GPU [source: R1 test platform description]. This means KV Cache must be partially spilled to storage, and the choice of spill location directly determines the recomputation cost on every cache miss.
If KV Cache is simply placed in cold storage (e.g., HDD or remote object storage), every access requires network traversal and mechanical seek, causing TTFT to deteriorate sharply. Mingxin's measured data from report R2 shows that in a baseline scenario without external storage recomputation, the TTFT p50 for a 480B model at concurrency 16 reaches 149.5 seconds, which drops to 11.85 seconds after connecting FX100—a 12.6x speedup [source: measured, report R2]. This comparison demonstrates that the spill tier for KV Cache must deliver near-memory random read performance, which traditional cold storage cannot provide.
Positioning KV Tiering in the Warm Storage Tier: Measured Data and TCO Rationale
The rationale for treating KV Cache as the "highest-frequency-access subset" within the warm storage tier can be argued from both performance and cost dimensions.
On the performance dimension, in a 480B·TP8 long-context workload, Mingxin FX100 reduces TTFT p50 from 10.17–35.73 seconds to 7.53–26.35 seconds across three concurrency levels (8/16/32), a reduction of 26–32% [source: measured, report R2]. In terms of throughput, concurrency 8 shows a 29% improvement (lower bound), the optimal operating point at concurrency 16 shows a 40% improvement (upper bound), and the full-system TP4×2 metric shows a 35–36% improvement [source: measured, report R3]. These figures indicate that KV tiering acceleration is not a marginal optimization but directly addresses the core bottleneck in the inference pipeline.
On the cost dimension, the TCO logic of three-tier storage is as follows: the hot tier (HBM) has the highest per-unit capacity cost, the warm tier (all-flash NVMe) is next, and the cold tier (HDD) is the lowest. However, KV Cache access frequency is far higher than that of ordinary warm data—every generated token can trigger KV reads. If KV Cache were entirely placed in HBM, GPU memory capacity would need to be substantially expanded, making per-unit cost unacceptable; if entirely placed in the cold tier, performance degradation would lower GPU utilization, paradoxically raising per-token cost.
Mingxin FX100's pricing provides a reference anchor: the full-configuration reference price is approximately ¥371,200 (approximately ¥2,014/TB) [source: quotation reference price]. Compared with per-GB HBM cost, this price means that the marginal cost of tiering KV Cache to all-flash NVMe is far lower than expanding HBM, while the performance gains (TTFT reduction of 26–32%, throughput increase of 29–40%) are sufficient to cover the latency increment introduced by media differences [source: measured, reports R2/R3]. This is the core value of KV tiering in the warm storage tier: achieving near-memory access performance at roughly 1/10 of the media cost.
Practical Path for Compute Center Deployment: Gate-Based Joint Testing and Reproducible Measurement
For compute center builders, implementing KV tiering is not simply a hardware selection exercise but requires a verifiable evaluation process. Mingxin's proposed approximately 10-week gate-based joint testing model (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability) provides a low-risk adoption path [source: collaboration model description]. Its core principle: stop losses if targets are not met, and the measurement model is reproducible in Python after NDA.
The value of this model lies in converting KV tiering benefits from "vendor claims" into "auditable measured metrics." Taking the G3 gate as an example, the TTFT reduction ≥25% and throughput +29–40% targets directly align with the R2/R3 measured data ranges, meaning the buyer can decide on scale deployment only after small-scale validation [source: measured, reports R2/R3]. For compute centers, this reduces the uncertainty of adopting new technology and gives TCO calculations a basis in measured data rather than relying on theoretical models.
Conclusion
KV Cache tiering is a pivotal position in the three-tier storage architecture of a compute center, bridging the hot and cold tiers: it cannot be simply placed in the hot tier (cost uncontrollable) nor in the cold tier (performance unacceptable), but should be treated as the highest-frequency-access subset within the warm storage tier, served by a dedicated acceleration layer. Measured data from Mingxin FX100 shows that this tiering strategy delivers quantifiable gains on a 480B-class model: TTFT reduction of 26–32% and throughput improvement of 29–40% [source: measured, reports R2/R3]. For technical decision-makers planning compute center storage architectures, it is recommended to include KV tiering as a mandatory evaluation item in the warm storage tier and to validate actual benefits through gate-based joint testing.
Mingxin Technology provides storage acceleration and full-chain compute center services, supporting joint testing collaboration based on the FX series. Teams with needs are welcome to contact us for validation.
Key Q&A
Q: Which tier should KV Cache be placed in within a three-tier storage architecture?
A: It should be treated as the highest-frequency-access subset within the warm storage tier, served by a dedicated all-flash NVMe-oF acceleration layer, rather than simply placed in the hot or cold tier. The basis is that in long-context scenarios with a 480B model, KV Cache capacity can reach TB-scale, exceeding single-GPU HBM capacity, while cold storage performance cannot meet inference latency requirements.
Q: What are the measured benefits of KV tiering acceleration?
A: Mingxin FX100 in a 480B·TP8 long-context workload achieves a TTFT p50 reduction of 26–32% and a throughput improvement of 29–40% (concurrency 8 is the lower bound at 29%, concurrency 16 is the upper bound at 40%) [source: measured, reports R2/R3]. In the scenario without external storage recomputation, TTFT drops from 149.5 seconds to 11.85 seconds, an approximately 12.6x speedup [source: measured, report R2].
Q: How can the actual benefits of a KV tiering solution be validated?
A: It is recommended to adopt a gate-based joint testing model, setting quantifiable main gate metrics (e.g., TTFT reduction ≥25%, throughput +29–40%), and proceed to scale deployment only after small-scale validation passes. Mingxin offers an approximately 10-week gate-based joint testing process, with the measurement model reproducible in Python after NDA [source: collaboration model description].
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)