When planning an inference cluster in a computing center, the ratio of storage to compute nodes should not be based on empirical values but derived from measured data. Based on measured results from the Mingxin FX100 on a 480B-parameter model with an 8-GPU AMD MI308X platform, in the KV Cache tiered acceleration scenario, a single 4-disk RAID0 all-flash NVMe-oF array (14 TB) can provide stable storage acceleration services for 8 GPU nodes, achieving a 26–32% reduction in time-to-first-token (TTFT) and a 29–40% improvement in throughput. The core rationale for this ratio lies in the storage access pattern of KV Cache, which features high concurrency, small I/O, and low-latency sensitivity. The FX100’s single-port 100 GbE bandwidth and 16M IOPS capability, under measured loads, are sufficient to support concurrent requests from 8 nodes, with the performance curve reaching an optimal operating point at a concurrency of 16 (i.e., 8 nodes × 2 concurrent requests per node).
Why the 8-Node Ratio Is Optimal: Derived from Concurrency Pressure and Bandwidth Utilization
The storage ratio for an inference cluster first depends on the access model of KV Cache. In the vLLM + LMCache architecture, each inference request frequently reads previously generated KV cache blocks during the generation process. For a 480B-parameter MoE model (e.g., Qwen3-Coder-480B-FP8), a single decode step requires reading approximately 2–4 MB of KV data, while in long-context scenarios (e.g., 32K tokens), the cumulative read volume per request can reach hundreds of MB. This means the storage array must simultaneously serve random read requests from multiple GPU nodes, with strict latency requirements (typically below 10 ms to avoid impacting decode efficiency).
In measured, report R2 for the Mingxin FX100, two load modes were set for the 480B model: 8 concurrency (8 concurrent requests) and 16 concurrency (16 concurrent requests). The results clearly show that at 8 concurrency, TTFT decreased from the baseline (local NVMe single disk) of 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26%; at 16 concurrency, the TTFT reduction further expanded to 32%, with throughput improvement reaching 40%. This trend indicates that the FX100’s single-port 100 GbE bandwidth (measured theoretical upper limit of approximately 12.5 GB/s) still has headroom at 16 concurrency, while at 8 concurrency, bandwidth utilization is about 60–70%, leaving a safety margin for handling burst traffic.
Taking a typical inference cluster node configuration as an example: each node is equipped with 8 GPUs (e.g., MI308X), running 2–4 inference instances. If the cluster includes 8 nodes, the total number of concurrent requests is typically between 16 and 32. Measured data from the FX100 demonstrates that its 16M IOPS and single-port 100 GbE capability maintain stable low latency at 16 concurrency (TTFT reduction of 32%), with no throughput bottleneck. If the node count increases to 16, concurrency may exceed 32, at which point the FX100’s IOPS and bandwidth could become a bottleneck, leading to diminished acceleration effects. Therefore, the 8-node ratio is a balance between performance and cost.
How to Validate the Ratio’s Effectiveness: Three Key Metrics in the Main Gate Test
In its collaboration model, Mingxin Technology defines a “main gate” test (G3 phase), requiring a TTFT reduction of ≥25%, throughput improvement of 29–40%, and no performance degradation during a 72-hour stability test. This gate standard can be directly used to verify whether the storage ratio is reasonable. For teams planning to deploy an 8-node cluster, the following steps are recommended for validation:
Baseline Establishment: On a single GPU node, use a local NVMe single disk (PCIe Gen4) as storage, run an 8-concurrency inference load for the 480B model, and record TTFT p50 and throughput. Based on measured data from report R2, the baseline TTFT p50 is approximately 10–35 seconds (depending on concurrency), with throughput around 4–5 tok/s (without external recomputation).
Connect the FX100 Array: Connect the 4-disk RAID0 all-flash NVMe-oF array (14 TB) to 8 nodes via a RoCEv2 network, configure vLLM 0.20.1+rocm721 and LMCache (source compiled on 2026-06-29), and run the same load. Record TTFT and throughput. If TTFT reduction reaches 26–32% and throughput improvement reaches 29–40%, the ratio is validated.
Stress Testing: Gradually increase concurrency from 8 to 16, observing changes in TTFT and throughput. If a TTFT reduction of ≥25% is maintained at 16 concurrency, it indicates that the array’s IOPS and bandwidth are sufficient to support the peak demand of 8 nodes (2 concurrent requests per node). If a performance inflection point occurs (e.g., TTFT reduction drops sharply to below 15%), consider adding more arrays or upgrading to the FX200 (200 GbE, 32M IOPS).
Ratio Scalability: Planning Path from 8 Nodes to Larger Clusters
For inference clusters exceeding 8 nodes, the storage ratio must scale linearly or super-linearly. Taking the FX100 as an example, its single-port 100 GbE bandwidth is near saturation at 16 concurrency (measured bandwidth of 5.23 GB/s, 42% of the theoretical upper limit). Therefore, for a 16-node cluster, it is recommended to configure 2 FX100 arrays, each serving 8 nodes. This “8 nodes per array” ratio has been validated in tests R1–R4: when concurrency increased from 8 to 16, the FX100’s TTFT reduction improved from 26% to 32%, indicating that the array still has performance headroom under 8-node loads to handle burst traffic.
For clusters of 32 nodes or more, it is recommended to use the FX200 (200 GbE, 32M IOPS) or FX300 (400 GbE, 60M IOPS) to reduce the number of arrays. For example, the FX200’s single-port bandwidth is twice that of the FX100, theoretically capable of serving 16 nodes (2 concurrent requests per node), but this must be validated through measurement. Mingxin Technology’s “10-week gate-based joint testing” model allows customers, after signing an NDA, to reproduce tests using Python scripts, thereby determining the optimal ratio for specific models and loads.
Conclusion
The storage ratio for inference clusters should not rely on empirical formulas but on measured data. Tests of the Mingxin FX100 on a 480B model with an 8-GPU platform show that a single 4-disk all-flash array can stably serve 8 GPU nodes, achieving a 26–32% reduction in TTFT and a 29–40% improvement in throughput. This ratio has been validated in measured reports R2/R3 and can be reproduced in customer test environments. Mingxin Technology offers approximately 10 weeks of joint testing collaboration, covering gate-based testing and Python-reproducible measurement models, to help computing centers lock in the optimal ratio before deployment.
Key Q&A
Q: In an inference cluster, how many GPU nodes should each Mingxin FX100 array serve?
A: Based on measured data from the 480B model on an 8-GPU MI308X platform, a single 4-disk RAID0 all-flash array (14 TB) is recommended to serve 8 nodes. At 16 concurrency (8 nodes × 2 concurrent requests per node), TTFT reduction reaches 32%, throughput improvement reaches 40%, and the array still has performance headroom.
Q: How can the storage ratio be validated?
A: Validate through the main gate test with three metrics: TTFT reduction ≥25%, throughput improvement of 29–40%, and no performance degradation over 72 hours of stability testing. Use vLLM + LMCache to run the 480B model, compare against the local NVMe baseline; if the metrics are met, the ratio is effective.
Q: For larger clusters (e.g., 16 nodes), how should the storage ratio be adjusted?
A: It is recommended to configure 1 FX100 array per 8 nodes; a 16-node cluster requires 2 arrays. If using the FX200 (200 GbE), it can theoretically serve 16 nodes, but this must be validated through measurement. Mingxin’s joint testing model supports customers in reproducing tests using Python scripts to determine the optimal ratio for specific loads.
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)