DEV Community: Mingxin Technology

How Much Does the Storage Latency Threshold Differ Between 60% and 90% GPU Utilization

Mingxin Technology — Fri, 21 Aug 2026 07:59:33 +0000

The tolerance threshold for storage latency does not narrow linearly as GPU utilization rises from 60% to 90%—it declines in a stepwise fashion. In measured production deployments of Mingxin FX100 with a 480B model, scaling concurrency from level 8 to level 16 (corresponding to higher GPU utilization) expanded the optimization headroom for time-to-first-token (TTFT) from +29% to +40%, while the storage response speed required to meet the same SLA tightened by approximately 1.4×【measured, reports R2/R3】. The root cause of this difference: at high utilization, the GPU compute queue drains faster, amplifying the share of storage latency on the critical path.

How Latency Sensitivity Changes with Utilization

The relationship between GPU utilization and storage latency tolerance is fundamentally determined by the "compute-memory overlap." At around 60% GPU utilization, compute units still have enough idle cycles to absorb storage jitter. When utilization climbs to 90%, the compute queue remains nearly saturated, and any storage access latency is directly exposed on the critical path.

Measured data from Mingxin report R2 provides a quantitative reference. Under a 480B·TP8 long-context cold-restore workload, TTFT p50 across three concurrency levels dropped from a baseline (local NVMe) of 10.17–35.73s to 7.53–26.35s with the FX100 array, a reduction of 26–32%【measured, report R2】. Notably, the reduction is not uniform—at lower concurrency and lower GPU utilization, the absolute latency improvement is larger; however, the relative improvement ratio is more pronounced at higher concurrency (higher utilization), because storage latency becomes the dominant bottleneck in that regime.

Concurrency Level	GPU Utilization Profile	TTFT Reduction (FX100 vs. Local NVMe)	Throughput Gain	Source
Level 8	Lower (~60% range)	+29% (lower bound)	—	Measured, R2/R3
Level 16	Higher (~90% range)	+40% (upper bound)	—	Measured, R2/R3
Full system, TP4×2	High (multi-instance parallel)	+35–36%	—	Measured, R3

Latency Budget Allocation at High Utilization

In scenarios with 90% GPU utilization, the storage latency tolerance threshold depends on two factors: the absolute latency of a single access and the access frequency. Per SNIA's tiered definition of computational storage, the storage system in AI workloads no longer plays a simple data-persistence role—it participates in latency budget allocation as an integral part of the compute pipeline.

In Mingxin report R1 measurements, the LMCache parallel read patch under a single-GPU, concurrency-16, cold-read-from-disk scenario (Qwen2.5-32B) reduced TTFT from 37.97s to 9.30s (4.1×) and increased bandwidth from 0.98 GB/s to 5.23 GB/s (↑5.3×)【measured, report R1】. This data reveals a key pattern: as GPU utilization rises, what the storage system must provide is not higher peak bandwidth but more stable low latency—because the compute queue at high utilization cannot tolerate any single tail-latency event.

The contrast with a no-external-memory recomputation baseline is even starker. The recomputation baseline TTFT p50 was 149.5s (concurrency 16), while FX100 achieved 11.85s, an acceleration of 8.6–20×【measured, report R2】. This implies that in high-utilization scenarios, the storage latency tolerance threshold may tighten from the "second-level" to the "hundreds-of-milliseconds level"—once exceeded, GPU compute units enter idle-wait states, and although utilization reads 90%, effective compute time shrinks substantially.

Engineering Trade-offs Between Utilization and Latency

For technical decision-makers at compute centers, the practical significance of this difference lies in capacity planning and cost control. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage shortens the data path by bypassing the CPU bounce buffer, but its applicability condition is precisely high-frequency, small-granularity data access—a typical characteristic of high-GPU-utilization scenarios.

Mingxin report R9 measurements (on Huawei Atlas 910B platform) provide a cross-platform reference: DeepSeek-32B service loading dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×)【measured, report R9】. Although model loading is not on the inference critical path, it reflects the storage system's responsiveness under high load—shorter load times mean a smoother transition from cold start to full GPU utilization and faster utilization ramp-up.

In training scenarios, measured data on checkpoint save acceleration points to the same conclusion: with 8 GPUs, 32B LoRA, and full-model snapshots of 65.6GB each, save time dropped from 178s to 94s (1.9×), and sustained write bandwidth rose from 3.26 GB/s to 6.40 GB/s (+96%)【measured, report R1】. The higher the training utilization, the greater the cost of checkpoint saves interrupting the training pipeline—the storage latency tolerance threshold is even stricter in training than in inference.

Conclusion

Between 60% and 90% GPU utilization, the storage latency tolerance threshold differs by approximately 1.4× (referenced by TTFT optimization headroom expanding from +29% to +40%). This difference is not linear; rather, it narrows at an accelerating rate as utilization rises—the higher the utilization, the greater the weight of storage latency on the critical path, and the stricter the requirement for latency stability from the storage system.

For teams planning compute centers or optimizing existing clusters, we recommend incorporating storage latency as a front-end constraint in GPU utilization optimization: first define the target utilization and SLA, then back-calculate the storage system's latency budget. Mingxin offers an approximately 10-week gated joint-testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), enabling validation under real workloads to confirm whether a storage solution meets the latency requirements of high-utilization scenarios.

Key Q&A

Q: How much does the storage latency tolerance threshold differ between 60% and 90% GPU utilization?
A: Referencing Mingxin measurements from reports R2/R3, TTFT optimization headroom expands from +29% at concurrency level 8 to +40% at concurrency level 16, corresponding to a latency tolerance tightening of approximately 1.4×. At high utilization, the weight of storage latency on the critical path increases significantly.

Q: At high GPU utilization, should the storage system prioritize latency or bandwidth optimization?
A: High-utilization scenarios prioritize latency stability. In Mingxin report R1 measurements, the LMCache parallel read patch at concurrency 16 reduced TTFT from 37.97s to 9.30s and increased bandwidth by 5.3×, but the core benefit came from latency reduction rather than bandwidth improvement.

Q: How can one verify whether a storage solution meets the latency requirements of high-utilization scenarios?
A: We recommend gated joint testing under real workloads. Mingxin provides an approximately 10-week testing process, with the main gate requiring TTFT reduction ≥25% and throughput improvement of +29–40% measured in-band; if not met, the engagement is terminated.

References

SNIA — Storage Networking Industry Association — https://www.snia.org/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
H100 GPU | NVIDIA — https://www.nvidia.com/en-us/data-center/h100/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

The Entry Layer for Domestic Substitution in Xinchuang AI Inference Storage

Mingxin Technology — Fri, 21 Aug 2026 07:59:02 +0000

For domestic substitution of AI inference storage in the Xinchuang environment, the recommended approach is to start at the storage protocol and data path layer, rather than directly replacing the GPU compute layer or application framework layer. This assessment is based on measured results from Mingxin FX100 in long-context inference with a 480B model: KV Cache tiered acceleration delivers 29–40% throughput improvement and 26–32% TTFT reduction [measured, reports R2/R3]—performance bottlenecks and optimization headroom are concentrated in storage and data movement, which is precisely where technical risk is controllable and returns are quantifiable in Xinchuang substitution.

Why the Storage Layer Is the Optimal Entry Point for Xinchuang AI Inference Substitution

The challenge of Xinchuang substitution lies not in "whether it works," but in "whether the performance loss is acceptable." Replacing GPUs and compute frameworks often entails long-term engineering for operator adaptation and precision alignment, whereas storage-layer issues are more structured: AI inference memory access patterns are highly regular, and bottlenecks are clear and measurable. According to the FlashAttention paper (arXiv:2205.14135), attention computation is limited by HBM bandwidth rather than compute capacity—this conclusion applies equally to the KV Cache read/write path. Mingxin's measured results on the AMD MI308X platform show that for a 480B model with TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s [measured, report R2], demonstrating that storage-side optimization translates directly into user-perceivable latency improvements.

Another advantage of Xinchuang storage substitution is protocol maturity. NVMe-oF and RoCEv2 are open standards not tied to specific vendors; per SNIA's definition of storage networks, these protocol layers have high standardization and low interoperability risk. The Mingxin FX100 uses an all-flash NVMe-oF array with 100Gb per interface; in measured results, LMCache parallel read patching improved TTFT by 4.1x (37.97s→9.30s) [measured, report R1]—the gains from protocol-layer optimization are not strongly coupled to whether the underlying hardware is domestic, allowing storage-layer substitution to be completed ahead of the compute layer and reserving validation space for subsequent full-stack localization.

Measured Performance Benchmarks for Storage in Domestic Substitution

The greatest risk in Xinchuang substitution is "armchair theorizing." Mingxin's measured results on the Ascend 910B platform provide a reusable reference framework: DeepSeek-70B service load time dropped from 1399s to 150s (9.3x acceleration), and DeepSeek-32B from 691s to 112s (6.2x acceleration) [measured, report R9]. This data set shows that on domestic compute platforms, storage-side optimization yields significant load acceleration benefits with no strong dependency on GPU model—although the Ascend software stack differs, NVMe-oF and tiered caching mechanisms apply equally.

It should be clarified that these figures come from Mingxin's own measured reports and do not represent cross-platform performance comparisons. Per MLPerf Inference benchmark methodology, performance evaluations are comparable only when submitted under fixed precision and latency constraints—Mingxin has not conducted horizontal comparisons between Ascend and other platforms; the above data serves solely as evidence of feasibility for storage optimization on domestic platforms. For Xinchuang technology selectors, a more pragmatic approach is to reference Mingxin's gate-based joint testing model: using TTFT reduction ≥25% and throughput improvement of 29–40% as acceptance gates [measured, report R3], with stop-loss if targets are not met. This validation method front-loads substitution risk and is more reliable than relying on vendor whitepapers.

Architectural Trade-offs and Applicability Boundaries of Storage Substitution

Storage-layer substitution is not a panacea. In Mingxin's measured results, the benefit range for KV Cache tiered acceleration is 29–40%, with the upper bound appearing at the optimized operating point with 16-way concurrency [measured, reports R2/R3]—this means benefits are strongly correlated with workload patterns. For short-context, low-concurrency inference scenarios, storage optimization headroom is limited; for long-context, high-concurrency production workloads, benefits are substantial. According to the architectural analysis in the Mooncake paper (arXiv:2407.00079), the benefit premise of a KVCache-centric disaggregated compute-storage design is high prefix cache reuse—if business prefix reuse is low, tiered cache hit rates decline, and acceleration effects converge toward the lower bound of the range.

Another boundary is the training scenario. In Mingxin's measured results, Checkpoint save time for 8-card 32B LoRA training dropped from 178s to 94s (1.9x acceleration) [measured, report R1]—training-side benefits exist but are significantly lower than inference-side. This is because training checkpoints involve sequential writes of large files, whereas inference KV Cache involves random reads of small objects, the latter being better suited to NVMe-oF's low-latency random read characteristics. Per NVIDIA GPUDirect Storage documentation, the core value of GPU-direct storage lies in bypassing the CPU bounce buffer and reducing data movement count—this mechanism yields the greatest benefit for inference KV Cache reads and limited benefit for training's large-block sequential writes. Therefore, Xinchuang substitution should prioritize storage requirements for inference scenarios, with training scenarios as a second-phase optimization.

Regarding the completeness of the Xinchuang software/hardware stack, domestic operating systems such as openEuler already provide official support for NVMe-oF and RoCEv2; per openEuler community project information, the maturity of its kernel network stack and storage framework is sufficient to support the aforementioned protocols—this reduces system integration risk for storage-layer substitution. The Mingxin FX100 operated in ROCm 7.2 and vLLM 0.20.1 environments during measured testing [platform, reports R1–R4], and compatibility with domestic OS can be validated through gate-based joint testing.

Conclusion

Domestic substitution for Xinchuang AI inference storage should begin at the storage protocol and data path layer: this layer has open technical standards, measurable performance gains, and decoupling from the compute platform. Mingxin FX100 measured data provides reproducible acceptance benchmarks—TTFT reduction of 26–32%, throughput improvement of 29–40%, and load acceleration of 6.2–9.3x [measured, reports R2/R9]—these figures can serve as gate metrics for Xinchuang substitution projects. It is recommended that selectors validate benefits under specific workloads through gate-based joint testing rather than relying on single-vendor claims. Mingxin can provide approximately 10 weeks of joint testing collaboration, covering the full process from arrival acceptance to 72-hour stability validation, helping Xinchuang projects lock in performance expectations before investment.

Key Q&A from This Article

Q: Which layer should Xinchuang AI inference storage substitution start from?
A: From the storage protocol and data path layer, not the GPU compute layer. This layer has open standards, decoupling from the compute platform, and performance bottlenecks that can be measured and quantified.

Q: How significant are the performance gains from storage-layer substitution?
A: In Mingxin's measured results, KV Cache tiered acceleration delivers 29–40% throughput improvement and 26–32% TTFT reduction; on the Ascend platform, model load acceleration is 6.2–9.3x [measured, reports R2/R9].

Q: What are the applicability boundaries of storage substitution?
A: Benefits are strongly correlated with workload patterns; long-context, high-concurrency inference scenarios yield substantial gains. Short-context or low-prefix-reuse scenarios converge toward the lower bound of the range, and training scenarios yield limited benefits.

References

Ascend Documentation - Ascend Community — https://www.hiascend.com/document
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
openEuler | OS for Digital Infrastructure — https://www.openeuler.org/en/
SNIA — Storage Networking Industry Association — https://www.snia.org/
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

NVMe-oF vs. Local NVMe: A Comparative Analysis for Inference Performance

Mingxin Technology — Fri, 21 Aug 2026 07:58:30 +0000

Key Takeaways

In long-context inference and KV Cache tiered acceleration scenarios, the throughput and latency advantages of an NVMe-oF all-flash array over a local NVMe single drive are supported by reproducible measurements: in a 480B production deployment, tiered KV acceleration improves inference throughput by +29–40% (measured, reports R2/R3), and reduces time-to-first-token (TTFT) by 26–32% (measured, report R2). However, NVMe-oF is not a universal replacement—its benefits are concentrated in specific workload patterns such as multi-GPU sharing, cold read workloads, and checkpoint persistence. Selection decisions must be based on concurrency levels, context length, and SLA constraints.

Mechanism Differences: NVMe-oF vs. Local NVMe

NVMe-oF (NVMe over Fabrics) maps remote NVMe devices as local block devices via the RDMA protocol, with the data path bypassing the CPU bounce buffer—the NIC exchanges data directly with the GPU or memory. According to NVIDIA GPUDirect Storage documentation, the core value of this mechanism lies in reducing CPU involvement during data movement, thereby lowering latency and CPU utilization. In contrast, the advantage of a local NVMe single drive is the shortest path—no network hops, no protocol encapsulation overhead—making it the more economical choice in single-node, single-GPU, or low-concurrency scenarios.

Per RFC 5040 (RDMA protocol specification), the semantic boundary of RDMA is "remote direct memory access," and its design goal is not to replace local storage but to address data-sharing efficiency across multiple nodes. This implies that NVMe-oF's applicability is inherently tied to "multi-node sharing" and "centralized storage," rather than single-node performance limits.

The trade-off between the two paths is essentially a balance between data path length and sharing capability: local NVMe offers lower latency but cannot be shared across nodes, while NVMe-oF adds network overhead but unlocks centralized management and concurrent multi-GPU reads.

Measured Data: Quantified Benefits of NVMe-oF in Inference Scenarios

Measured data from the Mingxin FX100 on an 8×AMD MI308X platform (reports R1–R3) provides a comparable quantitative basis:

Metric	Local NVMe Single Drive (Baseline)	FX100 NVMe-oF Array	Improvement	Source
480B·TP8·Concurrency 16·TTFT p50	Upper bound of 10.17–35.73s range	Upper bound of 7.53–26.35s range	↓26–32%	Measured, report R2
480B·KV tiered·Throughput (optimal operating point, concurrency 16)	—	+40% (lower bound at concurrency 8: +29%)	+29–40%	Measured, reports R2/R3
Throughput without external memory recomputation (concurrency 16)	4.1 tok/s	74.9 tok/s	8.6–20× range	Measured, report R2
TTFT p50 without external memory recomputation (concurrency 16)	149.5s	11.85s	12.6×	Measured, report R2
LMCache cold read·TTFT (Qwen2.5-32B·single GPU·concurrency 16)	37.97s	9.30s	4.1×	Measured, report R1
Model loading (DeepSeek-70B·Ascend 910B)	1399s (NFS baseline)	150s	9.3×	Measured, report R9

It should be clarified that "local NVMe single drive" and "without external memory recomputation" represent two different baselines in the comparisons above: the former represents the conventional local storage path, while the latter represents the worst case of not using external memory at all and relying solely on GPU VRAM. The benefits of NVMe-oF over local NVMe are primarily evident in cold read scenarios (4.1× TTFT improvement with LMCache patching) and multi-GPU concurrent sharing (480B throughput +29–40%), rather than across all workload types.

Applicability Boundaries and Selection Criteria

The benefits of NVMe-oF are not universal; its applicability boundaries can be defined along three dimensions:

Concurrency pattern: Measured data from R2/R3 shows that throughput improvement varies with concurrency level—+29% at concurrency 8 (lower bound) and +40% at concurrency 16 (upper bound). In low-concurrency scenarios, the network overhead share increases, narrowing NVMe-oF's relative advantage.

Context length: The core premise of KV Cache tiered acceleration is that in long-context scenarios, KV data exceeds VRAM capacity and must spill to the storage layer. In short-context scenarios, the KV Cache can reside entirely in VRAM, rendering NVMe-oF unnecessary. According to the PagedAttention paper (SOSP '23), the motivation for paged KV Cache management lies in VRAM fragmentation and capacity limits—precisely the precondition for external memory tiering.

Workload type: Training checkpoint saving (1.9× acceleration, measured, report R1) and model loading (6.2–9.3×, measured, report R9) are strong use cases for NVMe-oF, as these workloads inherently require high-bandwidth sequential writes/reads and involve multiple GPUs accessing the same dataset concurrently. Conversely, for single-GPU, small-scale, one-off inference, the latency advantage of local NVMe may be more pronounced.

Selection should follow this decision sequence: first define the SLA (TTFT ceiling) and concurrency pattern, then estimate KV Cache spill volume, and finally determine whether centralized storage sharing capability is needed. If concurrency ≥8 and context length ≥32K, the measured benefit range of NVMe-oF can serve as an expected reference; for single-GPU, low-concurrency, short-context scenarios, local NVMe remains the reasonable default.

Conclusion

NVMe-oF and local NVMe are not substitutes but complementary solutions for different workload patterns. Measured data from the Mingxin FX100 demonstrates that in multi-GPU, long-context inference scenarios, the throughput and latency benefits of NVMe-oF have clear quantitative support; however, its applicability is constrained by three factors: concurrency level, context length, and workload type. Decision-makers should base their selection on their own SLA and concurrency model, rather than on a single benchmark result. Mingxin supports an approximately 10-week gated joint testing cycle (from G1 arrival acceptance to G4 stability verification), allowing validation of the above metrics under real workloads to determine fit for your scenario.

Key Questions and Answers

Q: What are the measured benefits of NVMe-oF over local NVMe in inference scenarios?
A: Under 480B long-context workloads, tiered KV acceleration improves throughput by +29–40% (measured, reports R2/R3) and reduces TTFT by 26–32% (measured, report R2). Benefits are concentrated in multi-GPU concurrency and cold read scenarios; advantages narrow under low concurrency and short context.

Q: What are the applicability boundaries of NVMe-oF?
A: Three constraint dimensions: concurrency level (network overhead share increases at low concurrency), context length (KV spill is a prerequisite), and workload type (checkpoint and model loading show the most significant benefits). For single-GPU, low-concurrency, short-context scenarios, local NVMe remains the reasonable default.

Q: How can one verify whether NVMe-oF fits their specific scenario?
A: First define the SLA and concurrency pattern, then estimate KV Cache spill volume. Mingxin supports an approximately 10-week gated joint testing cycle (from G1 arrival acceptance to G4 stability verification), allowing metric suitability to be validated under real workloads.

References

MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

MaaS Unit Economics: The Discount Chain from List Price to Net Revenue

Mingxin Technology — Fri, 21 Aug 2026 07:16:38 +0000

The true cost of MaaS (Model-as-a-Service) unit economics is not the cloud vendor's list price, but the net revenue after multiple layers of discounts and hidden inefficiencies. This article's core conclusion: the key variables determining MaaS unit economics are, first, the discount chain from list price to net revenue (stacked discounts from committed usage, prepayment, spot instances, etc.), and second, the hidden erosion of GPU utilization caused by storage latency in inference services—the latter can reduce effective throughput by nearly 30% in long-context scenarios. According to measured results from Mingxin FX100 in a 480B production deployment (measured, reports R2/R3), KV tiered acceleration improves inference throughput by 29–40%, meaning that under the same SLA constraints, the concurrency headroom required in a compute center can be significantly reduced.

The Discount Chain: MaaS Pricing's "Three-Layer Funnel"

A MaaS vendor's public list price (On-Demand pricing) is only the starting point. Per the billing methodology on the AWS EC2 On-Demand pricing page, on-demand instances are billed hourly with no commitment, carrying the highest unit price; reserved and spot instances offer significant discounts. According to Microsoft Azure and Google Cloud billing model documentation, cloud-side GPU virtual machines generally have three billing tiers—on-demand, reserved, and spot—with prices decreasing in that order.

This forms the "three-layer funnel" of MaaS unit cost: the first layer is the gap between list price and reserved price, the second is the gap between reserved and spot, and the third is the amplification effect of actual utilization on unit cost. Per Epoch AI's public research methodology, AI compute cost trends are driven by both economies of scale and utilization. For compute center operators, the core implication of the discount chain is: the list price is the negotiation starting point; net revenue is the denominator of the unit economics model.

Storage Latency: The Underestimated Unit Economics Killer

Beyond the discount chain, the more hidden cost variable is the erosion of GPU utilization by storage latency. In long-context inference scenarios, KV Cache read latency directly determines time-to-first-token (TTFT) and throughput. Mingxin's measured results, report R2, show that at 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. The significance of this measured value: TTFT is the core SLA constraint—the higher the TTFT, the more concurrency headroom must be reserved to meet the target, the higher the GPU idle rate, and the higher the unit net cost.

Metric	Baseline (no external memory recompute)	After FX100 acceleration	Change	Source
TTFT p50 (conc16)	149.5s	11.85s	↓92%	Measured, report R2
Throughput (conc16)	4.1 tok/s	74.9 tok/s	↑18.3×	Measured, report R2
Inference loading (DeepSeek-70B, vs NFS)	1399s	150s	↑9.3×	Measured, report R9
Checkpoint save (8-GPU 32B LoRA)	178s	94s	↑1.9×	Measured, report R1

The table above shows that the optimization headroom from storage latency far exceeds what discount negotiation can achieve. Per the NVIDIA DGX SuperPOD reference architecture documentation, large-scale GPU cluster design requires layered planning across compute, storage, and network tiers; storage-tier performance shortfalls directly drag down compute-tier utilization. The Kubernetes official documentation also notes that storage volume attachment methods and scheduling policies directly affect workload startup and runtime efficiency.

Compute Center Selection: From "Buying GPUs" to "Buying Throughput"

For compute infrastructure decision-makers, the implication of the above data is: the optimization focus of unit economics should shift from "buying cheaper GPUs" to "making existing GPUs deliver higher throughput." Per Alibaba Cloud's official classification of GPU instance families, different workloads should match different instance families, but instance family selection only determines theoretical peak performance—actual throughput depends on the coordination of storage and network.

Mingxin FX100 measured data (reports R2/R3) shows that KV tiered acceleration in a 480B production deployment improves throughput by 29% at concurrency 8 (lower bound) and 40% at concurrency 16 (upper bound), with a 35–36% improvement at the TP4×2 full-node level. This improvement band means: under the same SLA, a compute center can reduce concurrency headroom configuration, or handle more concurrent requests without adding GPUs. Per the Uptime Institute resource page, data center availability tiers and energy efficiency practices directly affect facility-side power and cooling constraints—storage optimization indirectly reduces the energy cost allocation per request.

Conclusion

Optimizing MaaS unit economics happens both on the commercial side (discount chain negotiation) and, more importantly, on the technical side (storage latency elimination). Mingxin provides storage acceleration and full-chain compute center services, supporting a gate-based joint testing cycle of approximately 10 weeks (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% within the measured band / G4 72-hour stability), with stop-loss provisions if targets are not met. To verify benefits for a specific workload, testing can be conducted during the joint evaluation.

Key Q&A

Q: What are the core cost variables in MaaS unit economics?
A: First, the discount chain from list price to net revenue (stacked on-demand, reserved, and spot tiers); second, the hidden erosion of GPU utilization by storage latency. The latter can reduce effective throughput by nearly 30% in long-context scenarios, with optimization headroom greater than discount negotiation.

Q: What is the measured throughput improvement of Mingxin FX100 for inference?
A: In a 480B production deployment: 29% at concurrency 8, 40% at concurrency 16, and 35–36% at the TP4×2 full-node level (measured, reports R2/R3). TTFT p50 reduction is 26–32%.

Q: What metric should compute center selection focus on?
A: Focus on "effective throughput per GPU" rather than raw "per-card compute power." Storage latency directly determines TTFT and throughput; storage optimization reduces concurrency headroom requirements and lowers unit net cost.

References

Uptime Institute Resource Page — https://uptimeinstitute.com/resources
Epoch AI — https://epoch.ai/
Kubernetes Documentation — https://kubernetes.io/docs/home/
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Utilization Is What Matters: A Profit/Loss Comparison of Compute Centers at 30% vs. 60% Utilization

Mingxin Technology — Fri, 21 Aug 2026 07:16:06 +0000

The dividing line between profit and loss for a compute center lies not in installed capacity, but in actual utilization. At 30% versus 60% utilization, the unit cost of compute can differ by more than double, directly determining whether a project falls into a loss-making or profitable range. This article provides a reproducible measurement framework for compute center investment decision-makers, covering three levels: TCO breakdown methodology, break-even logic, and selection constraints.

Why Utilization Is the Primary Variable in Compute Center Profitability

The cost structure of a compute center has one notable characteristic: a very high proportion of fixed costs. Civil works for the data center, power supply and cooling, network infrastructure, and server depreciation—these costs are incurred regardless of whether compute capacity is being called upon. According to industry research from Uptime Institute, data center availability classifications and energy efficiency practices directly constrain the investment levels in facility-side power, cooling, and tiering, and these investments account for a rigid proportion of total costs.

Variable costs are mainly electricity and operations staffing, but even these two items do not drop to zero when compute is idle—equipment in standby still consumes power, and facilities still require personnel on site. This means that for every percentage point utilization drops, the fixed cost allocated to each unit of effective compute rises correspondingly.

Using Mingxin FX100 measured performance as a reference: under a 480B production deployment with long-context cold-recovery workloads, KV tiered acceleration delivers throughput gains of +29–40% (measured, report R2/R3) and reduces first-token latency by 26–32% (measured, report R2). The significance of this data set is that with the same hardware scale, storage acceleration can lift effective throughput by roughly 30–40%. Translated into utilization terms, this means that under the same SLA constraints, the concurrency headroom required to meet targets can be reduced—this directly improves utilization, not merely peak performance.

Cost Structure Differences at 30% vs. 60% Utilization

To understand how utilization affects profit and loss, a TCO breakdown framework must first be established. A reproducible measurement framework should include at least six dimensions: card-hour cost (GPU/accelerator depreciation and maintenance), power cost (including cooling), facility rental or self-built amortization, network bandwidth, storage systems, and operations staffing. Unit costs should be normalized per million tokens, per concurrent request, or per QPS, depending on the business model.

Comparing 30% and 60% utilization, assume fixed cost is F and variable cost grows linearly with utilization. At 30% utilization, the fixed cost allocated per unit of effective compute is F/0.3; at 60% utilization, it is F/0.6. On this single item alone, the unit cost differs by a factor of two. If variable costs account for 30% of total costs, the total unit cost difference is approximately 40%—and this does not yet account for hidden losses at low utilization, such as higher equipment failure rates and compressed maintenance windows.

Mingxin measured data from report R1 provides a quantifiable anchor: training checkpoint save acceleration of 1.9×, with 8-card 32B LoRA full-model snapshots of 65.6GB each reduced from 178s to 94s, and sustained write bandwidth increased from 3.26 to 6.40 GB/s (+96%) (measured, report R1). The direct effect of this type of acceleration is to shorten wait time in the training cluster—GPUs are idle while waiting for checkpoint writes. Halving save time means the idle window for GPUs within each training cycle shrinks, and utilization rises. This is the typical path by which storage-side optimization exerts a positive effect on utilization.

Break-Even Decision Framework: Define Constraints First, Then Optimize

The break-even point of a compute center is not a static number, but a dynamic result under a set of constraints. Decision-makers must first clarify three constraints: SLA requirements (upper and lower bounds for first-token latency and throughput), context length distribution, and concurrency patterns (online inference, batch processing, or mixed workloads). These three constraints determine the lower bound of hardware scale, while utilization determines the degree of match between actual demand and hardware scale.

According to the public benchmark methodology of MLPerf Inference, publicly comparable inference performance tests are submitted under fixed precision and latency constraints, providing a neutral evaluation framework for "who is faster." However, utilization in benchmark tests and utilization in production environments are two different things—benchmark tests typically run at full load, while production environments exhibit clear peaks and valleys. Therefore, selection should not be based solely on peak performance, but rather on effective throughput under target SLA conditions and typical load curves.

Mingxin FX100's acceleration factor of 8.6–20× in no-external-memory-recompute scenarios (measured, report R2) provides an extreme case: the recompute baseline TTFT p50 was 149.5s (conc16), compared to FX100's 11.85s; throughput improved from 4.1 to 74.9 tok/s. In long-context scenarios, if KV Cache tiered management is not used and recompute is chosen instead, GPUs will be in a waiting state for extended periods, severely dragging down utilization. The value of storage acceleration lies not only in shortening individual response times, but in freeing GPUs from waiting so they remain in a sustained state of effective computation.

Selection Criteria: Utilization-Oriented Storage Decisions

From a utilization perspective, storage selection criteria can be summarized in three points. First, whether the storage system can maintain SLA under target concurrency and context length—this requires measured data support, not vendor spec sheets. Second, whether storage optimization covers both inference and training workloads—checkpoint saving and model loading equally affect utilization. Third, whether the deployment form of the storage solution is compatible with existing orchestration systems. According to the official Kubernetes documentation, there are clear mechanisms for resource scheduling and storage volume attachment in inference clusters, and storage solutions need to integrate seamlessly into this system.

Mingxin FX100 measured results on the Huawei Atlas 910B platform show model inference loading acceleration of 6.2–9.3× (vs. NFS baseline): DeepSeek-32B service loading reduced from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×) (measured, report R9, Ascend platform). Model loading is an unavoidable step in service scaling, failure recovery, and version updates—the longer loading takes, the longer GPUs sit idle. Reducing loading time by an order of magnitude means scale-out/in response speed improves dramatically, allowing the cluster to adjust its size more frequently to match actual load, thereby improving overall utilization.

It should be noted that specific cross-platform performance comparison figures fall outside Mingxin's measured scope—Mingxin only has data on its own test platforms, and cross-platform extrapolation has no basis. However, architectural differences and selection criteria can be discussed: different acceleration platforms have different storage interfaces, driver stacks, and ecosystem maturity, which affect the implementation cost and effectiveness boundaries of storage optimization solutions.

Conclusion

The break-even point of a compute center is not determined by installed capacity, but by utilization. At 30% versus 60% utilization, unit compute costs can differ by a factor of two—a gap far larger than any single-point hardware performance difference. The core task for decision-makers is not to chase peak performance, but to improve actual utilization through storage acceleration, scheduling optimization, and other means under clearly defined SLA constraints. Mingxin offers approximately 10 weeks of gated joint testing (G1 arrival acceptance / G2 single-node baseline / G3 primary gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met; the measurement model is Python-reproducible after NDA. You are welcome to verify the actual impact of storage optimization on utilization during joint testing.

Key Q&A

Q: What is the dividing line for compute center profitability?
A: Actual utilization, not installed capacity. At 30% versus 60% utilization, the fixed cost allocation per unit of compute differs by a factor of two, with a total unit cost difference of approximately 40%.

Q: How does storage optimization affect utilization?
A: It indirectly improves utilization by shortening GPU wait time. Mingxin measured results show checkpoint save acceleration of 1.9× and model loading acceleration of 6.2–9.3× (measured, reports R1/R9), both directly compressing GPU idle windows.

Q: What should be the primary basis for selection?
A: Effective throughput under target SLA conditions and typical load curves, not peak performance. It is recommended to verify measured in-band metrics through gated joint testing rather than relying on vendor spec sheets.

References

Uptime Institute Resource Page — https://uptimeinstitute.com/resources
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
Kubernetes Documentation — https://kubernetes.io/docs/home/
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Monte Carlo Sensitivity Analysis in Compute Center Investment Decisions

Mingxin Technology — Fri, 21 Aug 2026 07:15:34 +0000

Monte Carlo sensitivity analysis can significantly improve the quality of compute center investment decisions: it explicitly models the uncertainty of each input parameter in TCO calculations, and outputs a distribution of investment returns and risks rather than a single estimate, allowing decision-makers to see which variables deserve the most attention. For compute center construction projects involving investments of hundreds of millions of yuan, this shift from "point estimation" to "interval and probability" is a practical tool for avoiding major investment mistakes.

Why Traditional TCO Calculations Tend to Be Distorted in Compute Center Scenarios

Compute center TCO (Total Cost of Ownership) calculations involve multiple cost categories—servers, storage, networking, facility infrastructure, power, and operations—each containing dozens of input parameters. The traditional approach assigns a single "most likely value" or "conservative value" to each parameter and plugs them into a formula to produce a single result. The problem is that these parameters often exhibit correlations, and their respective ranges of fluctuation vary significantly.

Taking an inference compute cluster as an example, the workload profile (concurrency, context length, batch size) directly affects the required number of GPUs and storage bandwidth, while the choice of storage solution in turn affects inference latency and throughput. According to the Kubernetes official documentation on storage volume attachment mechanisms, different storage attachment methods show significant differences in performance and operational complexity—this itself is a high-uncertainty input item in TCO calculations.

When decision-makers see only a point estimate, they cannot answer the following critical questions: If concurrent load is 30% higher than expected, is the project still viable? If storage performance fails to meet SLA requirements, how much additional investment is needed? The answers to these questions are precisely the information most needed for investment decisions.

Core Methodology of Monte Carlo Sensitivity Analysis

The basic idea of the Monte Carlo method is not complicated: define a probability distribution for each key input parameter (rather than a single value), then run tens of thousands of random sampling simulations, with each simulation yielding a complete set of TCO and return results. Finally, aggregate all simulation results to form probability distributions of output metrics.

In compute center scenarios, typical input parameters include: GPU utilization, inference workload growth rate, electricity price trends, storage performance attainment probability, and project delay risk. For storage performance specifically, measured data from Mingxin's proprietary test platform can be directly referenced—for example, under a 480B production deployment with long-context cold-recovery workloads, KV hierarchical acceleration delivers inference throughput improvements in the +29–40% range [measured, report R2/R3]; time-to-first-token (TTFT) is reduced by 26–32% [measured, report R2]. Such measured intervals can serve as input ranges for storage solution performance parameters in simulations, which is far more reliable than estimating a fixed value arbitrarily.

Core output metrics typically include: project Net Present Value (NPV) distribution, Internal Rate of Return (IRR) distribution, payback period distribution, and "project attainment probability"—the proportion of simulations where each metric meets preset thresholds. Through sensitivity analysis (such as Sobol indices or regression coefficients), the input parameters contributing most to output variance can also be identified, guiding subsequent due diligence and risk mitigation priorities.

Application Boundaries and Common Pitfalls of the Method

Monte Carlo analysis is not a panacea; its effectiveness depends on two prerequisites: reasonable input distribution settings and a correct model structure. If input distributions are set too optimistically or too conservatively, the output probability distributions will shift systematically, giving decision-makers a false sense of security instead.

Another common pitfall is ignoring correlations between parameters. For example, GPU utilization and inference workload growth are often positively correlated, and electricity prices are related to the energy policies of the facility's location. If these parameters are treated as independent variables in simulations, the output variance will be underestimated, making risks appear smaller than they actually are. According to Epoch AI's public research on AI compute scale and cost trends, the growth of compute demand itself is highly uncertain, further demonstrating that parameter correlation issues cannot be ignored in real-world scenarios.

Additionally, Monte Carlo analysis cannot replace judgment about the business itself. It answers "how results will be distributed given certain assumptions," not "whether those assumptions hold." Decision-makers still need to make independent judgments on macro-level issues such as industry trends and technology roadmap choices.

Implementation Path: A Closed Loop from Calculation to Decision

In practice, a four-step approach is recommended: First, establish a baseline TCO model, clarifying all input parameters and calculation logic. Second, assign probability distributions to key parameters, prioritizing variables with high impact on outputs and high uncertainty. Third, run Monte Carlo simulations (typically 5,000+ iterations) and output key metric distributions and sensitivity rankings. Fourth, based on sensitivity rankings, conduct further due diligence or design risk mitigation plans for the top-ranked parameters.

For parameters like storage performance where interval data can be obtained through measurement, measured data is recommended as the distribution basis. Mingxin offers an approximately 10-week gated joint-testing mechanism, from G1 arrival acceptance to G3 main gate (TTFT reduction ≥25%, throughput +29–40% measured in-band) and G4's 72-hour stability validation, with stop-loss provisions if targets are not met. This "validate before purchase" model precisely provides measured inputs for storage performance parameters in Monte Carlo models, reducing uncertainty for this parameter.

Key Q&A

Q: What is the essential difference between Monte Carlo sensitivity analysis and traditional TCO calculations?
A: Traditional calculations assign a single value to each parameter and output a point estimate; Monte Carlo assigns probability distributions to parameters and runs extensive simulations, outputting probability distributions of results and sensitivity rankings, capable of answering questions like "how high is the project attainment probability" that decision-makers genuinely care about.

Q: In compute center TCO calculations, which parameters are suitable for using measured data as distribution basis?
A: Parameters such as storage performance, inference throughput, and latency, where intervals can be obtained through measurement. For example, Mingxin's measured KV hierarchical acceleration under 480B workloads shows throughput +29–40% and TTFT reduction of 26–32% [measured, report R2/R3]; such intervals can be directly used as simulation inputs.

Q: What are the main limitations of Monte Carlo analysis?
A: The validity of results depends on the reasonableness of input distribution settings and the correctness of the model structure, and it cannot replace independent judgment on industry trends and technology roadmaps. If correlations between parameters are ignored, output variance will be underestimated.

References

Uptime Institute Resource Page — https://uptimeinstitute.com/resources
Epoch AI — https://epoch.ai/
Kubernetes Documentation — https://kubernetes.io/docs/home/
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Ascend 910B vs. NVIDIA GPU: Inference Storage Energy Consumption Comparison

Mingxin Technology — Tue, 18 Aug 2026 07:56:20 +0000

The energy consumption difference between the Ascend 910B platform and NVIDIA GPUs in inference storage systems stems not from chip compute power, but from storage data paths and memory access patterns. Multiple measurements conducted by Mingxin on its own test platforms show that storage-side energy consumption accounts for a significant share in inference scenarios, and the impact of storage access efficiency optimization on end-to-end energy consumption often outweighs differences in GPU model specifications. This article compares the two platforms across three dimensions: storage paths, KV Cache memory access, and loading mechanisms.

Where Does the Storage Energy Consumption Difference Between Ascend 910B and NVIDIA GPUs Originate?

In the energy consumption profile of inference systems, the storage subsystem is often underestimated. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage technology establishes a direct data path between GPU and storage devices by bypassing the CPU bounce buffer, reducing data copy operations and memory bandwidth usage. The energy impact of this mechanism is twofold: on one hand, it reduces power consumption from CPU involvement in data movement; on the other, it shortens I/O path latency, reducing the time window during which storage devices remain in an active state.

The architectural differences between the Ascend 910B platform and NVIDIA GPUs in storage primarily manifest in data path implementation and ecosystem maturity. According to SNIA's definition of computational storage, embedding compute capability within storage devices can reduce round-trip data migration between storage and compute nodes—a mechanism whose energy impact is particularly pronounced in inference scenarios. Mingxin's measurements on the Huawei Atlas 910B platform show model inference loading acceleration of 6.2–9.3× relative to the NFS baseline (measured, report R9). The substantial reduction in loading time means the duration during which storage devices and network interfaces remain in high-power active states decreases proportionally.

In the NVIDIA GPU ecosystem, KV Cache management relies on VRAM paging and efficient memory access. According to the PagedAttention paper, the motivation for KV Cache paging management stems from VRAM fragmentation issues, and this mechanism enables vLLM to push VRAM utilization close to theoretical limits. Higher VRAM utilization directly reduces the frequency of KV Cache swapping to and from storage devices, and each swap operation incurs power overhead on both the PCIe bus and storage devices.

KV Cache Memory Access Energy: Measured Comparison Between Ascend 910B and NVIDIA GPUs

The impact of KV Cache memory access patterns on inference energy consumption is significantly amplified in long-context scenarios. According to the FlashAttention paper, the bottleneck in attention computation is fundamentally HBM bandwidth rather than compute power, and I/O-aware optimization can substantially reduce memory access energy. This conclusion applies equally to the Ascend 910B and NVIDIA GPUs, but differences in HBM bandwidth and memory hierarchy design between the two lead to different energy profiles when KV Cache spills to external storage.

Mingxin's measured data for FX100 on the AMD MI308X platform serves as a reference for understanding KV Cache memory access energy. Under a 480B production deployment with long-context cold-recovery load, KV hierarchical acceleration improved inference throughput by 29–40% (measured, reports R2/R3) and reduced first-token latency by 26–32% (measured, report R2). Higher throughput means shorter storage access time per token, reducing the total duration storage devices remain in active states, thereby lowering per-token storage energy consumption.

The acceleration factor for scenarios without external memory recomputation reaches 8.6–20× (measured, report R2). This data reveals the extreme case of storage path energy: when KV Cache fully spills to external storage without optimization, the recomputation baseline TTFT p50 reaches 149.5 seconds (at 16 concurrent streams), compared to FX100's 11.85 seconds. Long latency means GPUs remain idle but not in sleep state while waiting for data—energy waste in this state is often overlooked.

Loading and Checkpoint Energy: Practical Gains from Storage Path Optimization

Model loading and training checkpoint saving represent another important scenario for storage energy consumption. Mingxin's measurements on the Huawei Atlas 910B platform show DeepSeek-32B service loading reduced from 691 seconds to 112 seconds (6.2× acceleration), and DeepSeek-70B from 1399 seconds to 150 seconds (9.3× acceleration) (measured, report R9). Shorter loading times directly reduce the high-power active duration of storage devices and network interfaces.

For training checkpoint saving, in an 8-card 32B LoRA scenario, the save time for each 65.6 GB full-model snapshot dropped from 178 seconds to 94 seconds (1.9× acceleration), with sustained write bandwidth improving from 3.26 to 6.40 GB/s (+96%) (measured, report R1). Higher write bandwidth means storage devices complete data writes in less time, reducing the total duration of high-power states.

According to public research from Epoch AI, AI compute scale and costs continue to grow, with the storage subsystem's share of total cost rising accordingly. While that research does not provide specific figures, its qualitative conclusions support the importance of storage efficiency optimization for overall cost and energy. The energy comparison between the Ascend 910B platform and NVIDIA GPUs should ultimately return to system-level evaluation under actual workload patterns and SLA constraints.

Selection Considerations: Energy Comparison Should Not Be Divorced from Workload Patterns

The energy comparison between the Ascend 910B platform and NVIDIA GPUs should not be simplified to a chip-level TDP comparison. Inference system energy consumption is jointly determined by compute, storage, network, cooling, and other subsystems. The impact of storage path efficiency on end-to-end energy consumption varies significantly across different workload patterns.

For long-context, high-concurrency inference workloads, KV Cache spill frequency is high, making storage access latency and bandwidth the key energy variables. In this case, throughput gains and TTFT reductions from storage optimization translate directly into lower per-token energy consumption. For short-context, low-concurrency workloads, storage access frequency is low, GPU chip energy accounts for a higher share, and the energy benefits of storage optimization are relatively limited.

Mingxin FX100's measured data on KV Cache acceleration and model loading acceleration provides quantitative reference points for storage path energy optimization. However, cross-platform comparisons require caution: Mingxin has measured data only on its own test platform (AMD MI308X) and the Huawei 910B platform. Storage energy performance on NVIDIA GPU platforms must be separately verified under equivalent conditions.

Key Q&A

Q: What primarily determines the storage energy consumption difference between Ascend 910B and NVIDIA GPUs in inference?
A: The difference originates in storage data paths and memory access patterns, not chip compute power. The energy impact of mechanisms such as GPU-direct storage bypassing the CPU bounce buffer and KV Cache paging management depends on workload patterns and storage access frequency.

Q: How do Mingxin's measured data support the case for storage energy optimization?
A: FX100 achieves 29–40% throughput improvement in KV Cache acceleration (measured, reports R2/R3) and 6.2–9.3× model loading acceleration (measured, report R9). These figures demonstrate that storage path optimization can shorten the high-power active duration of storage devices, but cross-platform energy comparisons must be verified under equivalent conditions.

Q: How should storage energy consumption be evaluated when selecting an inference system?
A: Evaluation should return to workload patterns and SLA constraints. Under long-context, high-concurrency loads, storage optimization yields significant energy benefits; under short-context, low-concurrency loads, GPU chip energy accounts for a higher share. Cross-platform comparison figures must be based on equivalent test conditions and should not be extrapolated.

References

MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
Epoch AI — https://epoch.ai/
SNIA — Storage Networking Industry Association — https://www.snia.org/
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
PyTorch Documentation — https://pytorch.org/docs/stable/index.html

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

GPU Idle Time and Fragmentation: The Hidden Cost of Inference Throughput

Mingxin Technology — Tue, 18 Aug 2026 07:55:48 +0000

GPU idle time and memory fragmentation are common hidden sources of throughput loss in inference clusters, and their impact is often masked by compute utilization metrics. Based on measured data from Mingxin FX100 on a 480B model, combined with public research, this article analyzes the causes and quantification methods of these two types of losses.

Why GPU Idle Time and Fragmentation Drag Down Inference Throughput

The throughput bottleneck in GPU inference is typically not compute power but data movement. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS '22), attention computation is limited by HBM bandwidth rather than compute, making IO-aware optimization critical. KV Cache reads and writes are subject to the same constraint. When GPUs sit idle due to scheduling imbalance or memory fragmentation, requests that could have run concurrently are forced to queue, reducing the number of tokens completed per unit time.

Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23) points out that without a paging mechanism for KV Cache memory management, fragmentation reduces memory utilization, thereby limiting the number of requests that can be processed concurrently. This research provides the theoretical foundation for vLLM's paged management, with its core motivation being to reduce memory waste caused by fragmentation. This mechanism-level analysis aligns with what Mingxin observed on the 480B model: when memory fragmentation is severe, GPU compute units remain in a waiting state, and throughput falls significantly below theoretical peak.

Measured Data: How Fragmentation and Idle Time Affect End-to-End Latency

Measured results from Mingxin FX100 in a 480B production deployment configuration show that under long-context cold-restore workloads, KV tiered acceleration improves inference throughput by 29–40% (concurrency 8 tier: lower bound +29%; concurrency 16 tier: optimal operating point +40%; TP4×2 full-system basis: +35–36%; source: measured, reports R2/R3). The root cause of this improvement: after offloading KV Cache from GPU memory to an NVMe-oF array, memory fragmentation pressure is transferred to external storage, allowing GPU compute units to remain continuously saturated.

Time-to-first-token (TTFT) improvements are equally significant. Under 480B·TP8 with three concurrency tiers, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% (source: measured, report R2). In the baseline without external recomputation, the recompute baseline TTFT p50 was 149.5s (concurrency 16), compared to FX100's 11.85s, achieving an 8.6–20× speedup; throughput increased from 4.1 tok/s to 74.9 tok/s (source: measured, report R2). These figures demonstrate that when GPUs are forced to recompute or wait due to insufficient KV Cache capacity, idle time translates directly into user-perceivable latency.

Metric	Baseline (no external recompute)	FX100 measured	Improvement	Source
TTFT p50 (concurrency 16)	149.5s	11.85s	8.6–20×	Measured, R2
Throughput (concurrency 16)	4.1 tok/s	74.9 tok/s	—	Measured, R2
TTFT p50 (three concurrency tiers)	10.17–35.73s	7.53–26.35s	↓26–32%	Measured, R2
Inference throughput (480B cold restore)	—	—	+29–40%	Measured, R2/R3

How to Quantify the True Cost of Idle Time and Fragmentation

To assess the cost of GPU idle time and fragmentation, we recommend evaluating from the following three perspectives:

First, normalize by concurrency profile. Under the same SLA, a TTFT reduction within the measured band means the concurrency headroom required to meet the target can be lowered. For example, if the baseline requires 16 concurrent requests to satisfy the TTFT constraint and the optimized system meets it at 8, the proportion of idle GPUs decreases accordingly. This logic can be read directly from the concurrency-latency curves in the R2 measurements.

Second, normalize per million tokens. Inference cost should be broken down into six components: GPU-hours, electricity, facility, network, storage, and operations. GPU idle time means GPU-hour costs are spread across fewer outputs, raising the per-token cost. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv), the KV Cache-centric disaggregated compute-storage architecture reduces GPU idle waiting precisely through cross-node KV pooling.

Third, match storage bandwidth. When KV Cache is offloaded to external storage, storage bandwidth must match the GPU's consumption rate. Under the LMCache parallel read patch, Mingxin FX100 achieved a TTFT reduction from 37.97s to 9.30s on a single GPU at concurrency 16 with cold disk reads (Qwen2.5-32B), with bandwidth improving from 0.98 GB/s to 5.23 GB/s (↑5.3×, source: measured, report R1). If storage bandwidth is insufficient, GPUs will still sit idle waiting for data.

Conclusion and Recommendations

The cost of GPU idle time and fragmentation is fundamentally a matching problem between memory management strategy and storage bandwidth. PagedAttention's paging mechanism addresses memory fragmentation but not capacity limits; KV tiered acceleration shifts the capacity pressure to external storage, thereby freeing GPU compute resources. For procurement and budget decision-makers, we recommend using "concurrency headroom under the same SLA" as the selection constraint rather than comparing peak compute alone. Mingxin offers a gated joint testing program of approximately 10 weeks (G1: arrival acceptance / G2: single-node baseline / G3: primary gate: TTFT reduction ≥25%, throughput within the +29–40% measured band / G4: 72-hour stability), which can validate the quantification methods above under real workloads, with stop-loss provisions if targets are not met.

Key Q&A

Q: How does GPU fragmentation affect inference throughput?
A: Fragmentation reduces memory utilization, limits the number of concurrent requests, and leaves GPU compute units idle while waiting for data. PagedAttention's paging management can mitigate fragmentation, but capacity limits still require external storage to resolve.

Q: What is the measured improvement of Mingxin FX100 on the 480B model?
A: With KV tiered acceleration, inference throughput improves by 29–40% (concurrency 8 tier: +29%; concurrency 16 tier: +40%; TP4×2 full-system basis: +35–36%); TTFT is reduced by 26–32% (source: measured, reports R2/R3).

Q: How can the hidden cost of GPU idle time be assessed?
A: Normalize by concurrency profile (concurrency headroom required under the same SLA), normalize per million tokens (GPU-hour cost dilution), and match storage bandwidth (storage must keep pace with GPU consumption rate). Specific figures require joint testing under real workloads for validation.

References

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Building a Domestic AI Inference Storage Acceleration Ecosystem

Mingxin Technology — Tue, 18 Aug 2026 07:55:17 +0000

Building a domestic AI inference storage acceleration ecosystem requires coordinated advancement across three layers: hardware, software, and standards. At the hardware layer, the disaggregated storage-compute architecture based on NVMe-oF and RoCEv2 has become the mainstream choice; at the software layer, acceleration optimizations targeting KV Cache are emerging as the key breakthrough for inference performance; at the standards layer, adaptation of domestic operating systems and storage interfaces is a necessary prerequisite for ecosystem deployment. Measured data from the Mingxin FX100 under long-context workloads with a 480B model demonstrates that this path delivers quantifiable performance gains.

Hardware Layer: Disaggregated Storage-Compute and High-Speed Interconnect as the Foundation

Building a domestic AI inference storage acceleration ecosystem first requires solving the data path problem. Traditional local NVMe direct-attached solutions face bottlenecks of low resource utilization and limited scalability in GPU cluster scenarios. According to the NVIDIA GPUDirect Storage Documentation, GPU direct-attached storage technology establishes a direct data path between GPUs and storage devices by bypassing the CPU's bounce buffer—a mechanism that provides the hardware foundation for disaggregated storage-compute architectures.

In the domestic context, this architecture must adapt to the specific constraints of domestic servers and operating systems. Official project information from domestic operating systems such as openEuler shows that the Xinchuang software/hardware stack has formed a complete hierarchy from kernel to user space, and storage acceleration solutions must be deeply compatible with this stack. The Mingxin FX100 employs an all-flash NVMe-oF array with RoCEv2 networking, and has completed systematic validation on an 8-GPU AMD MI308X platform. This configuration path can serve as a reference baseline for domestic substitution.

Software Layer: KV Cache Acceleration as the Optimal Entry Point

The core bottleneck in software-level inference storage acceleration optimization lies in KV Cache memory access. According to the FlashAttention paper, attention computation is limited by HBM bandwidth rather than compute capacity, and this IO-aware optimization logic applies equally to the external memory tiering of KV Cache. When KV Cache size exceeds GPU memory capacity, tiering and offloading it to remote storage with efficient readback becomes the key to improving long-context inference throughput.

Measured data from Mingxin report R2 shows that under long-context cold-restore workloads in a 480B production deployment configuration, KV tiering acceleration delivers inference throughput gains of +29–40%: +29% at concurrency level 8 (lower bound), +40% at the optimal operating point of concurrency 16 (upper bound), and +35–36% at the full-machine TP4×2 level. Time-to-first-token (TTFT) decreases by 26–32% under 480B·TP8 across three concurrency levels, with p50 dropping from 10.17–35.73s to 7.53–26.35s. This gain comes directly from fast KV Cache readback capability rather than simple cache hits—against a baseline without external memory recomputation, the speedup factor reaches 8.6–20× (recompute baseline TTFT p50 is 149.5s, compared to FX100's 11.85s; throughput of 4.1 vs. 74.9 tok/s).

Metric	Baseline (no external recompute)	FX100 measured	Improvement	Source
TTFT p50 (conc16)	149.5s	11.85s	8.6–20×	R2 measured
Throughput (conc16)	4.1 tok/s	74.9 tok/s	18.3×	R2 measured
Inference loading (vs NFS, DeepSeek-70B)	1399s	150s	9.3×	R9 measured (Ascend platform)
Checkpoint save (8-GPU 32B LoRA)	178s	94s	1.9×	R1 measured

Another key software-layer optimization is KV Cache paging management. According to the PagedAttention paper, paged management of KV Cache resolves GPU memory fragmentation issues and significantly improves memory utilization. Mingxin's measurements on the LMCache parallel-read patch further confirm this direction: under single-GPU concurrency-16 cold disk-read scenarios (Qwen2.5-32B), TTFT drops from 37.97s to 9.30s (4.1× improvement), and bandwidth increases from 0.98 to 5.23 GB/s (5.3×). This indicates that parallel read optimization on the storage side and paging management on the GPU memory side must be combined to unlock full performance.

Standards and Ecosystem: Neutral Benchmarks and Reproducible Validation Are Both Essential

The third layer of ecosystem building is the standards and validation system. Inference performance comparisons must rely on neutral, public benchmarks. According to the MLPerf Inference: Datacenter Benchmark Suite Results, this evaluation requires submissions under fixed precision and latency constraints, making it the only neutral public reference for discussing inference performance. Performance claims from domestic storage acceleration solutions should proactively align with the methodology of such public benchmarks rather than self-defined metrics.

At the same time, design trade-offs in disaggregated storage-compute architectures require industry consensus. According to the Mooncake paper, the KVCache-centric disaggregated storage-compute architecture involves design trade-offs between prefix cache reuse and cross-node KV pooling, and these trade-offs directly affect the applicability boundaries of storage acceleration solutions. SNIA's definitions of storage tiering and computational storage provide the terminology and framework foundation for this discussion.

Mingxin employs an approximately 10-week gate-based validation process in joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. The value of this mechanism lies in converting performance claims into reproducible measured data rather than relying on vendor self-reported figures. For procurement and budget decision-makers, this is the most critical aspect to evaluate when assessing domestic solutions.

Conclusion

Building a domestic AI inference storage acceleration ecosystem is not a matter of replacing a single product, but rather the coordinated deployment of three layers: hardware paths, software optimization, and standards validation. Measured data from the Mingxin FX100 demonstrates that KV Cache tiering acceleration delivers 29–40% throughput gains and 26–32% TTFT reductions in long-context inference scenarios, with these benefits validated at the 480B model scale. To verify the suitability of this architecture on your own platform, reproducible measurement models can be obtained through the joint testing mechanism.

Key Q&A

Q: What is the core path for building a domestic AI inference storage acceleration ecosystem?
A: At the hardware layer, adopt a disaggregated storage-compute architecture using NVMe-oF and RoCEv2; at the software layer, focus on KV Cache tiering and parallel read optimization; at the standards layer, validate performance claims through neutral benchmarks and gate-based joint testing. The three layers advance in coordination.

Q: What are the measured gains of KV Cache acceleration in long-context inference?
A: Mingxin R2 measurements show throughput gains of 29–40% under long-context cold-restore workloads with a 480B model (+29% at concurrency 8, +40% at concurrency 16), TTFT reductions of 26–32%, and speedup factors of 8.6–20× against the baseline without external memory recomputation.

Q: How can the real performance of domestic storage acceleration solutions be validated?
A: Use a gate-based joint testing process with TTFT reduction ≥25% and throughput +29–40% measured in-band as core gates, supplemented by 72-hour stability validation, ensuring performance claims are reproducible and auditable.

References

NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
openEuler | OS for Digital Infrastructure — https://www.openeuler.org/en/
Epoch AI — https://epoch.ai/
SNIA — Storage Networking Industry Association — https://www.snia.org/
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

GPU Compute Platform Acceptance: From Benchmark Testing to Stability Stress Testing

Mingxin Technology — Tue, 18 Aug 2026 07:16:43 +0000

GPU compute platform acceptance cannot be completed by running a single benchmark pass before delivery. A complete acceptance process should cover three layers: single-card and cluster baseline performance, acceleration effectiveness validation under real workloads, and long-duration stability stress testing. This article presents a reusable acceptance framework and gate criteria based on Mingxin Technology's validated process on the AMD Instinct MI308X platform.

Why Passing Benchmarks Does Not Equal Platform Qualification

The challenge in accepting a GPU compute platform lies in the fact that high single-card benchmark scores do not guarantee strong cluster inference performance. Inference performance is constrained by the coordination of VRAM capacity, interconnect bandwidth, storage systems, and scheduling frameworks. According to the definitions in MLPerf Inference: Datacenter Benchmark Suite Results, inference benchmarks require submissions under fixed precision and latency constraints, providing a neutral basis for determining "which is faster"—but MLPerf measures model inference itself and does not cover storage-related aspects such as external KV Cache offloading or checkpoint saving.

The platform used by Mingxin in R1–R4 testing consisted of 8× AMD Instinct MI308X (192 GB HBM per card) + 2× AMD EPYC 9654, running on ROCm 7.2 with vLLM 0.20.1+rocm721. Testing revealed that when model weights reached 450 GB (Qwen3-Coder-480B-FP8), a local NVMe single drive used as the KV Cache baseline became a clear bottleneck—a blind spot that benchmark testing does not cover.

Acceptance Layer 1: Baseline Performance and Acceleration Effectiveness Validation

The first step in acceptance is confirming hardware specifications meet requirements. According to the official specification sheet in H100 GPU | NVIDIA, VRAM type, capacity, and interconnect topology are mandatory acceptance checks; however, meeting specifications is only the starting point—the key lies in performance under real workloads.

Under the 480B production deployment long-context cold-restore workload, Mingxin FX100 delivered a +29–40% improvement in KV-tiered accelerated inference throughput (concurrency level 8 at +29% as the lower bound, optimal operating point at concurrency level 16 at +40% as the upper bound, TP4×2 full-system basis at +35–36%, source: measured, reports R2/R3). Time-to-first-token (TTFT) decreased by 26–32%, with p50 dropping from 10.17–35.73s to 7.53–26.35s across three concurrency levels on 480B·TP8 (source: measured, report R2).

Metric	Baseline (local NVMe)	Mingxin FX100	Improvement	Source
Inference throughput (tok/s)	4.1	74.9	18.3×	Measured, R2
TTFT p50 (conc16)	149.5s	11.85s	12.6×	Measured, R2
Checkpoint save (8-card 32B)	178s	94s	1.9×	Measured, R1

Training-side validation is equally necessary. R1 measurements showed that full-model snapshot saving for 8-card 32B LoRA at 65.6 GB per copy dropped from 178s to 94s, with sustained write bandwidth increasing from 3.26 to 6.40 GB/s (+96%). If the platform is used for mixed inference and training workloads, checkpoint save acceleration should be included as an acceptance item.

Acceptance Layer 2: Stability Stress Testing and Gate Criteria

After benchmarks pass, the process must proceed to stability stress testing. Mingxin employs an approximately 10-week gated joint-testing process with four gate nodes in sequence: G1 arrival acceptance, G2 single-node baseline, G3 primary gate (TTFT reduction ≥25%, throughput within the +29–40% measured band), and G4 72-hour stability run. The key principle of this process is "stop loss when criteria are not met"—each gate has clear quantitative criteria, preventing acceptance from dragging into endless debugging.

Stress testing duration should be no less than 72 hours, covering three workload patterns: cold start, peak concurrency, and long context. In R2 measurements, TTFT data across three concurrency levels on 480B·TP8 was fully recorded and can serve as a comparison baseline during stress testing. Stability criteria should include: throughput fluctuation range, TTFT tail latency (p99), no OOM, and no kernel crashes.

Acceptance Layer 3: Cost and Selection Criteria Confirmation

Acceptance is not only technical validation but also calibration of the cost model. According to the billing frameworks in EC2 On-Demand Instance Pricing and Pricing - Linux Virtual Machines | Microsoft Azure, public cloud GPUs are billed hourly with pricing differentiated by instance family; per the mechanism described in VM instance pricing | Google Cloud, committed-use discounts can reduce unit costs. These public pricing pages provide a basis for understanding cost structures, but specific values change frequently—acceptance should reference current quotes at the time of validation.

Per the classification framework in Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud, different workloads require different instance families—inference, training, and rendering have significantly different hardware requirements. Acceptance should confirm that platform selection matches the workload rather than simply pursuing peak compute capacity.

Key Points Q&A

Q: What are the minimum stages required for GPU compute platform acceptance?
A: Three layers: baseline performance validation, real-workload acceleration effectiveness validation, and stability stress testing of 72+ hours. Each layer should have quantitative gate criteria.

Q: What is the measured improvement of Mingxin FX100 in 480B model inference?
A: KV-tiered acceleration improves inference throughput by +29–40% (measured, R2/R3), with TTFT reduced by 26–32% (measured, R2). Acceleration over the no-external-memory-recompute baseline is 8.6–20×.

Q: How should gate criteria for stability stress testing be set?
A: Reference Mingxin's gated joint testing: TTFT reduction ≥25%, throughput within the +29–40% measured band, and 72 hours without anomalies. Stop loss when criteria are not met.

Mingxin Technology specializes in storage acceleration and domestic compute platforms, offering the FX100/FX200/FX300 series all-flash NVMe-oF arrays and gated joint-testing services. To validate platform performance in KV Cache tiering, checkpoint saving, and other areas, each item can be confirmed against the G1–G4 gates during joint testing.

References

Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
Epoch AI — https://epoch.ai/
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
H100 GPU | NVIDIA — https://www.nvidia.com/en-us/data-center/h100/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Optimization Paths and Measured Results for LLM Inference Under Insufficient VRAM

Mingxin Technology — Tue, 18 Aug 2026 07:16:11 +0000

When LLM inference encounters insufficient VRAM, optimization paths can be grouped into three categories: offloading the KV Cache to external storage, sharding and offloading model states, and restructuring attention computation from an I/O-aware perspective. These approaches are not mutually exclusive and are often combined in practice. Mingxin's FX100, in a 480B production deployment, achieved a 29–40% inference throughput improvement through tiered KV acceleration (measured, report R2)—a quantitative example of the offloading path.

The Essence of the VRAM Bottleneck: Why Compute Is Often Idle

The VRAM pressure in LLM inference stems from the attention mechanism's continuous occupation of intermediate states (KV Cache). According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the bottleneck in attention computation is HBM bandwidth rather than compute—meaning that when insufficient VRAM causes frequent KV Cache swapping, GPU compute sits idle waiting on data. The I/O-aware optimization proposed in that paper approaches the compute ceiling precisely by reducing HBM access counts.

Data from Mingxin on an AMD MI308X ×8 platform (measured, report R2) corroborates this: under TP8 with three concurrency levels on a 480B model, time-to-first-token (TTFT) p50 dropped from 10.17–35.73s to 7.53–26.35s, a 26–32% reduction. The direct source of this latency reduction is that the KV Cache, once evicted from VRAM, no longer needs to be recomputed from scratch.

KV Cache Offloading: From Recompute to Tiered Storage

When VRAM cannot hold the full KV Cache, the common practice is to discard and recompute. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache solves VRAM fragmentation but not the capacity ceiling. Mingxin FX100 measurements show the recompute baseline TTFT p50 at 149.5s (concurrency 16), while FX100 achieves 11.85s; throughput rises from 4.1 tok/s to 74.9 tok/s (measured, report R2)—a speedup of 8.6–20×.

Metric	Recompute Baseline	FX100	Improvement	Source
TTFT p50 (concurrency 16)	149.5s	11.85s	12.6×	Measured, R2
Throughput (concurrency 16)	4.1 tok/s	74.9 tok/s	18.3×	Measured, R2
480B inference throughput (concurrency 8)	—	—	+29%	Measured, R2
480B inference throughput (concurrency 16)	—	—	+40%	Measured, R2

FX100's tiered KV acceleration keeps hot data in VRAM and offloads cold data to an NVMe-oF array, accessed via RoCEv2 networking at 100GbE per port. With the LMCache parallel read patch, in a single-GPU concurrency-16 cold-read-from-disk scenario (Qwen2.5-32B), TTFT dropped from 37.97s to 9.30s, and bandwidth rose from 0.98 GB/s to 5.23 GB/s (measured, report R1). This indicates that the benefit of the offloading path depends on whether storage-side bandwidth can match GPU consumption speed.

State Sharding and Offloading: Migrating Training-Side Experience to Inference

Another optimization path for insufficient VRAM comes from the training domain. According to ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, VRAM usage in large-model training can be divided into parameters, gradients, optimizer states, and intermediate activations. ZeRO distributes or offloads these states across multiple devices or external storage via sharding and offloading. Although that work targets training, its "state sharding" concept has migrated to inference—sharding model weights or KV Cache across multiple GPUs and, when necessary, offloading to CPU memory or NVMe.

Mingxin measurements on the Ascend platform (report R9) show model loading acceleration of 6.2–9.3× relative to an NFS baseline: DeepSeek-32B service loading dropped from 691s to 112s, and DeepSeek-70B from 1399s to 150s. The essence of loading acceleration is storage-side throughput gains, not compute optimization—which suggests that when selecting solutions, one should distinguish between "insufficient VRAM" and "storage bottleneck" as separate problems.

Disaggregated Compute-Storage Architecture: A System-Level View of KV Cache Pooling

If the perspective widens from a single machine to a cluster, the insufficient-VRAM problem becomes a resource scheduling problem. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated architecture turns VRAM from "private per GPU" into "shared across the cluster" through prefix cache reuse and cross-node KV pooling. This architecture complements Mingxin FX100's offloading path: the former addresses cross-machine sharing, the latter addresses tiering within a single machine.

Optimization Dimension	Representative Approach	Applicable Scenario	Source
KV Cache offloading	Mingxin FX100 tiered acceleration	Long context, high concurrency	Measured, R2
State sharding	ZeRO concept migration	Multi-GPU parallelism, weights exceeding single GPU	arXiv:1910.02054
Attention I/O optimization	FlashAttention	Short context, compute-bound	arXiv:2205.14135
Cluster-level KV pooling	Mooncake architecture	Multi-machine sharing, prefix reuse	arXiv:2407.00079

It should be noted that the external sources cited above are qualitative conclusions; Mingxin has not reproduced their quantitative benefits on its own platforms. Cross-platform comparisons fall outside the scope of measured results—for example, statements like "Ascend is X% faster than a certain GPU" are not supported by Mingxin data. When selecting solutions, public benchmarks such as MLPerf should serve as the neutral basis (per the metric definitions in MLPerf Inference: Datacenter Benchmark Suite Results).

Selection Criteria: Define Constraints First, Then Choose the Path

There is no silver bullet for insufficient VRAM. The decision sequence should be: first define the SLA (TTFT ceiling, throughput floor) and context length; then determine whether the bottleneck is VRAM capacity or storage bandwidth; finally choose the offloading, sharding, or pooling path. Mingxin FX100's measured envelope (TTFT reduction ≥25%, throughput +29–40%) can serve as a reference baseline for the offloading path, but actual gains depend on workload shape—in short-context, low-concurrency scenarios, offloading benefits may be less than computing directly in VRAM.

Conclusion

Optimizing for insufficient VRAM is fundamentally a trade-off of "storage for VRAM, bandwidth for latency." Mingxin FX100's tiered KV acceleration provides reproducible measured evidence, and its roughly 10-week gated joint testing cycle (from G1 arrival acceptance to G4 72-hour stability) allows users to validate benefits on their own workloads, with the option to stop if targets are not met. To evaluate the offloading path under specific models and concurrency profiles, validation can be performed within the joint testing framework.

Key Q&A

Q: What is the most effective optimization strategy for LLM inference under insufficient VRAM?
A: Offloading the KV Cache to external storage is the most directly beneficial path. Mingxin FX100 measured a 29–40% throughput improvement on a 480B model (measured, report R2) and a 26–32% TTFT reduction.

Q: How much faster is KV Cache offloading compared to recomputation?
A: In Mingxin measurements, the recompute baseline TTFT p50 was 149.5s, while FX100 achieved 11.85s—a speedup of 8.6–20× (measured, report R2). The benefit depends on whether storage bandwidth can match GPU consumption speed.

Q: Do these optimization strategies apply to all inference scenarios?
A: No. In short-context, low-concurrency scenarios, offloading benefits are limited, and cross-platform comparisons should rely on neutral benchmarks such as MLPerf. Selection should first define SLA and context constraints, then determine whether the bottleneck is VRAM or storage.

References

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — https://arxiv.org/abs/1910.02054
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Comparing Four Approaches to Reducing LLM TTFT

Mingxin Technology — Tue, 18 Aug 2026 07:15:39 +0000

The first-token latency (TTFT) of large language model inference is a direct metric affecting user experience and SLA compliance. The paths to reducing TTFT are not singular but span four layers: compute, GPU memory, storage, and architecture. This article compares four mainstream approaches and cites measured data from Mingxin Technology on the AMD MI308X platform to illustrate the practical benefits of storage-side acceleration in long-context scenarios.

Why TTFT Is the Primary Target of Inference Optimization

TTFT refers to the time interval from when a user sends a request to when the first output token is received. In interactive applications, an excessively long TTFT directly degrades the user experience; in batch processing scenarios, it determines whether a request can complete within the SLA deadline. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the core bottleneck of attention computation lies in HBM bandwidth rather than compute capacity, which provides a foundation for understanding the composition of TTFT: model weight reads, KV Cache reads, and the computation itself are all constrained by data movement speed.

The composition of TTFT can be broken down into: request queuing, weight loading (if swapping is required), prefill computation, and KV Cache writes and reads. Among these, the prefill phase must process all input tokens, requiring substantial computation and memory access. As context length grows, the KV Cache size expands accordingly, and memory access pressure becomes the dominant factor.

Approach 1: Compute and Operator Optimization

FlashAttention reduces HBM read/write operations through I/O-aware attention computation and is currently the most fundamental optimization technique. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, vLLM's PagedAttention manages the KV Cache through paging to reduce GPU memory fragmentation and improve batch throughput. These methods operate at the compute and GPU memory management layers, and their improvement to TTFT is bounded by hardware bandwidth limits.

The advantage of this approach is its broad applicability without requiring hardware changes; the drawback is that when the KV Cache exceeds GPU memory capacity, it must rely on external storage or recomputation, at which point the memory access bottleneck shifts from HBM to PCIe and network links.

Approach 2: GPU Memory Expansion and KV Cache Tiering

When model weights and KV Cache exceed single-GPU memory capacity, a common practice is to offload the KV Cache to CPU memory or remote storage. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture reduces redundant prefill overhead through prefix cache reuse and cross-node KV pooling. The benefit of such architectures lies in improved GPU memory utilization, but the cost is added data-path latency.

Mingxin's FX100 KV tiering acceleration solution falls into this category: cold KV Cache is tiered onto NVMe-oF arrays while hot data remains in GPU memory. According to Mingxin's measured report R2 (2026-07-05, 480B·TP8 long-context·production release), with a 480B model and TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. This data indicates that in long-context scenarios, storage-side latency contributes non-negligibly to TTFT.

Approach 3: Storage Acceleration and Data-Path Optimization

GPU Direct Storage (GDS) technology bypasses the CPU's bounce buffer, allowing GPUs to access storage devices directly and reducing data movement latency. According to NVIDIA GPUDirect Storage Documentation, this technology suits large-block data transfer scenarios but offers limited improvement for small I/O operations. Mingxin's FX100 measured data demonstrates the potential of storage acceleration:

Metric	Baseline (local NVMe)	FX100 array	Improvement	Source
TTFT p50 (conc16, no external recompute)	149.5s	11.85s	12.6×	Measured, report R2
Throughput (conc16)	4.1 tok/s	74.9 tok/s	18.3×	Measured, report R2
Model loading (DeepSeek-70B, Ascend 910B)	1399s	150s	9.3×	Measured, report R9

According to Mingxin's measured report R2, the acceleration factor for scenarios without external recompute is 8.6–20×. This implies that in extreme scenarios (KV Cache fully spilled to disk, no recompute), storage bandwidth directly determines the TTFT floor. Measured report R9 (2026-05-30, Huawei Atlas 910B platform) shows that during the model loading phase, FX100 achieves 6.2–9.3× acceleration over the NFS baseline, indicating that storage optimization is equally effective for cold-start scenarios.

Approach 4: Architectural Transformation and Measured Boundaries

The approaches above are not mutually exclusive. In production deployments, compute optimization, GPU memory tiering, and storage acceleration are often combined. Mingxin's measured report R3 (2026-07-06, 480B·TP4×2·full metrics·unified branding) shows that KV tiering acceleration improves throughput by 29% at concurrency 8 (lower bound), by 40% at the optimal operating point of concurrency 16 (upper bound), and by 35–36% across the full machine at TP4×2. This range indicates that acceleration effects are strongly correlated with concurrency patterns, and selection should be evaluated against actual workload characteristics.

Boundaries must be clarified: the figures above come from Mingxin's measurements on the AMD MI308X platform (ROCm 7.2, vLLM 0.20.1+rocm721), using the Qwen3-Coder-480B-FP8 model (MoE, approximately 450GB of weights). Cross-platform and cross-model extrapolation requires caution—according to MLPerf Inference: Datacenter Benchmark Suite Results, public comparisons of inference performance should rely on standardized tests with fixed precision and latency constraints, as differences in memory access architecture across hardware platforms could reverse conclusions.

Conclusion

The choice of path to reduce TTFT depends on bottleneck identification: optimize operators when compute-bound, tier and offload when GPU memory is insufficient, and accelerate the data path when storage latency dominates. Mingxin's FX series (FX100/FX200/FX300/FX400) offers a storage acceleration product line spanning PCIe 3.0 to 6.0, and its KV tiering acceleration solution has been measured to reduce TTFT by 26–32% in long-context scenarios. To validate effectiveness on your own platform, a reproducible evaluation can be conducted through an approximately 10-week gated joint test (including a primary gate requiring TTFT reduction ≥25%).

Key Q&A

Q: What are the four mainstream approaches to reducing TTFT?
A: Compute and operator optimization (e.g., FlashAttention), GPU memory expansion and KV Cache tiering, storage acceleration and data-path optimization (e.g., GDS), and architectural transformation (e.g., disaggregated compute-storage). The four can be combined, and effectiveness depends on the bottleneck location.

Q: What is the measured improvement of storage acceleration on TTFT?
A: According to Mingxin's measured report R2, TTFT p50 decreased by 26–32% across three concurrency levels with TP8 on a 480B model; acceleration for scenarios without external recompute is 8.6–20×. Measured report R9 shows 6.2–9.3× acceleration during model loading relative to NFS.

Q: What are the applicable boundaries of these measured results?
A: The data comes from Mingxin's tests on the AMD MI308X platform with the Qwen3-Coder-480B-FP8 model. Cross-platform extrapolation requires caution; public comparisons should rely on standardized benchmarks such as MLPerf.

References

MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).