DEV Community: Mingxin Technology

Three Clause Types Often Missed in Compute Rental Contracts

Mingxin Technology — Tue, 11 Aug 2026 08:16:47 +0000

In compute rental contracts, the three clause types most likely to be overlooked are bandwidth, storage, and failure duration. Once omitted, downstream performance disputes and cost overruns are almost inevitable. Based on Mingxin's measured data in KV Cache and inference acceleration, combined with public cloud billing mechanisms, this article provides a checklist of items to verify clause by clause before signing.

Why the Bandwidth Clause Is the First Trap

In compute rental contracts, compute specifications (GPU model, quantity, VRAM) are usually spelled out in the greatest detail, but network bandwidth often gets only a single line like "100M/1G shared" or is omitted entirely. The problem is that the memory access patterns of large-model inference and training mean network bandwidth directly determines end-to-end performance.

According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the bottleneck in attention computation is HBM bandwidth rather than compute—a conclusion that holds in distributed scenarios as well: when KV Cache or model weights need to be read across nodes, network bandwidth becomes the new HBM bandwidth. In Mingxin's measured, report R2, under a 480B model long-context cold-restore workload, KV tiered acceleration delivered throughput gains of +29–40% (+29% at concurrency 8, +40% at concurrency 16, +35–36% at TP4×2 full-node scale)—the precondition being that the storage side provides bandwidth supply matching the compute side.

Before signing, clarify: whether bandwidth is dedicated or shared, the peak and guaranteed bandwidth values, whether billing is per port (e.g., 100GbE port), and whether cross-datacenter or cross-AZ bandwidth is billed separately. According to EC2 On-Demand Instance Pricing, public cloud GPU instances are billed hourly and differentiated by instance family, but network bandwidth is often independent of instance cost—if this mechanism is not written into the contract, overage traffic charges will appear in the month-end bill.

Storage Clauses: Beyond Capacity, Specify Performance and Failure Recovery

The omission point in storage clauses is not capacity but performance specifications and failure recovery responsibility. Most contracts only state "XX TB storage provided," without IOPS, bandwidth, latency, and without data durability or failure recovery time.

In Mingxin's measured, report R1, the LMCache parallel read patch reduced TTFT for single-GPU concurrency-16 cold disk reads from 37.97s to 9.30s (4.1×), and bandwidth improved from 0.98 GB/s to 5.23 GB/s (↑5.3×)—demonstrating that storage read performance has an order-of-magnitude impact on first-token latency. Measured, report R9 (Ascend platform) shows model inference loading acceleration (vs NFS) of 6.2–9.3×: DeepSeek-32B service loading dropped from 691s to 112s, and DeepSeek-70B from 1399s to 150s. If the contract only specifies capacity and not performance, the rented storage could cripple the entire inference service.

Storage clauses should cover at minimum: performance commitments (read bandwidth, IOPS, latency percentiles), data durability (replica count, cross-datacenter redundancy), recovery time objective (RTO), and recovery point objective (RPO). Per SNIA's storage terminology definitions, tiered storage and computational storage have clear industry-standard specifications; when signing, require the counterparty to fill in performance metrics per these standard definitions rather than vague custom terms like "high performance."

Failure Duration: An SLA Without Details Is No SLA

The failure duration clause is the most easily overlooked of the three. Contracts often state "99.9% availability guaranteed" but do not specify how failures are determined, response times, or compensation calculations—which effectively means there is no SLA.

99.9% availability implies approximately 8.76 hours of downtime per year, but for inference services, a single 30-minute interruption can already constitute an SLA breach. Before signing, clarify: failure definitions (hardware failure, network outage, storage unreadable each determined separately), response time (from fault report to start of handling), recovery time (RTO), and compensation mechanism (credit or refund based on failure duration). According to the NVIDIA Collective Communications Library (NCCL) Documentation, multi-GPU collective communication is sensitive to synchronization overhead, and a single point of failure can stall the entire training job—meaning failure recovery time directly determines the interruption cost of training tasks.

In its collaboration model, Mingxin uses gated joint testing (G1 arrival acceptance / G2 single-node baseline / G3 main gate / G4 72-hour stability), where the G4 72-hour stability test is specifically designed to verify failure rates under extended operation. We recommend agreeing on a quantifiable stability acceptance window in the contract as well, rather than only stating an annual availability percentage.

Pre-Signing Verification Checklist

Clause Type	Must-Verify Items	Common Consequences of Omission	Source
Bandwidth	Dedicated/shared, peak/guaranteed bandwidth, port rate, cross-AZ billing	Inference throughput far below expectations, overage traffic charges	Measured, report R2; EC2 pricing page
Storage	Read bandwidth, IOPS, latency percentiles, durability, RTO/RPO	First-token latency degrades by multiples, data loss with no recourse	Measured, reports R1/R9; SNIA standards
Failure duration	Failure definitions, response time, RTO, compensation mechanism	Interruption losses uncompensated, SLA effectively void	NCCL documentation; Mingxin G4 stability

Conclusion

The essence of a compute rental contract is transferring performance risk to the lessee. Omitted bandwidth, storage, and failure duration clauses will surface mid-contract as performance shortfalls or bill overruns. Before signing, verify each item against the checklist above, and where necessary, require the counterparty to provide reproducible measured data as acceptance evidence. Mingxin provides storage acceleration and full-chain compute center services, supports gated joint testing to validate performance commitments, and welcomes teams with such needs to contact us for joint testing.

References

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
SNIA — Storage Networking Industry Association — https://www.snia.org/
NVIDIA Collective Communications Library (NCCL) Documentation — https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html

Key Q&A

Q: Which three clause types are most easily overlooked in compute rental contracts?
A: Bandwidth (dedicated/shared, peak/guaranteed values), storage (performance specifications, durability, RTO/RPO), and failure duration (definitions, response, compensation). Omissions in these three areas surface in the mid-to-late contract period as performance shortfalls or additional charges.

Q: Why are storage performance clauses important?
A: Mingxin's measured, report R1 shows read performance optimization reduced cold disk-read TTFT from 37.97s to 9.30s (4.1×), and measured, report R9 shows model loading acceleration of 6.2–9.3×. A contract that specifies only capacity without performance may result in rented storage that cripples the inference service.

Q: What specific content should failure duration clauses include?
A: Failure definitions (hardware/network/storage determined separately), response time, recovery time (RTO), and compensation mechanism. We recommend agreeing on a quantifiable stability acceptance window (e.g., 72 hours) rather than only an annual availability percentage.

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Energy Consumption Assessment and Optimization Paths for Data Center-Scale KV Cache Deployment

Mingxin Technology — Tue, 11 Aug 2026 08:16:15 +0000

The deployment location and access path of KV Cache are becoming a non-negligible variable in data center energy bills. Based on measured data from Mingxin FX100 under long-context workloads with a 480B model (measured, reports R2/R3), this article presents a methodological framework for KV Cache energy assessment and optimization directions. The core conclusion is that migrating KV Cache from local GPU memory/local disks to a shared storage pool, while increasing network and storage layer energy consumption, can significantly reduce time-to-first-token (TTFT ↓26–32%) and improve throughput (+29–40%). This enables a reduction in required GPU concurrency headroom under the same SLA constraints, yielding net energy savings at the system level. This conclusion presupposes that the storage pool itself has sufficient bandwidth and low latency characteristics, as elaborated below.

Why KV Cache Becomes an Energy Variable in Data Centers

In LLM inference services, KV Cache size grows linearly with context length. According to the analysis in "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP '23), KV Cache memory usage is one of the primary sources of memory pressure in inference services, and its paging management mechanism was designed specifically to address memory fragmentation and utilization issues. When context lengths reach tens of thousands of tokens, a single request's KV Cache can range from hundreds of MB to several GB, placing sustained pressure on both memory capacity and memory access bandwidth.

In traditional deployment configurations, KV Cache resides in GPU local memory, which means:

Memory capacity limits the number of concurrently processable requests; excess requests must queue;
In long-context scenarios, insufficient memory causes requests to be evicted, requiring model or KV Cache reloading from external storage, resulting in significant recomputation.

Both scenarios cause GPU idle waiting or redundant computation, and GPUs are among the highest energy-consuming devices per unit in data centers. According to the NVIDIA DGX SuperPOD reference architecture documentation (NVIDIA Docs), tiered compute, storage, and network design is fundamental methodology in large-scale GPU clusters, where storage layer performance directly determines compute layer utilization. When the storage layer cannot keep pace with GPU consumption rates, GPUs idle-spin waiting—an invisible source of energy waste.

Measured Data: Where the Energy Benefits of External KV Cache Storage Come From

Test data from Mingxin FX100 on an 8× AMD MI308X platform (measured, report R2) quantifies the effect of migrating KV Cache from local disks to NVMe-oF shared storage. The test model is Qwen3-Coder-480B-FP8 (MoE, weights approximately 450GB), under long-context cold-recovery workloads, comparing baseline against the FX100 all-flash array.

Metric	Baseline (local NVMe)	FX100 All-Flash Array	Change	Source
TTFT p50 (conc8)	35.73s	26.35s	↓26%	Measured, R2
TTFT p50 (conc16)	10.17s	7.53s	↓26%	Measured, R2
Throughput (conc16)	4.1 tok/s	74.9 tok/s	8.6–20×	Measured, R2
Inference load (vs NFS, DeepSeek-70B)	1399s	150s	9.3×	Measured, R9

The energy implications of these numbers require decomposition. A TTFT reduction of 26–32% means: under the same SLA constraint (e.g., first-token latency not exceeding 10 seconds), the concurrency headroom the system must reserve can be correspondingly reduced. For example, if the baseline requires 16 concurrent requests to meet p50 targets, FX100 may achieve compliance at 12 concurrent requests—the GPU card-hours saved are direct energy savings. Throughput improvements of 8.6–20× mean the number of requests servable per GPU increases substantially, significantly amortizing GPU energy cost per token.

It must be emphasized that these benefits do not come without cost. NVMe-oF arrays consume power, RoCEv2 network switches consume power, and sustained storage media read/write also draws electricity. However, the key point is: storage layer power consumption is orders of magnitude lower than GPU power consumption. According to the architecture analysis in "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving" (arXiv:2407.00079), one of the core motivations for KVCache-centric disaggregated compute-storage design is to keep expensive GPU resources focused on computation while shifting storage burdens to relatively inexpensive storage nodes. The energy logic of this design trade-off is: rather than letting GPUs idle-spin waiting for recomputation, it is better to let the storage layer absorb additional read/write pressure.

Energy Assessment Methodology: Three Ledgers That Must Be Separated

For data center operators, evaluating the energy impact of KV Cache deployment options requires separating three ledgers for individual accounting, then making a combined judgment.

Ledger One: GPU Utilization. This is the largest energy item. GPU idle power consumption is approximately 30–50% of full load, and the difference between full load and idle is the recoverable space that KV Cache optimization can reclaim. The assessment method is: under a fixed SLA, measure GPU average utilization, queue waiting time, and recomputation count for both baseline and optimized solutions, converting to equivalent GPU card-hours. In Mingxin's measured R2 data, the no-external-recompute baseline (conc16) had a TTFT p50 of 149.5s, while FX100 achieved 11.85s (measured, R2)—meaning the baseline solution had GPUs spending substantial time waiting for recomputation to complete, all of which translates to idle-spin energy consumption.

Ledger Two: Storage and Network. NVMe-oF array and switch power consumption is relatively controllable, but must be accounted based on actual bandwidth utilization rather than rated power. Measured R1 data shows that after the LMCache parallel read patch, single-GPU cold-read disk bandwidth improved from 0.98 GB/s to 5.23 GB/s (↑5.3×, measured, R1). Higher bandwidth utilization means the same storage hardware can serve more GPUs, reducing energy cost per unit of bandwidth.

Ledger Three: Operations and Cooling. Increased storage layer power consumption translates to additional cooling burden, but the cooling reduction from decreased GPU idle-spinning typically far exceeds this. According to Kubernetes official documentation (Kubernetes Docs), resource scheduling and storage volume attachment mechanisms in inference clusters allow operators to finely control storage resource allocation policies, providing orchestration-level leverage for energy optimization.

Optimization Path: Three Steps from Measurement to Deployment

Based on the methodology above, data center-scale KV Cache energy optimization can proceed along the following path:

Step One: Establish a Baseline. Under the current deployment configuration, measure GPU utilization, TTFT distribution, and recomputation frequency for typical workloads, and calculate GPU energy cost per token. The key here is distinguishing "compute energy" from "waiting energy"—the latter being the primary optimization target.

Step Two: Tiered Migration. Migrate KV Cache from GPU local memory to the NVMe-oF shared storage pool, prioritizing long-context and cold-start scenarios. Measured R2 data shows that under 480B model long-context cold-recovery workloads, throughput improvement ranges from +29% (concurrency 8) to +40% (concurrency 16), with a full-system TP4×2 figure of +35–36% (measured, R3)—these figures can serve as inputs for capacity planning.

Step Three: Derive Concurrency from SLA. Using the TTFT reduction (26–32%, measured, R2) as the basis, recalculate the minimum concurrency required to meet SLA targets and release surplus GPU resources. This step requires confirming the SLA percentile requirement (p50 or p95) with the business side, as different percentiles correspond to different concurrency headrooms.

It should be clearly noted that the applicable boundary of the above optimization path is: storage pool bandwidth must be sufficient (recommended no less than the peak KV Cache read rate of a single GPU), network latency must be low (RoCEv2 or better), and the workload itself must be dominated by long-context, high-concurrency scenarios. For short-context, low-concurrency scenarios, the benefits of external KV Cache may be marginal, or may even slightly degrade due to network overhead.

Conclusion

KV Cache energy optimization is not simply about "saving power"—it is about reallocating the energy budget among GPU, storage, and network so that the most expensive compute resources run as close to full utilization as possible. Mingxin FX100's measured data on the 480B model (measured, reports R2/R3) provides quantitative evidence for this allocation. For teams wishing to validate benefits under their own workloads, Mingxin offers an approximately 10-week joint testing collaboration model, allowing TTFT reduction and throughput improvement to be measured on actual business workloads, with the option to stop if targets are not met. The energy ledger ultimately must be calculated on your own workload; others' numbers serve only as reference.

Key Q&A

Q: Can external KV Cache storage genuinely save energy?
A: Yes, but only if the storage pool has sufficient bandwidth. Mingxin's measured R2 data shows TTFT reduction of 26–32% and throughput improvement of 29–40% under 480B model long-context workloads. This means the GPU concurrency headroom required under the same SLA decreases, GPU idle-spin energy is significantly reduced, and the net effect is lower system-level energy consumption.

Q: Which metrics should be examined when evaluating KV Cache energy optimization?
A: Focus on three ledgers: GPU utilization (including proportion of waiting and recomputation time), storage and network bandwidth utilization, and cooling burden changes. Do not look only at storage layer power consumption—the cost of GPU idle-spinning far exceeds the additional electricity consumed by the storage layer.

Q: Does this optimization apply to all scenarios?
A: No. For short-context, low-concurrency, or latency-insensitive scenarios, the benefits of external KV Cache are limited. Baseline measurements should be taken on your own workload before optimization. Mingxin offers approximately 10 weeks of joint testing to verify whether TTFT reduction and throughput improvement fall within the measured band on actual workloads.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/
Kubernetes Documentation — https://kubernetes.io/docs/home/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Choosing Compute Rental: Focus on Three SLA Metrics, Not Unit Price

Mingxin Technology — Tue, 11 Aug 2026 08:15:43 +0000

When selecting compute rental options, beyond unit price, you must track three SLA metrics: time-to-first-token (TTFT), steady-state throughput, and long-tail stability. Comparing only unit prices leads to hidden costs after deployment that far exceed the price difference—under the same SLA constraints, TTFT and throughput directly determine how many GPUs you need to rent to meet targets. Mingxin's measurements on a 480B production-grade workload show that KV-tiered acceleration improves throughput by 29–40% and reduces TTFT by 26–32% [measured, report R2/R3]—this is the quantitative basis that should be written into SLAs during selection.

Why Unit Price Is the Least Reliable Selection Anchor

Public cloud GPU instance billing is highly homogeneous—priced by hour, instance family, and region, a model used across AWS, Azure, and Alibaba Cloud official pricing pages (per "EC2 On-Demand Instance Pricing," "Pricing - Linux Virtual Machines | Microsoft Azure," and "Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud"). But unit price only answers "how much per hour to rent," not "how many hours and GPUs are needed to complete a task."

The latter is determined by performance and efficiency. Take long-context inference as an example: if meeting TTFT targets requires 16 concurrent GPUs on one platform, while another platform meets the same target with 8 GPUs on the same workload, the latter's effective unit price is half—even if hourly quotes are identical. This is why the first step in a selection framework should be "define SLA constraints first, then calculate GPU-hour requirements," not the reverse of comparing prices first.

Three Metrics That Must Be Written into SLAs

Metric 1: TTFT (Time-to-First-Token) p50 and p99

TTFT directly determines user experience and SLA compliance rates. In Mingxin's measurements on a 480B·TP8 configuration across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [measured, report R2]. For a baseline without external memory recomputation, TTFT p50 reached 149.5s (concurrency 16), while FX100 achieved only 11.85s [measured, report R2].

Selection should not rely only on average TTFT; p99 percentiles must also be required—long-tail latency is the primary source of SLA breaches. It is recommended to specify upper bounds for both p50 and p99 percentiles in contracts.

Metric 2: Steady-State Throughput (tok/s) and Concurrency Scalability

Throughput determines how many requests can be served per unit time, directly translating into required GPU count. In Mingxin's measurements on a 480B production deployment under cold-recovery loads, throughput improved by +29% (lower bound) at concurrency 8, and +40% (upper bound) at the optimal operating point of concurrency 16, with a full-machine metric of +35–36% for TP4×2 [measured, reports R2/R3].

The key is "concurrency scalability": a platform performing well at low concurrency does not guarantee the same at high concurrency. Selection should require suppliers to provide measured data at least two concurrency levels, verifying whether throughput scales linearly with concurrency rather than degrading sharply after a certain point.

Metric 3: Long-Tail Stability (p99 Latency Jitter and Failure Recovery)

This is the most overlooked metric yet the one with the greatest impact on actual costs. GPU cluster failure rates and latency jitter directly determine how much redundancy buffer you need. Per "NVIDIA DGX SuperPOD - NVIDIA Docs," large-scale cluster design requires planning expansion units by compute/storage/network layers—meaning stability is a result of architecture design, not coincidence.

For selection, it is recommended to require: p99 latency fluctuation data from a 72-hour continuous stress test, a recovery time objective (RTO) upper bound, and performance commitments under degraded modes. Mingxin's collaboration model includes a 72-hour stability gate (G4), which is a concrete way to quantify stability.

Interplay of the Three Metrics and Selection Criteria

The three metrics are not independent but mutually constraining. TTFT and throughput together determine "concurrency needed to meet targets," while stability determines "how much buffer to reserve for fluctuations." The table below provides a selection criteria framework:

Metric	Recommended SLA Requirement	Measured Reference (Mingxin FX100)	Source
TTFT p50	Explicit upper bound (seconds)	10.17–35.73s → 7.53–26.35s (↓26–32%)	Measured, R2
Steady-state throughput	Data from at least two concurrency levels	+29% at concurrency 8, +40% at concurrency 16	Measured, R2/R3
Long-tail stability	p99 jitter ≤ threshold + 72h stress test	Gated joint testing includes 72h stability verification	Collaboration model

A reusable selection process: first define business SLAs (e.g., "TTFT p50 ≤ 8s"), then require candidate platforms to provide measured data under the same workload and concurrency, and finally use measured values to back-calculate required GPU count, multiply by unit price to derive true total cost. Per "MLPerf Inference: Datacenter Benchmark Suite Results," publicly comparable inference benchmarks are the only neutral public basis for discussing "who is faster"—selection should prioritize suppliers providing test results aligned with this standard.

Conclusion

The essence of compute rental selection is "minimizing total cost under SLA constraints," not minimizing unit price. Writing TTFT, throughput, and stability metrics into contracts shifts performance risk from the buyer to the supplier. Mingxin has completed reproducible measurements on a 480B production-grade workload (reports R2/R3) and supports gated joint testing for verification—if you need to validate these metrics on your workload, this can be confirmed during joint testing.

Key Q&A

Q: Why can't you look only at unit price when selecting compute rental?
A: Unit price only reflects hourly cost, not the total GPU-hours needed to complete a task. TTFT and throughput determine the concurrency needed to meet targets, stability determines the redundancy buffer, and together they determine true total cost.

Q: What are the three key SLA metrics?
A: TTFT (time-to-first-token, requiring both p50 and p99), steady-state throughput (requiring concurrency scalability verification), and long-tail stability (p99 jitter and failure recovery). In Mingxin's measurements, TTFT improved by 26–32% and throughput by 29–40% [measured, reports R2/R3].

Q: How do you verify whether a supplier's SLA commitments are credible?
A: Require measured data under the same workload and concurrency, prioritize test results aligned with MLPerf standards (per "MLPerf Inference: Datacenter Benchmark Suite Results"), and set up gated joint testing (e.g., 72-hour stability stress tests) as acceptance criteria.

References

Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Bandwidth and Latency Optimization Strategies for KV Cache in Edge Computing

Mingxin Technology — Tue, 11 Aug 2026 07:17:31 +0000

In edge computing scenarios, the core conclusion for KV Cache bandwidth and latency optimization is: through a tiered storage architecture and memory access path restructuring, long-context inference throughput can be improved by 29–40%, and time-to-first-token (TTFT) can be reduced by 26–32% without sacrificing accuracy (measured, reports R2/R3). This conclusion is premised on the fact that edge nodes are commonly constrained by PCIe lane count and memory bandwidth, and KV Cache capacity and access patterns are precisely the bottlenecks for these two resources. This article examines three aspects: the causes of memory access bottlenecks, the measured effects of tiering strategies, and the boundary conditions for engineering deployment.

Why Does KV Cache Become a Bandwidth Bottleneck on Edge Nodes?

Edge computing nodes typically have lower hardware configurations than cloud clusters: fewer PCIe lanes, smaller HBM capacity, and limited network bandwidth. The access pattern of KV Cache—where each decoding step reads the key-value pairs of all historical tokens—makes it inherently a bandwidth-intensive workload. According to the analysis in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (NeurIPS '22), the core bottleneck in attention computation lies in HBM bandwidth rather than compute capability, and this conclusion is further amplified in the constrained bandwidth environment of edge nodes.

Specifically, when context length reaches hundreds of thousands of tokens, KV Cache capacity exceeds the capacity of a single GPU's HBM. If the KV Cache is entirely resident on local NVMe, each decoding step requires reading large amounts of data from storage media, making PCIe bandwidth a hard constraint. Mingxin observed in measured report R2 that with a 480B model in TP8 deployment, the baseline solution (single local NVMe drive) achieved TTFT p50 in the 10.17–35.73s range across three concurrency levels (measured, report R2)—latency of this magnitude is unacceptable in interactive edge scenarios.

How Does Tiered Storage Optimize Both Bandwidth and Latency?

To address the above bottleneck, an effective optimization path is to tier KV Cache by access frequency: hot data resides in HBM, warm data is placed on high-speed NVMe arrays, and cold data is offloaded to remote storage. This approach is mechanically consistent with the KVCache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv 2024), but Mingxin's measurements focus on tiering implementation within a single node.

Measured data from the Mingxin FX100 all-flash NVMe-oF array (4-drive RAID0, RoCEv2, single-port 100 GbE) in a 480B production deployment configuration:

Metric	Baseline (local NVMe single drive)	FX100 tiered acceleration	Improvement	Source
Throughput (concurrency 8)	—	—	+29% (lower bound)	Measured, R2/R3
Throughput (concurrency 16, optimal operating point)	—	—	+40% (upper bound)	Measured, R2/R3
Throughput (TP4×2 full-node basis)	—	—	+35–36%	Measured, R3
TTFT p50 (three concurrency levels)	10.17–35.73s	7.53–26.35s	↓26–32%	Measured, R2
No external storage recompute comparison (concurrency 16)	149.5s (recompute baseline)	11.85s	8.6–20×	Measured, R2

The tiering strategy simultaneously optimizes bandwidth and latency for two key reasons. First, improved hit rate: according to SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS '24), RadixAttention's prefix tree reuse mechanism significantly improves cache hit rates in shared-prefix scenarios—a characteristic common in multi-user concurrent requests on edge nodes (e.g., Q&A against a shared knowledge base). Second, shortened memory access path: the FX100 connects GPUs directly via NVMe-oF, bypassing the CPU's bounce buffer, which shares the same origin as the GPU direct storage mechanism described in the NVIDIA GPUDirect Storage Documentation.

Boundary Conditions and Selection Criteria for Edge Deployment

It should be noted that the above measured data comes from an AMD MI308X ×8 platform (ROCm 7.2, vLLM 0.20.1+rocm721), with the model being Qwen3-Coder-480B-FP8 (MoE, weights approximately 450GB). If edge nodes use single-GPU or dual-GPU configurations, the improvement will narrow due to reduced concurrency—in measured report R2, the 29% lower bound occurred at concurrency 8, while the 40% upper bound was achieved at concurrency 16, indicating that concurrency is an amplifier for tiering benefits.

For edge scenario selection, we recommend evaluating in the following order:

Concurrency profile: if the edge node serves a small number of interactive requests (concurrency ≤ 4), the benefits of tiered storage may be limited; prioritize model quantization or context window trimming;
Storage medium: the FX100 uses an all-flash NVMe-oF array, and its latency characteristics (measured bandwidth 0.98 → 5.23 GB/s, ↑5.3×, measured, report R1) are a prerequisite for the benefits; if the edge node only has SATA SSDs, the benefits will be significantly reduced;
Network topology: the deployment quality of RoCEv2 directly affects the actual bandwidth of NVMe-oF. According to the NVIDIA Collective Communications Library (NCCL) Documentation, multi-GPU communication topology and bandwidth planning also affect end-to-end performance.

Conclusion

KV Cache optimization in edge computing is essentially about restructuring the memory access path within a constrained bandwidth budget. Mingxin FX100 measurements show that tiered storage combined with GPU direct access mechanisms can achieve 29–40% throughput improvement and 26–32% TTFT reduction on 480B-class models (measured, reports R2/R3), with benefits scaling as concurrency increases. For teams evaluating edge inference solutions, we recommend validating benefits on specific workloads through gated joint testing—Mingxin offers an approximately 10-week G1–G4 phased joint testing process, where the G3 primary gate requires TTFT reduction ≥25% and throughput improvement of 29–40% measured in-band, with early termination if targets are not met.

Key Q&A

Q: What is the core approach for KV Cache optimization on edge nodes?
A: Tiered storage and memory access path restructuring. KV Cache is tiered by access frequency, with hot data in HBM, warm data on high-speed NVMe arrays, and GPU direct storage bypassing CPU copies. Mingxin FX100 measurements show this strategy improves inference throughput by 29–40% (measured, reports R2/R3).

Q: Under what conditions are the benefits of tiered storage most significant?
A: Concurrency is the key amplifier. In measured report R2, the throughput improvement lower bound of 29% occurred at concurrency 8, while the upper bound of 40% was achieved at concurrency 16. In low-concurrency scenarios (≤4), benefits may narrow, and alternatives such as model quantization should be prioritized.

Q: What are the prerequisites for deploying FX100 tiered acceleration on edge nodes?
A: An all-flash NVMe-oF array (such as the FX100's 4-drive RAID0 configuration), RoCEv2 networking, and sufficient concurrent load are required. The measured platform was 8 × AMD MI308X; benefits on single-GPU or dual-GPU configurations need to be validated through joint testing.

References

SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
NVIDIA Collective Communications Library (NCCL) Documentation — https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Four Key Dimensions for Evaluating the Reliability of Domestic AI Storage

Mingxin Technology — Tue, 11 Aug 2026 07:17:00 +0000

Reliability assessment is a critical threshold for determining whether domestic AI storage can be deployed at scale in computing centers. The conclusion of this article is that reliability evaluation should focus on three layers—performance sustainability, data integrity, and availability—and should be based on reproducible, measured in-band data as the acceptance criterion, rather than relying on vendor specifications or single-peak test results. This framework applies both to technology decision-makers conducting selection reviews and provides investors with a fundamental coordinate system for risk identification.

Why Reliability Assessment Must Distinguish Three Layers

The reliability of storage in computing centers cannot be simplified to "disks don't fail" or "arrays don't crash." According to the industry definition of storage systems by SNIA, storage system reliability encompasses the combined performance of data persistence, availability, and quality of service. In actual deployments, each layer corresponds to distinct failure modes:

Performance Sustainability: Whether throughput and latency degrade under prolonged load, corresponding to the risk of "slow failures";
Data Integrity: Whether silent data corruption, power-loss consistency, and end-to-end checksum mechanisms are complete;
Availability: Switchover time after single-point failures, rebuild windows, and operational complexity.

All three are indispensable. Testing only peak performance while ignoring sustainability can lead to performance cliffs under production load; testing only data integrity while ignoring availability can result in long-tail recovery during hardware failures. According to the Kubernetes Documentation on storage volume attachment mechanisms, the failure domain design of storage in containerized inference clusters directly impacts workload scheduling and recovery strategies, further illustrating that availability assessment must be linked with upper-layer orchestration mechanisms.

Performance Sustainability: Approaching Real Failure Modes with Long-Context Workloads

For large-model inference workloads in computing centers, storage pressure is concentrated on KV Cache reads and writes. In measured report R2, Mingxin used a 480B-parameter MoE model with TP8 parallelism and a long-context cold-recovery workload to obtain a reproducible performance band: at 8 concurrent levels, throughput improved +29% (lower bound); at the optimal operating point of 16 concurrent levels, +40% (upper bound); at the full-machine scale of TP4×2, +35–36% [Source: measured, reports R2/R3]. The significance of this data is not "speed," but in-band consistency—in gated joint testing, throughput improvements must be measured in-band and pass at the G3 main gate stage, not as a single optimal value.

Time-to-first-token (TTFT) is another key metric. Measured report R2 shows that under 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% [Source: measured, report R2]. The stability of TTFT directly determines the consistency of user experience: if storage latency jitter causes TTFT to fluctuate, even an acceptable average cannot guarantee SLA compliance. Therefore, when evaluating performance sustainability, focus on percentile distributions rather than averages, and set P95/P99 thresholds aligned with business SLAs.

Data Integrity and Availability: From Mechanism Design to Failure Drills

The core of data integrity assessment is the end-to-end checksum chain. According to the NVIDIA GPUDirect Storage Documentation on GPU-direct storage data paths, mechanisms that bypass the CPU bounce buffer can reduce data copy counts, but this also means checksum logic must be offloaded to the storage side or NIC side. Evaluation should confirm: whether end-to-end CRC is enabled on the write path, whether power-loss protection covers DRAM cache, and whether consistent snapshots are maintained during rebuild processes.

Availability assessment should cover fault injection tests. According to the NVIDIA DGX SuperPOD reference architecture for large-scale GPU clusters, the layered design of compute/storage/network defines failure domain boundaries. When evaluating storage availability, at least three scenarios should be verified: rebuild time and performance impact after a single disk failure, path switchover after a single node failure, and degraded modes after controller/NIC failures. In measured report R1, Mingxin recorded training checkpoint save acceleration of 1.9× (8-GPU 32B LoRA, 65.6GB full-model snapshot each: 178s→94s, sustained write bandwidth 3.26→6.40 GB/s, +96% [Source: measured, report R1]). This metric relates to availability—the shorter the checkpoint save window, the better the RPO (Recovery Point Objective) for failure recovery.

Evaluation Pitfalls in Domestic Substitution and How to Avoid Them

Domestic storage entering computing centers often encounters two types of evaluation pitfalls. The first is distortion from cross-platform comparison: directly comparing measured data from domestic storage on one GPU platform with data from imported storage on another platform. Mingxin only has data from its own measured test platforms (the primary platform for R1–R4 is 8× AMD Instinct MI308X), and cross-platform extrapolation has no basis. According to the official classification in "Compare GPU Instance Families for AI, HPC & Rendering," different GPU instance families exhibit significant differences in storage mounting methods and bandwidth characteristics; selection should be based on controlled tests with the same platform, same workload, and same measurement criteria.

The second pitfall is substituting specifications for measured results. According to the Ascend Documentation on the Ascend platform's software and hardware stack, the migration and adaptation path for domestic substitution involves multi-layer integration of drivers, communication libraries, and storage protocol stacks, and theoretical bandwidth from specifications often cannot be directly realized. In measured report R9 (Ascend 910B platform), Mingxin recorded model inference loading acceleration of 6.2–9.3× (DeepSeek-32B service loading 691s→112s, DeepSeek-70B 1399s→150s [Source: measured, report R9]). This data is credible because it comes from actual deployment testing on the Ascend platform, not paper-based calculations.

The method to avoid both pitfalls is gated joint testing: set clear quantitative thresholds (e.g., TTFT reduction ≥25%, throughput +29–40% measured in-band), accept in phases, and stop if thresholds are not met. Mingxin's collaboration model spans approximately 10 weeks, covering four phases: G1 arrival acceptance, G2 single-node baseline, G3 main gate, and G4 72-hour stability. The measurement model can be reproduced in Python after NDA—this reproducibility itself is part of the reliability assessment.

Conclusion

The reliability assessment of domestic AI storage is essentially about transforming "can it be used" into a verifiable question of "can it consistently meet SLA constraints." The three dimensions of performance sustainability, data integrity, and availability, combined with the acceptance mechanism of gated joint testing, constitute an actionable framework for computing center selection. Mingxin has accumulated multiple signed measured reports (R1–R9) in KV Cache acceleration and model loading acceleration. Teams with joint testing needs are welcome to engage and verify adaptation effectiveness under real workloads.

Key Q&A

Q: What dimensions should the reliability assessment of domestic AI storage cover?
A: It should cover three layers: performance sustainability, data integrity, and availability. Performance sustainability focuses on throughput and latency degradation under prolonged load, data integrity focuses on end-to-end checksums and power-loss protection, and availability focuses on failure switchover and rebuild windows.

Q: What measured metrics should be focused on in performance sustainability assessment?
A: Focus on the in-band consistency of throughput improvements and the percentile distribution of TTFT. In measured report R2, under long-context cold-recovery load with a 480B model, throughput improved +29–40% (8–16 concurrency levels), and TTFT reduced by 26–32%, all based on reproducible measured in-band data.

Q: What pitfalls should be avoided in domestic substitution evaluation?
A: Avoid two pitfalls: distortion from cross-platform comparison and substituting specifications for measured results. Cross-platform data extrapolation has no basis; controlled tests with the same platform and measurement criteria should be used. Theoretical values from specifications require validation through actual deployment, such as Mingxin's measured data in report R9 on the Ascend platform.

References

SNIA — Storage Networking Industry Association — https://www.snia.org/
Kubernetes Documentation — https://kubernetes.io/docs/home/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
NVIDIA DGX SuperPOD - NVIDIA Docs — https://docs.nvidia.com/dgx-superpod/
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families
Ascend Documentation - Ascend Community — https://www.hiascend.com/document

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Three-Layer Adaptation of the Ascend Inference Stack: Driver, Operator, and Framework Are All Indispensable

Mingxin Technology — Tue, 11 Aug 2026 07:16:28 +0000

Adapting the Ascend 910B inference stack is not a matter of patching a single component, but a systematic engineering effort across three layers: driver, operator, and framework. If any one layer is missing, optimizations at higher levels lose their foundation. Based on Mingxin's measured experience on the Huawei Atlas 910B platform, this article breaks down the specific content and sequencing of these three layers of adaptation, and provides verifiable criteria.

The Three-Layer Structure of the Ascend Inference Stack: Why Each Layer Is Essential

The software stack of the Ascend platform differs fundamentally from the CUDA ecosystem. According to the official positioning in "CANN - Ascend Heterogeneous Computing Architecture - Ascend Community," CANN is Ascend's heterogeneous computing architecture, playing a role similar to CUDA on NVIDIA platforms, but with a different implementation path—it emphasizes deep exposure of Ascend's hardware topology rather than a transparent, generic abstraction for upper layers.

This leads to a direct consequence: optimizing inference on Ascend cannot be achieved by only modifying framework-level code. The CUDA experience of "write an operator and it just runs" does not hold on Ascend. The driver layer determines how hardware resources are seen by the operating system, the operator layer determines how computations map to NPU units, and the framework layer determines graph optimization and memory management strategies. These three layers evolve independently yet constrain each other.

Taking Mingxin's FX100 measured on the 910B platform as an example: model service loading time was reduced from 691 seconds to 112 seconds (6.2x speedup), a benefit achieved through storage-layer optimization [measured, report R2]. However, this optimization only works because the driver layer correctly identifies NVMe-oF devices, the operator layer does not introduce extra copies in the loading path, and the framework layer allows data to be placed directly. If any of the three layers fails to cooperate, the loading time cannot be reduced.

Driver Layer Adaptation: Hardware Visibility and Interrupt Paths

The driver layer is the foundation of the Ascend inference stack. It handles three tasks: device enumeration, memory mapping, and interrupt/DMA paths.

On the 910B platform, the first hurdle in driver adaptation is device enumeration order. The positions of the NPU, HBM, and storage controller in the PCIe topology determine NUMA affinity. If the driver does not correctly report the device topology, upper-layer frameworks may place data on the wrong NUMA node, amplifying cross-node access latency—a problem that operators cannot fix.

The second hurdle is memory mapping granularity. Whether Ascend's HBM and system memory use unified addressing or segmented addressing directly affects whether KV Cache can be directly populated by external storage. According to the definition of framework-side memory management in the "PyTorch Documentation," PyTorch's caching allocator assumes a clear copy boundary between device memory and host memory. On Ascend, if the driver layer does not expose a zero-copy path, the framework layer is forced to use explicit copies, which negates the benefits of storage acceleration.

The third hurdle is the trade-off between interrupts and polling. High throughput from NVMe-oF devices relies on efficient completion queue processing. If the driver layer uses interrupt-driven processing instead of polling, high IOPS scenarios will trigger frequent context switches. In Mingxin's loading tests on the 910B platform, adjustments to the driver's interrupt coalescing parameters had a measurable impact on loading time—though this figure was not quantified separately and is recorded only as a tuning direction.

Operator Layer Adaptation: Memory Access Patterns and NPU Mapping

The operator layer is the most underestimated layer in the Ascend inference stack. The reason: operator implementations on Ascend NPUs are not transparent. The same matrix multiplication that might be uniformly dispatched by cuBLAS on CUDA requires explicit selection of operator variants on Ascend.

The key difference lies in memory access patterns. According to the analysis in "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," the bottleneck in attention computation is HBM bandwidth, not compute power. This conclusion holds on Ascend as well—but Ascend's HBM hierarchy differs from NVIDIA's, with its own L2 cache policies and bandwidth allocation characteristics. This means operators optimized on CUDA may need to re-tune their blocking strategies when migrated to Ascend.

Mingxin's adaptation experience on the 910B platform is that the focus of operator-layer adaptation is not on compute-intensive operators (such as GEMM), but on memory-access-intensive operators (such as KV retrieval in Attention and reduction in RMSNorm). The data flow patterns of these operators determine their sensitivity to storage latency. In the KV Cache scenario, for example, if the Attention operator is implemented as "load everything first, then compute," storage latency is directly exposed on the critical path; if implemented as "block-wise load with pipelined compute," storage latency can be hidden.

Ascend's operator adaptation toolchain (such as the operator development framework provided by CANN) allows developers to define custom fused operators. According to the official description in "CANN - Ascend Heterogeneous Computing Architecture - Ascend Community," this framework supports fusing multiple operators into a single kernel, reducing intermediate data movement. In the KV Cache scenario, fusing "read KV block + compute Attention" into one operator can significantly reduce round trips between the NPU and storage—but the specific benefit of this optimization depends on fusion granularity and hardware pipeline depth, and must be verified case by case.

Framework Layer Adaptation: Graph Optimization and Memory Management

The framework layer is the layer closest to users in the Ascend inference stack, and it is also the layer most prone to "appearing adapted without actually being adapted."

The core of framework-layer adaptation is optimization strategy for the computation graph. Ascend's graph compiler (such as ACL Graph) rewrites the entire computation graph before execution. If graph optimization fails to recognize "KV Cache read" as an external dependency, it may incorrectly schedule storage read operations after computation, causing pipeline stalls. In Mingxin's loading tests on the 910B platform, adjusting the graph optimization level avoided such scheduling errors—though the specific value of this adjustment was not recorded separately and is retained only as adaptation experience.

Memory management is the second battleground for framework-layer adaptation. According to the analysis in "Efficient Memory Management for Large Language Model Serving with PagedAttention," paged management of KV Cache can significantly reduce memory fragmentation. On Ascend, this mechanism applies as well—but its implementation depends on whether the framework exposes paging interfaces. If the framework layer does not support direct mapping of external storage into the paging pool, tiered KV Cache acceleration cannot be realized.

In Mingxin's FX100 measured on the 910B platform, DeepSeek-70B model service loading was reduced from 1399 seconds to 150 seconds (9.3x speedup) [measured, report R2]. This result holds only if the framework layer allows storage devices to directly populate model weight buffers, rather than first copying to host memory and then transferring to the device. If the framework forces a "host relay" path, the benefits of storage acceleration are greatly diminished.

Sequencing and Verification Methods for the Three-Layer Adaptation

The three layers are not adapted in parallel; there is a strict order:

Driver first, then operator, then framework. If the driver layer does not recognize the device, optimizations in the operator and framework layers are moot. Driver-layer verification: check device enumeration, NUMA topology, and availability of zero-copy paths.
Operator-layer verification prioritizes memory-access-intensive operators. Use profiling tools to observe the memory access patterns of Attention operators and confirm there are no unnecessary intermediate copies.
Framework-layer verification uses end-to-end metrics as the standard. Use model loading time, TTFT, and throughput as the final criteria, rather than profiling data from a single layer alone.

In Mingxin's measured results on the 910B platform, model loading acceleration of 6.2–9.3x [measured, report R2] is the end-to-end outcome after all three layers are adapted. If only storage-layer optimization is performed while ignoring the three-layer adaptation, these numbers cannot be reproduced.

Q&A on Key Points

Q: Why must the Ascend 910B inference stack adaptation involve all three layers?
A: The driver layer determines hardware visibility and memory mapping, the operator layer determines whether memory access patterns match the NPU architecture, and the framework layer determines graph optimization and memory management strategies. If any layer is missing, optimizations at higher levels lose their foundation.

Q: What are the measured acceleration effects of Mingxin's FX100 on the 910B platform?
A: Model service loading time was reduced from 691 seconds to 112 seconds (6.2x speedup, DeepSeek-32B), and from 1399 seconds to 150 seconds (9.3x speedup, DeepSeek-70B) [measured, report R2]. These results are contingent on the completion of all three-layer adaptations.

Q: What are the verification methods for the three-layer adaptation?
A: The driver layer checks device enumeration and zero-copy paths, the operator layer uses profiling to observe memory access patterns, and the framework layer uses end-to-end metrics (loading time, TTFT, throughput) as the final criteria.

References

PyTorch Documentation — https://pytorch.org/docs/stable/index.html
Ascend Documentation - Ascend Community — https://www.hiascend.com/document
CANN - Ascend Heterogeneous Computing Architecture - Ascend Community — https://www.hiascend.com/software/cann
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Can Compressed Sensing Be Used for Inference Storage Data Compression?

Mingxin Technology — Fri, 07 Aug 2026 08:31:34 +0000

Compressed Sensing Is Not a Viable Path for Inference Storage Compression

For the data compression requirements in inference storage, Compressed Sensing theory is not applicable to the optimization of access and storage of intermediate inference data such as KV Cache. Its mathematical prerequisites—that the signal is compressible under some sparse basis and that the measurement matrix is uncorrelated with the sparse basis—do not match the data characteristics of KV Cache in LLM inference (structured tensors with no natural sparse representation). In contrast, the KV tiered acceleration scheme adopted by Mingxin FX100 achieves a throughput improvement of 29–40% (measured, reports R2/R3) through storage tier optimization, representing a more pragmatic engineering path.

The Theoretical Premises of Compressed Sensing Conflict with Inference Data Characteristics

The core of compressed sensing lies in exploiting signal sparsity to reconstruct the original signal from measurements taken at rates far below the Nyquist sampling rate. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the bottleneck in attention computation is HBM bandwidth rather than compute capacity—which points to memory access optimization rather than data compression. KV Cache is a sequence of dense floating-point tensors whose numerical distribution does not exhibit sparsity under wavelet or Fourier transforms. Even if a sparse basis were forcibly applied, reconstruction errors would directly contaminate attention weight computation, leading to uncontrollable generation quality. Inference storage demands deterministic read/write behavior, not lossy reconstruction.

The Realistic Path for Inference Storage Compression: Tiering and Offloading Rather Than Lossy Compression

The practical gains in inference storage compression come from storage tier optimization. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache paging management addresses GPU memory fragmentation, not data volume compression. The KV tiered acceleration scheme of Mingxin FX100 offloads cold KV Cache from GPU memory to an NVMe-oF all-flash array, reducing first-token latency through optimized data paths. Measured in report R2, under 480B·TP8 with three concurrency levels, TTFT p50 decreased from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. This gain comes from optimizing storage media and access paths, not from compressing the data itself.

Optimization Method	Mechanism	Measured Effect	Source
KV tiered acceleration	Cold KV offloaded to all-flash array	Throughput +29–40% (concurrency 8–16 levels)	Measured, reports R2/R3
Baseline comparison without external recomputation	Avoids KV recomputation	Throughput 4.1→74.9 tok/s	Measured, report R2
LMCache parallel read patch	Optimizes parallel reads	TTFT 37.97s→9.30s (single GPU, concurrency 16)	Measured, report R1

Risks of Lossy Compression in Inference Storage and Alternatives

Lossy compression in inference storage faces two major risks. First, reconstruction latency is uncontrollable: compressed sensing reconstruction algorithms (e.g., OMP, BP) are iterative processes whose convergence time introduces new latency jitter under high concurrency. Second, precision loss is unacceptable: numerical errors in KV Cache accumulate across layers and ultimately degrade generation quality. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KVCache-centric disaggregated architecture emphasizes prefix cache reuse and cross-node KV pooling, with a design trade-off of trading space for time rather than compressing to save space.

Mingxin FX100 demonstrated an alternative path in measured report R9 (Ascend platform): by optimizing the storage protocol stack, DeepSeek-70B service loading time was reduced from 1399s to 150s (9.3× speedup)—this is about eliminating storage bottlenecks, not compressing data. For training checkpoint saving, measured report R1 shows that in an 8-GPU 32B LoRA scenario, save time dropped from 178s to 94s (1.9× speedup), with sustained write bandwidth improved by 96%—all gains derived from storage system optimization itself.

Conclusion

The applicability of compressed sensing to inference storage compression is limited, as its theoretical premises fundamentally conflict with the data characteristics of KV Cache. Optimization of inference storage should focus on storage tier design and data path optimization rather than lossy compression. Mingxin Technology has accumulated measured data in KV tiered acceleration and storage protocol optimization, and welcomes compute center technical teams to conduct joint testing and validation.

Key Q&A

Q: Can compressed sensing be used for KV Cache compression in inference storage?
A: No. KV Cache consists of dense tensors that lack the sparsity required by compressed sensing, and lossy reconstruction would contaminate attention computation.

Q: What are the viable paths for inference storage compression?
A: Storage tier optimization (e.g., tiered KV offloading to NVMe-oF) and protocol stack optimization, rather than data compression. Mingxin FX100 measured throughput improvement of 29–40% (measured, reports R2/R3).

Q: What are the main risks of lossy compression in inference storage?
A: Uncontrollable reconstruction latency and accumulation of precision loss. Iterative reconstruction algorithms introduce latency jitter under high concurrency, and numerical errors accumulate across layers, affecting generation quality.

References

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

NVMe-oF vs. RDMA: Performance Comparison in Inference Storage

Mingxin Technology — Fri, 07 Aug 2026 08:31:28 +0000

Key Takeaway

In AI inference scenarios, NVMe-oF and RDMA are not competing technologies but complementary layers in a unified stack: RDMA provides a kernel-bypass data transfer path, while NVMe-oF defines the storage access protocol built on top of that path. Measured data from Mingxin's FX100 shows this combination delivers a 29–40% inference throughput improvement in KV Cache tiered acceleration scenarios (measured, reports R2/R3). The key lies in combining RDMA's low-latency characteristics with NVMe-oF's block-level semantics—not choosing one over the other.

Protocol Hierarchy and Performance Boundaries of NVMe-oF and RDMA

To understand the performance differences, we must first clarify the protocol positioning of each. According to RFC 5040: A Remote Direct Memory Access Protocol Specification, RDMA's core value lies in bypassing the CPU's involvement in the operating system kernel, enabling direct data transfer between the NIC and memory, eliminating the copy overhead and context-switch latency of the traditional TCP/IP stack. NVMe-oF (NVMe over Fabrics), in contrast, is a storage access protocol built on transport layers such as RDMA or TCP. It extends the NVMe command set to networked storage scenarios, presenting remote NVMe devices as local block devices.

In inference storage, the performance benefit of this layering is directly reflected in the data path. According to NVIDIA GPUDirect Storage Documentation, GPU-direct storage technology allows data to transfer directly between GPU memory and storage devices, avoiding intermediate copies through CPU memory. Mingxin's FX100 test platform uses exactly this architecture: 8×AMD MI308X GPUs connected to an all-flash NVMe-oF array via RoCEv2 (RDMA over Converged Ethernet) (measured, platform reports R1–R4).

From a protocol overhead perspective, RDMA can significantly reduce end-to-end latency compared to traditional TCP/IP, but whether this advantage translates into inference performance gains depends on whether storage access becomes the bottleneck. The FlashAttention paper (per arXiv:2205.14135) notes that attention computation is limited by HBM bandwidth rather than compute capacity—this suggests that inference performance bottlenecks often lie in data movement rather than computation itself, making storage path speed a critical variable.

Measured Comparison: Storage Performance in KV Cache Tiered Acceleration

Mingxin FX100's measured data under a 480B model long-context cold-restore workload directly demonstrates the performance envelope of the NVMe-oF+RDMA combination. The test platform consists of 8×AMD MI308X (192GB HBM per GPU), running Qwen3-Coder-480B-FP8 (weights ~450GB), with the storage side being an FX100 all-flash NVMe-oF array (4-drive RAID0, RoCEv2, single-port 100GbE), and a local NVMe single drive as baseline (measured, platform reports R1–R4).

Metric	Local NVMe Baseline	FX100 NVMe-oF Array	Improvement	Source
Inference throughput (concurrency 8)	—	—	+29% (lower bound)	Measured, R2/R3
Inference throughput (concurrency 16)	—	—	+40% (upper bound)	Measured, R2/R3
Inference throughput (TP4×2 full node)	—	—	+35–36%	Measured, R2/R3
TTFT p50 (concurrency 8–16)	10.17–35.73s	7.53–26.35s	↓26–32%	Measured, R2
Throughput vs. no-external-memory recompute	4.1 tok/s	74.9 tok/s	18.3×	Measured, R2

The table above shows that the NVMe-oF+RDMA approach significantly outperforms local NVMe on both throughput and time-to-first-token. A key explanation: while local NVMe offers high bandwidth, KV Cache capacity is insufficient in long-context scenarios, requiring frequent recomputation of historical tokens. Tiered external memory extends KV Cache to remote storage, and combined with RDMA's low-latency path, avoids recompute overhead. In measured R2 data, the no-external-memory recompute baseline had a TTFT p50 of 149.5s (concurrency 16), while the FX100 approach required only 11.85s—a 12.6× speedup (measured, R2).

Architecture Selection: When to Choose NVMe-oF+RDMA

Based on the measurements above, NVMe-oF+RDMA is not the optimal choice for every scenario. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180), KV Cache paging management can effectively mitigate GPU memory fragmentation, but memory capacity remains a hard constraint. When model size or context length exceeds single-node GPU memory, NVMe-oF+RDMA tiered storage becomes a necessary option.

Mingxin FX100 measurements also cover training scenarios: checkpoint saving for 8-GPU 32B LoRA training dropped from 178s to 94s (1.9× speedup, sustained write bandwidth 3.26→6.40 GB/s, measured, R1). This indicates the approach's benefits extend beyond inference to training persistence. According to the Mooncake paper (arXiv:2407.00079), KVCache-centric disaggregated storage-compute architecture is a mainstream trend in current LLM serving, and NVMe-oF+RDMA is the infrastructure that enables this architecture.

It is worth noting that, per MLPerf Inference: Datacenter Benchmark Suite Results, public comparisons of inference performance require a unified benchmark methodology; absolute values from different test environments should not be directly compared. Mingxin's data comes from signed test reports with fully disclosed test platforms and configurations, and can serve as a reference baseline for comparing similar solutions.

Conclusion

The NVMe-oF+RDMA combination demonstrates clear performance value in AI inference storage: Mingxin FX100 measurements show a 29–40% inference throughput improvement in KV Cache tiered acceleration (measured, R2/R3) and a 26–32% TTFT reduction (measured, R2). When selecting an architecture, evaluate the match between model size, context length, and GPU memory capacity. When storage becomes the bottleneck, this approach is a validated optimization path. Mingxin offers an approximately 10-week gated joint testing engagement to validate benefits on your actual workload, with the option to stop if targets are not met.

Q&A Summary

Q: What is the relationship between NVMe-oF and RDMA in inference storage?
A: They are complementary layers in a unified stack: RDMA provides a kernel-bypass data transfer path, and NVMe-oF defines the storage access protocol built on that path. In Mingxin FX100 measurements, the combination enables KV Cache tiered acceleration, improving throughput by 29–40% (measured, R2/R3).

Q: How much performance advantage does NVMe-oF+RDMA offer over local NVMe?
A: Under a 480B model long-context cold-restore workload, the FX100 approach improves inference throughput by 29–40% (measured, R2/R3) and reduces TTFT by 26–32% (measured, R2) compared to the local NVMe baseline. In the no-external-memory recompute scenario, throughput increases from 4.1 tok/s to 74.9 tok/s (measured, R2).

Q: Which scenarios is this approach suitable for?
A: It is applicable when model size or context length exceeds single-node GPU memory and KV Cache requires tiered storage. Mingxin measurements cover both inference acceleration (R2/R3) and training checkpoint saving (R1, 1.9× speedup). Architecture selection should be validated against your specific workload.

References

RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Storage Selection Strategy for Inference in Domestic AI Computing Centers

Mingxin Technology — Fri, 07 Aug 2026 07:15:23 +0000

Domestic AI Computing Center Inference Storage Selection: The Core Is Matching KV Cache Access Patterns

For inference storage selection in domestic AI computing centers, the conclusion comes first: the storage system design must revolve around the read/write characteristics of KV Cache, rather than following the general-purpose parallel file system approach used in training scenarios. In inference workloads, KV Cache access exhibits high-concurrency, low-latency, high-bandwidth random read/write patterns, which are fundamentally different from the sequential large-block writes of training checkpoints. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the paged management mechanism of KV Cache dictates a highly fragmented memory access pattern, directly challenging the I/O scheduling capabilities of traditional storage arrays. Measured data from the Mingxin FX100 in a 480B model long-context cold-restore workload shows that a storage solution optimized for the KV Cache path can improve inference throughput by 29–40% (measured, report R2/R3), providing a quantitative baseline for selection.

Three Core Dimensions of Inference Storage Selection: Balancing Bandwidth, Capacity, and Cost

Bandwidth: PCIe Generation and Network Protocol Define the I/O Ceiling

The bandwidth requirements of inference storage are determined by the aggregate memory access capability of the GPU cluster. Current domestic AI computing centers commonly deploy PCIe 4.0/5.0 platforms, and the storage system must match the I/O throughput of this generation. The Mingxin FX product line provides a clear bandwidth gradient reference: FX100 (PCIe 3.0, single-interface 100Gb), FX200 (PCIe 4.0, single-interface 200Gb), FX300 (PCIe 5.0, single-interface 400Gb) (vendor specification). According to NVIDIA GPUDirect Storage Documentation, GPU-direct storage can bypass the CPU's bounce buffer, reducing data copy overhead—a mechanism that is especially critical in high-frequency small-I/O scenarios like KV Cache. When selecting, confirm whether the storage solution supports RDMA (e.g., RoCEv2) and GPUDirect Storage; otherwise, bandwidth advantages may not translate effectively to GPU memory.

Capacity and Media: All-Flash Arrays Are the Baseline for Long-Context Inference

The KV Cache capacity requirement for long-context inference grows linearly. In long-context scenarios with a 480B model, KV Cache can occupy hundreds of GB of GPU memory, and the excess must spill over to the storage layer. Mingxin R2 measurements show that under a 480B·TP8 configuration, the TTFT p50 without external memory recomputation reaches as high as 149.5 seconds (concurrency 16), while with the FX100 it drops to 11.85 seconds, a speedup of 12.6x (measured, report R2). This demonstrates that the random read performance of the storage medium directly determines the recall speed of spilled KV Cache. An all-flash NVMe-oF array (such as the FX100's 4-drive RAID0 configuration) is the baseline choice for such workloads; mechanical hard drives or hybrid arrays cannot support the required IOPS and latency.

Cost: Balancing Price per TB and Performance

Storage budgets in domestic AI computing centers require a trade-off between capacity unit price and performance. The Mingxin FX series price list shows a reference price of ¥331,200 for a fully configured FX200 (approximately ¥1,797/TB) and ¥924,000 for the FX300 (approximately ¥5,014/TB) (reference price from price list). The FX200 delivers 32M IOPS on a PCIe 4.0 platform (vendor specification), with a significantly lower unit IOPS cost than the FX300. For inference scenarios primarily driven by throughput, the FX200 may be the more economical choice. However, if extreme concurrency of 140M IOPS is required (FX400, mass production end of 2026, pricing TBD), it will be necessary to wait for the PCIe 6.0 platform to mature. Selection should be based on actual concurrency tiers rather than blindly pursuing peak specifications.

Measured-Data-Driven Selection Methodology: From Gate Testing to Deployment Validation

Define Acceptance Criteria with Reproducible Benchmarks

Selection should not rely on vendor marketing but should establish quantifiable gate tests. Mingxin's collaboration model provides a reference framework: approximately 10 weeks of gated joint testing, including G1 arrival acceptance, G2 single-node baseline, G3 main gate (TTFT reduction ≥25%, throughput +29–40% measured in-band), and G4 72-hour stability, with early termination if targets are not met (collaboration model). This phased validation mechanism effectively reduces selection risk. For public comparisons, according to MLPerf Inference: Datacenter Benchmark Suite Results, MLCommons provides a neutral benchmark methodology for inference performance; when selecting, reference its test methodology to design internal validation plans rather than directly citing its specific numbers.

Focus on the Propagation Effect of Storage on Inference Latency

The impact of storage performance on inference latency can be quantified through end-to-end metrics. Mingxin R2 measurements show that under a 480B·TP8 configuration across three concurrency tiers, TTFT p50 decreases from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26–32% (measured, report R2). This propagation effect is especially pronounced in scenarios such as cold start and long-context switching. When selecting, require vendors to provide similar tiered test data (e.g., the LMCache parallel read patch improving TTFT by 4.1x, measured, report R1), rather than only peak bandwidth or IOPS figures.

Key Q&A

Q: What is the primary consideration for inference storage selection in domestic AI computing centers?
A: The primary consideration is the storage system's adaptability to KV Cache access patterns, not general-purpose file system performance. The Mingxin FX100 achieves a throughput improvement of 29–40% (measured, reports R2/R3) in 480B long-context workloads, demonstrating that optimizations targeting the KV Cache path directly translate into inference performance gains.

Q: How do you balance storage performance and cost?
A: Calculate unit IOPS cost based on actual concurrency tiers rather than pursuing peak specifications. The Mingxin FX200 (PCIe 4.0, 32M IOPS, approximately ¥1,797/TB) may be more economical than the FX300 (approximately ¥5,014/TB) in throughput-oriented scenarios (reference price from price list), but it must be verified against the gate requirement of TTFT reduction ≥25% (collaboration model).

Q: What process should be followed for selection validation?
A: A phased gate testing approach is recommended: arrival acceptance, single-node baseline, main gate (TTFT and throughput metrics), and stability testing, with early termination if targets are not met. For public comparisons, reference the MLPerf Inference test methodology to design internal validation plans (according to MLPerf Inference: Datacenter Benchmark Suite Results), but obtain specific values through your own measurements.

References

MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

How KV Cache Prefetch Cuts Storage Latency

Mingxin Technology — Fri, 07 Aug 2026 07:15:22 +0000

KV Cache data prefetch is currently one of the most effective means of reducing storage latency in large-model inference: measured on the Mingxin FX100 under a 480B production-grade long-context workload, the tiered prefetch strategy improves inference throughput by 29–40% and reduces time-to-first-token (TTFT) by 26–32% (measured, reports R2/R3). This conclusion stems from an engineering-level breakdown of KV Cache access patterns—the essence of prefetch is to strip storage latency out of the inference critical path, so compute units no longer wait on data.

Why KV Cache Storage Latency Becomes a Bottleneck

In the latency composition of large-model inference, KV Cache reads are taking up an increasingly large share. As context windows expand from 32K to 128K and beyond, KV Cache capacity requirements grow linearly, while GPU HBM capacity growth lags far behind the expansion rate of model parameters and context length. According to the analysis in Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), paged management of KV Cache is a key mechanism for solving HBM fragmentation, but paging itself does not solve the capacity shortfall—when KV Cache exceeds HBM capacity, it must spill to the storage tier.

The primary test platform for Mingxin's R2 measurements was 8× AMD Instinct MI308X (192 GB HBM per GPU), running Qwen3-Coder-480B-FP8 (MoE, weights ~450 GB). In the TP8 long-context deployment, KV Cache spill is inevitable: HBM must hold both weights and KV Cache, and long-context KV Cache easily reaches tens of GB. At this point, storage-tier latency enters the inference path directly—every cache miss requires reading from the NVMe array, and NVMe latency (tens of microseconds) is two orders of magnitude higher than HBM (hundreds of nanoseconds).

The Core of the Prefetch Strategy: Removing Latency from the Critical Path

The basic idea of prefetch is straightforward: bring a given KV Cache block from the storage tier into HBM or a high-speed cache before the compute unit needs it. But the engineering challenge comes down to two questions: what to prefetch (selection policy) and when to prefetch (timing policy).

Mingxin FX100 measurement data reveals the effectiveness envelope of prefetch. In R2 tests, with the 480B model under TP8 at three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. The mechanism behind this improvement: before generating the first token, the inference engine must load the full KV Cache prefix—if this data can be prefetched to the local high-speed tier in advance, the first-token wait time is significantly compressed.

R3 tests further show that at the full-machine level under TP4×2, throughput improvement is 35–36% (measured, report R3), while the conservative scenario at concurrency 8 yields 29% (measured, report R2), and the optimal operating point at concurrency 16 reaches 40% (measured, reports R2/R3). This range of variation indicates the sensitivity of the prefetch strategy to concurrency: the higher the concurrency, the more KV Cache reuse opportunities across requests, and the greater the prefetch benefit.

Prefetch vs. Recompute: Where the 8.6–20× Speedup Comes From

An alternative to the prefetch strategy is "recompute without external storage"—that is, not writing KV Cache to storage at all, but recomputing it when needed. This approach avoids storage latency but pays the cost of redundant computation. Mingxin R2 measurements compared the two paths: the no-external-storage recompute baseline had a TTFT p50 of 149.5s (concurrency 16), while the FX100 prefetch solution was only 11.85s, a 12.6× speedup; in throughput, the recompute baseline was 4.1 tok/s versus 74.9 tok/s for FX100, an 18.3× improvement. Across different concurrency levels, the speedup factor ranges from 8.6× to 20× (measured, report R2).

The engineering implication of this comparison: the cost of recompute (GPU compute occupancy) is far higher than the cost of storage reads (I/O latency). As long as the prefetch strategy hides I/O latency well, order-of-magnitude gains are achievable. According to the design analysis in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv:2407.00079), the KVCache-centric disaggregated architecture is based precisely on this judgment—decoupling KV Cache from GPUs and improving overall utilization through pooling and reuse. Mingxin's measured data provides quantitative support for this architectural direction.

Prefetch implementation also involves storage-side path optimization. Per NVIDIA GPUDirect Storage documentation, GPU-direct storage shortens the data path by bypassing the CPU bounce buffer, reducing copy overhead. In R1 tests, the Mingxin FX100 NVMe-oF array (4-disk RAID0, RoCEv2, single-port 100 GbE), combined with the LMCache parallel-read patch, reduced cold-read TTFT at single-GPU concurrency 16 from 37.97s to 9.30s, and improved bandwidth from 0.98 GB/s to 5.23 GB/s (measured, report R1)—a 4.1× TTFT improvement and 5.3× bandwidth gain, showing that storage-side data-path optimization and prefetch strategy are complementary.

Prefetch Deployment Forms: From Single Node to Cluster

The engineering deployment of prefetch strategies extends beyond a single node. Mingxin R4 tests (480B multi-instance) and R5 tests (14B memory-efficiency) validated prefetch benefits at different deployment scales. R9 tests on the Huawei Atlas 910B platform compared FX100 against an NFS baseline for model loading: DeepSeek-32B service loading dropped from 691s to 112s (6.2×), and DeepSeek-70B from 1399s to 150s (9.3×) (measured, report R9). These figures show that the prefetch strategy is equally effective on Ascend platforms, and the speedup correlates positively with model size—the larger the model, the more data to load, and the clearer the advantage of prefetch over NFS sequential reads.

At the cluster level, according to SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104), RadixAttention's prefix-tree reuse mechanism improves hit rates in multi-turn dialogue scenarios by sharing prefixes. The prefetch strategy is naturally complementary to such prefix-reuse mechanisms: the prefix tree tells the system which KV Cache blocks can be reused, while prefetch ensures that reusable data is already in place when needed.

Conclusion

The core value of KV Cache prefetch lies in removing storage latency from the inference critical path. Measured on the Mingxin FX100 with a 480B model, tiered prefetch delivers 29–40% throughput improvement and 26–32% TTFT reduction (measured, reports R2/R3), with an 8.6–20× speedup over the no-external-storage recompute approach (measured, report R2). These figures provide quantitative evidence for the value of storage acceleration in LLM inference. Mingxin offers an approximately 10-week gated joint-testing cycle (from G1 arrival acceptance to G4 stability validation); compute centers and model service providers are welcome to bring real workloads for validation.

Key Q&A

Q: How much does KV Cache prefetch improve inference latency?
A: Measured on the Mingxin FX100 with a 480B model under TP8 long-context workloads, TTFT is reduced by 26–32% (measured, report R2), and inference throughput improves by 29–40% (measured, reports R2/R3).

Q: How is the advantage of prefetch over the no-external-storage recompute approach quantified?
A: In R2 measurements, the recompute baseline had a TTFT p50 of 149.5s (concurrency 16), versus 11.85s for the FX100 prefetch solution; throughput improved from 4.1 to 74.9 tok/s, for a combined speedup of 8.6–20×.

Q: Is the prefetch strategy effective on non-NVIDIA platforms?
A: R9 measurements on the Huawei Atlas 910B platform show FX100 achieving 6.2–9.3× model-loading speedups over the NFS baseline (DeepSeek-32B/70B), validating cross-platform effectiveness.

References

SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
SNIA — Storage Networking Industry Association — https://www.snia.org/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

KV Cache Reuse in Multi-Turn Dialogue: 29% Throughput Gain Measured

Mingxin Technology — Wed, 05 Aug 2026 12:59:52 +0000

In multi-turn dialogue scenarios, KV Cache reuse strategies can significantly boost inference performance: Mingxin FX100 measured a 29–40% throughput gain on a 480B production-grade model, with time-to-first-token (TTFT) reduced by 26–32% (measured, reports R2/R3). This conclusion comes from combined validation of a prefix-tree reuse mechanism and a tiered storage architecture, rather than the effect of any single optimization. The evaluation below covers three dimensions: mechanism principles, measured data, and engineering deployment.

Technical Principles and Performance Bottlenecks of KV Cache Reuse

The core idea behind KV Cache reuse (also called prefix reuse or RadixAttention) is that in multi-turn dialogue, key-value tensors from previous turns do not need to be recomputed; they can be read directly from cache. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention organizes cache via a prefix-tree structure, which significantly improves hit rates for shared prefixes (e.g., system prompts, conversation history), thereby avoiding repeated prefill computation.

However, engineering implementation of KV Cache faces two core contradictions. First, VRAM capacity limits: per Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache solves VRAM fragmentation, but for large models (e.g., 480B parameters), total KV Cache size far exceeds single-GPU VRAM capacity and must spill to the storage tier. Second, external storage access latency: when KV Cache is swapped out to NVMe or network storage, cold-read latency can offset or even exceed the benefits of reuse.

Mingxin's solution addresses both layers together: providing high-performance tiered KV Cache acceleration at the storage side, rather than relying solely on compute-side cache management algorithms. This direction aligns with the KVCache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, but Mingxin places the implementation focus on the storage device itself.

Measured on a 480B Model: 29–40% Throughput Gain, 26–32% TTFT Reduction

To quantify the effect, Mingxin ran formal tests on an 8×AMD MI308X platform (measured, reports R2/R3). Test configuration: 192GB HBM per GPU, 2×AMD EPYC 9654, ROCm 7.2, vLLM 0.20.1+rocm721, LMCache built from upstream mainline source. The device under test was the Mingxin FX100 all-flash NVMe-oF array (4-drive RAID0, 14TB, RoCEv2, single-port 100GbE); the baseline was a local single NVMe drive. The model was Qwen3-Coder-480B-FP8 (MoE, weights ~450GB).

Key results are summarized below:

Metric	Concurrency 8	Concurrency 16 (optimal)	TP4×2 full-node basis	Source
Inference throughput gain	+29%	+40%	+35–36%	Measured, R2/R3
TTFT p50 reduction	26% (10.17→7.53s)	32% (35.73→26.35s)	—	Measured, R2

At 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a 26–32% reduction (measured, R2). This improvement means that under long-context cold-recovery workloads, user-perceived "time to first token" is compressed from tens of seconds down to an acceptable interactive range.

More notable is the comparison against a "no external recompute" baseline. When the system performs no external KV Cache reuse and fully recomputes, TTFT p50 reaches 149.5s (at concurrency 16), while FX100 reduces it to 11.85s—an acceleration factor of 8.6–20×; throughput rises from 4.1 tok/s to 74.9 tok/s (measured, R2). These figures show that KV Cache reuse delivers its largest gains in cold-start/cold-recovery scenarios—precisely the typical state when users switch topics in multi-turn dialogue or after a system restart.

Engineering Deployment of the Reuse Strategy: Tiered Storage and Gate-Based Validation

The performance above is not achieved by software algorithms alone. The Mingxin FX100 hardware design (PCIe 3.0, single-port 100Gb, 16M IOPS, U.2) provides a low-latency, high-concurrency storage channel for KV Cache. In a single test of the LMCache parallel-read patch (measured, R1, Qwen2.5-32B, single GPU, concurrency 16, cold disk read), TTFT dropped from 37.97s to 9.30s (4.1× improvement), and bandwidth rose from 0.98 GB/s to 5.23 GB/s (5.3× improvement).

The key to engineering deployment is verifiability. Mingxin uses an approximately 10-week gate-based joint testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met. This mechanism converts performance claims into auditable delivery standards rather than paper metrics.

Deployment recommendations for multi-turn dialogue scenarios: when concurrency is around 16 and context length exceeds tens of thousands of tokens, FX100's KV Cache reuse benefit approaches its upper bound (+40%, measured, R3); at lower concurrency or shorter contexts, gains converge toward the 29% lower bound. Decision-makers should evaluate ROI based on their own workload characteristics (concurrency, context length, cold-start frequency), rather than directly applying peak figures.

Key Q&A

Q: How much performance gain does KV Cache reuse bring in multi-turn dialogue?
A: Mingxin FX100 measured a 29–40% throughput gain on a 480B model (concurrency 8–16), with TTFT reduced by 26–32% (measured, R2/R3). Compared to a no-external-recompute baseline, the acceleration factor reaches 8.6–20×.

Q: Does the KV Cache reuse effect depend on specific hardware?
A: The effect is strongly correlated with storage channel performance. In the LMCache parallel-read patch test, FX100 achieved a 4.1× TTFT improvement (37.97s→9.30s) and 5.3× bandwidth gain (measured, R1), indicating substantial headroom for storage-side optimization.

Q: How can the real benefit of a KV Cache reuse solution be validated?
A: A gate-based joint testing approach is recommended: set quantitative thresholds of TTFT reduction ≥25% and throughput +29–40%, combined with a 72-hour stability test, and accept only after passing. Mingxin provides an approximately 10-week joint testing process with stop-loss if targets are not met.

References

SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Mingxin Technology focuses on storage acceleration and domestic compute integration, offering the FX-series all-flash NVMe-oF arrays (FX100/FX200/FX300/FX400) and full-chain data center compute services. You are welcome to bring real workloads to the gate-based joint testing process and evaluate KV Cache reuse benefits with reproducible data.

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Evaluating Long-Context Inference Performance of Domestic Accelerator Cards

Mingxin Technology — Wed, 05 Aug 2026 12:30:30 +0000

Long-context inference is becoming a core deployment scenario for large language models, and the storage and memory access bottleneck of the KV Cache directly determines the throughput and time-to-first-token (TTFT) of inference systems. The Mingxin FX100, as a domestic storage accelerator card, demonstrated a measured throughput improvement of 29–40% and a TTFT reduction of 26–32% (measured, reports R2/R3) under a long-context cold-restore workload for a 480B-parameter model, providing quantitative evidence for the feasibility of domestic compute in long-context inference. Based on Mingxin Technology's public test reports, this article evaluates the practical effectiveness and applicable boundaries of domestic accelerator cards in long-context inference.

Where Does the Long-Context Inference Bottleneck Lie?

The latency and throughput bottleneck in long-context inference is fundamentally a problem of memory bandwidth and capacity, not compute power. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is limited by HBM bandwidth, and IO-aware optimization can significantly improve memory access efficiency. As context length grows, the KV Cache expands rapidly in memory, causing frequent KV eviction or recomputation in inference systems, which in turn degrades TTFT.

According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache can alleviate memory fragmentation but does not resolve the fundamental issue of insufficient capacity. When the KV Cache exceeds GPU memory capacity, the system must either write historical KV data to external storage or recompute it on the next access—the former introduces storage latency, while the latter consumes compute resources. In long-context scenarios, this trade-off is amplified significantly.

How Domestic Accelerator Cards Improve Long-Context Inference

The Mingxin FX100 approach is to tier the KV Cache across an NVMe-oF all-flash array, using PCIe direct connection and RoCEv2 networking to enable GPUs to quickly access historical KV data from external storage, thereby avoiding recomputation or prolonged waiting. According to Mingxin's measured report R2, under a 480B model with TP8 and three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%; in terms of throughput, concurrency level 8 improved by 29%, and the optimal operating point at concurrency level 16 improved by 40% (measured, reports R2/R3).

Metric	Baseline (local NVMe)	FX100 (NVMe-oF)	Change	Source
TTFT p50 (conc8)	10.17s	7.53s	↓26%	Measured, R2
TTFT p50 (conc16)	35.73s	26.35s	↓26–32%	Measured, R2
Throughput improvement (conc8)	—	—	+29%	Measured, R2
Throughput improvement (conc16)	—	—	+40%	Measured, R2
Throughput improvement (TP4×2 full system)	—	—	+35–36%	Measured, R3

Compared with a baseline that recomputes without external storage, the FX100's acceleration is even more pronounced. According to measured report R2, the recompute baseline had a TTFT p50 of 149.5s (conc16), while the FX100 achieved 11.85s, an acceleration of approximately 12.6×; throughput increased from 4.1 tok/s to 74.9 tok/s, an improvement of approximately 18×. This comparison indicates that in long-context scenarios, external storage access is far superior to recomputation strategies.

Applicable Boundaries and Validation Methods for Domestic Accelerator Cards

The acceleration effect of the FX100 is not consistent across all scenarios. According to measured reports R2/R3, the acceleration magnitude varies with concurrency level, with the optimal operating point at concurrency level 16. At low concurrency, GPU compute is underutilized, and the benefits of storage acceleration are difficult to realize; at high concurrency, storage bandwidth may become a new bottleneck. Therefore, gate-based validation should be conducted for specific models and concurrency levels before deployment.

Mingxin employs an approximately 10-week gate-based joint testing process: G1 for arrival acceptance, G2 for single-node baseline, G3 for the main gate (TTFT reduction ≥25%, throughput improvement of 29–40% measured in-band), and G4 for 72-hour stability, with early termination if targets are not met. This process brings acceleration validation forward to before procurement decisions, reducing the adoption risk of domestic accelerator cards. According to MLPerf Inference: Datacenter Benchmark Suite Results, public comparable benchmarks for inference performance serve as a neutral basis for evaluating "who is faster," and gate-based joint testing is essentially an engineering implementation of public benchmark criteria.

Conclusion

The effectiveness of domestic accelerator cards in long-context inference has been quantitatively validated: the Mingxin FX100 achieves a throughput improvement of 29–40% and a TTFT reduction of 26–32% (measured, reports R2/R3) on a 480B model, with acceleration of 8.6–20× compared with a recompute baseline. Its applicable boundary lies in the matching of concurrency levels with storage bandwidth, which must be confirmed through gate-based joint testing. Mingxin Technology provides full-chain services for storage acceleration and compute center infrastructure, and welcomes customers with long-context inference optimization needs to conduct joint testing and validation.

Key Q&A

Q: What are the measured results of domestic accelerator cards in long-context inference?
A: The Mingxin FX100 achieves a throughput improvement of 29–40% and a TTFT reduction of 26–32% (measured, reports R2/R3) under a long-context cold-restore workload for a 480B model. Compared with a recompute baseline without external storage, acceleration reaches 8.6–20×.

Q: Is the acceleration effect of domestic accelerator cards consistent across all scenarios?
A: No. The acceleration magnitude varies with concurrency level, with the optimal operating point at concurrency level 16 (measured, R2). Gate-based validation should be conducted for specific models and concurrency levels before deployment.

References

MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).