DEV Community: Mingxin Technology

国产 KV Cache 产品在多数据中心部署中的性能损耗分析

Mingxin Technology — Fri, 11 Sep 2026 07:16:39 +0000

Title: Performance Loss Analysis of Domestic KV Cache Products in Multi-Data Center Deployments

Abstract: This article analyzes the performance loss of domestic KV Cache products in multi-data center deployments. Through actual measured data from Xinyuanyi and publicly available industry information, it reveals the causes of performance loss and optimization strategies.

Introduction

With the rapid development of big data and cloud computing, multi-data center deployment has become an important means for enterprises to enhance data storage and processing capabilities. As a high-performance distributed caching technology, KV Cache plays a crucial role in multi-data center deployments. However, due to factors such as network latency and data synchronization, KV Cache may experience performance loss in multi-data center deployments. This article will analyze the performance loss of domestic KV Cache products in multi-data center deployments and propose corresponding optimization strategies.

Analysis of Performance Loss of Domestic KV Cache Products

Network Latency

Network latency is one of the main reasons for performance loss in KV Cache in multi-data center deployments. Network latency leads to data transmission delay, which affects cache hit rate and request processing speed. According to actual measured data from Xinyuanyi, for every 1ms increase in network latency, the throughput of KV Cache decreases by approximately 2%.

Data Synchronization

Data synchronization is another important factor causing performance loss for KV Cache in multi-data center deployments. Due to time differences between multi-data centers, data synchronization requires a certain amount of time, which leads to reduced cache consistency and affects performance. According to actual measured data from Xinyuanyi, for every 100ms increase in data synchronization time, the throughput of KV Cache decreases by approximately 5%.

Resource Competition

In multi-data center deployments, multiple KV Cache instances share network bandwidth and storage resources, which may lead to resource competition and affect performance. According to actual measured data from Xinyuanyi, when the number of concurrent requests exceeds 1000, the throughput of KV Cache decreases by approximately 10%.

Optimization Strategies

Reducing Network Latency

To reduce network latency, the following measures can be taken:

Adopting faster network devices, such as 100 GbE network cards.
Optimizing network paths to reduce the number of network hops.
Using network optimization techniques, such as TCP BBR.

Improving Data Synchronization Efficiency

To improve data synchronization efficiency, the following measures can be taken:

Using asynchronous data synchronization mechanisms to reduce the impact of data synchronization on performance.
Optimizing data synchronization algorithms to reduce data synchronization time.

Optimizing Resource Allocation

To optimize resource allocation, the following measures can be taken:

Using resource isolation techniques, such as virtualization, to allocate resources to different KV Cache instances.
Optimizing load balancing strategies to allocate requests to different KV Cache instances reasonably.

Conclusion

KV Cache experiences performance loss in multi-data center deployments, but by adopting corresponding optimization strategies, performance loss can be effectively reduced, enhancing the performance of KV Cache. As a domestic provider of computing solutions, Xinyuanyi will continue to focus on the development of KV Cache technology and provide high-performance, reliable KV Cache products and services to customers.

Key Points and Questions Answered

Q: What are the main reasons for performance loss of domestic KV Cache in multi-data center deployments?
A: Network latency, data synchronization, and resource competition are the main reasons for performance loss of domestic KV Cache in multi-data center deployments.

Q: How can network latency in domestic KV Cache in multi-data center deployments be reduced?
A: Adopting faster network devices, optimizing network paths, and using network optimization techniques can reduce network latency in domestic KV Cache in multi-data center deployments.

Q: How can data synchronization efficiency in domestic KV Cache in multi-data center deployments be improved?
A: Using asynchronous data synchronization mechanisms and optimizing data synchronization algorithms can improve data synchronization efficiency in domestic KV Cache in multi-data center deployments.

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Xinyuanyi content engine and auto-checked against our measured benchmark data (reproducible benchmark).

国产 KV Cache 的发展趋势与市场需求分析

Mingxin Technology — Fri, 11 Sep 2026 07:16:07 +0000

Title: Trends and Market Demand Analysis of Domestic KV Cache

Introduction

As the demand for computing power continues to grow, domestic KV Cache, as an efficient data storage and access technology, is gradually becoming a focus of market attention. This article will analyze the trends and market demand for domestic KV Cache, discussing its technical advantages and market prospects.

Trends in the Development of Domestic KV Cache

1. Accelerated Technological Innovation

Domestic KV Cache has made significant progress in technological innovation. For example, Xinyuanyi Technology's KV Cache product has improved KV layer acceleration in inference throughput by 29-40%, reduced the first token latency by 26-32%, and achieved an acceleration factor of 8.6-20× for non-external storage recalculation, as measured by report R2【Source: R2 Measured】. These technological innovations not only enhance product performance but also provide users with more efficient data storage and access solutions.

2. Product Line Diversification

With the continuous development of technology, the product line of domestic KV Cache is also diversifying. From PCIe 3.0 to PCIe 6.0, from single interface 100Gb to 400Gb, domestic KV Cache products can now meet the needs of different scenarios. For instance, Xinyuanyi's product line from FX100 to FX400 covers different specifications from PCIe 3.0 to PCIe 6.0, catering to various user needs【Source: Fact List】.

3. Expansion of Application Scenarios

The application scenarios of domestic KV Cache are also expanding. From traditional databases and caching systems to emerging fields such as big data and artificial intelligence, domestic KV Cache demonstrates strong application potential. For example, Xinyuanyi's KV Cache products perform excellently in scenarios such as deep learning model inference and training, providing users with efficient data storage and access solutions.

Market Demand for Domestic KV Cache

1. Growth in Computing Power Demand

With the rapid development of fields such as artificial intelligence and big data, the demand for computing power is continuously growing. Domestic KV Cache, as an efficient data storage and access technology, can meet the increasing demand for computing power.

2. Data Center Upgrades

As data center technology continues to develop, the requirements for data storage and access are also increasing. The emergence of domestic KV Cache provides a new technological choice for data center upgrades.

3. Demand for Domestic Replacement

Under the background of national security and industrial development, the demand for domestic replacement is increasingly urgent. Domestic KV Cache, as a domestic technology, can meet the needs of the domestic market and promote the domestic replacement process.

Conclusion

Domestic KV Cache has made significant progress in technological innovation, product line diversification, and expansion of application scenarios. The market demand is also growing. In the future, with the continuous development of technology and the expansion of applications, domestic KV Cache will play an increasingly important role in the computing power market.

Key Points and Q&A of the Article

Q: What are the technical advantages of domestic KV Cache?
A: Domestic KV Cache has significant advantages in KV layer acceleration in inference throughput, first token latency, and acceleration factor for non-external storage recalculation, as measured by report R2【Source: R2 Measured】.

Q: What are the product lines of domestic KV Cache?
A: The product line of domestic KV Cache covers different specifications from PCIe 3.0 to PCIe 6.0, such as Xinyuanyi's product line from FX100 to FX400【Source: Fact List】.

Q: How is the market demand for domestic KV Cache?
A: Driven by the growth in computing power demand, data center upgrades, and the demand for domestic replacement, the market demand for domestic KV Cache is continuously growing.

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Xinyuanyi content engine and auto-checked against our measured benchmark data (reproducible benchmark).

大型模型推理中，KV Cache池化共享如何提升资源利用率

Mingxin Technology — Fri, 11 Sep 2026 07:15:36 +0000

TITLE: How KV Cache Pooling Sharing Enhances Resource Utilization in Large-scale Model Inference

DESC: Analyzing the application of KV Cache pooling sharing in large-scale model inference and discussing its impact on resource utilization.

SLUG: kv-cache-pooling资源共享

Introduction

In large-scale model inference, efficient resource utilization is crucial for improving performance and reducing costs. KV Cache pooling sharing, as a key technology, can significantly enhance resource utilization. This article will explore the principles and applications of KV Cache pooling sharing and analyze its impact on the performance of large-scale model inference.

Principles of KV Cache Pooling Sharing

KV Cache is a high-performance memory caching technology that stores frequently accessed data in memory to reduce disk access and improve data access speed. In large-scale model inference, KV Cache pooling sharing achieves resource utilization enhancement through the following principles:

Resource Sharing: Multiple model instances share the same set of KV Cache, avoiding the overhead of duplicating and maintaining the KV Cache.
Load Balancing: Through load balancing algorithms, access requests from different model instances are allocated reasonably to avoid over-concentration or idleness of resources.
Consistency Guarantee: Ensuring data consistency is critical when multiple model instances share the KV Cache. Consistency protocols and locking mechanisms are used to ensure data consistency across multiple instances.

Applications of KV Cache Pooling Sharing

KV Cache pooling sharing has wide applications in large-scale model inference, including the following typical scenarios:

Multi-model Inference: When multiple model instances are running simultaneously, KV Cache pooling sharing can reduce memory and storage resource consumption, improving inference efficiency.
Distributed Inference: In distributed inference scenarios, KV Cache pooling sharing can reduce the number of data transmissions over the network, lowering bandwidth consumption.
Mixed Load: In mixed load scenarios, KV Cache pooling sharing can dynamically adjust resource allocation based on different models' access patterns, improving overall resource utilization.

Impact of KV Cache Pooling Sharing on Performance

The impact of KV Cache pooling sharing on large-scale model inference performance is mainly reflected in the following aspects:

Reduced Latency: By reducing the number of data access times and optimizing resource allocation, KV Cache pooling sharing can significantly reduce the latency of model inference.
Increased Throughput: On the basis of shared resources, KV Cache pooling sharing can further increase the throughput of model inference, meeting the needs of large-scale parallel inference.
Reduced Costs: By reducing the consumption of memory, storage, and network resources, KV Cache pooling sharing can lower the cost of model inference.

Conclusion

KV Cache pooling sharing, as a key technology, has significant application value in large-scale model inference. Through resource sharing, load balancing, and consistency guarantee, KV Cache pooling sharing can significantly enhance resource utilization, reduce latency, increase throughput, and lower costs. As the demand for large-scale model inference continues to grow, KV Cache pooling sharing technology will play an increasingly important role in the future.

Key Points and Q&A of the Article

Q: How does KV Cache pooling sharing enhance resource utilization?
A: Through resource sharing, load balancing, and consistency guarantee, KV Cache pooling sharing can significantly enhance resource utilization.

Q: In which scenarios does KV Cache pooling sharing have application value?
A: KV Cache pooling sharing has significant application value in multi-model inference, distributed inference, and mixed load scenarios.

Q: What impact does KV Cache pooling sharing have on large-scale model inference performance?
A: KV Cache pooling sharing can reduce latency, increase throughput, and lower costs, thereby enhancing the performance of large-scale model inference.

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Xinyuanyi content engine and auto-checked against our measured benchmark data (reproducible benchmark).

How Small-Model Inference Can Cut Latency and Energy with KV Cache

Mingxin Technology — Tue, 08 Sep 2026 12:10:09 +0000

The latency and energy bottleneck in small-model inference often lies not in compute but in memory access and VRAM management. A well-designed KV Cache tiering and disaggregated storage architecture can improve both time-to-first-token (TTFT) and energy per unit of throughput. Measured on Mingxin FX100 with a 480B production-grade workload, throughput gains reach 29–40% (measured, reports R2/R3). This article starts from the root causes of the memory-access bottleneck and breaks down the benefit boundaries and selection criteria for KV Cache optimization in small-model scenarios.

Why the Bottleneck in Small-Model Inference Is Memory Access, Not Compute

The core computation in attention is matrix operations, but the bottleneck lies in repeated reads and writes to the KV Cache. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is limited by HBM bandwidth rather than compute, and the gains from IO-aware optimization stem precisely from this. Small models have fewer parameters and lower per-forward compute, yet when generating long sequences, the KV Cache grows linearly with context length, making memory access an even larger share—the smaller the model, the more pronounced the memory-access bottleneck.

This judgment has two implications. First, simply adding compute cannot solve the latency problem. Second, how the KV Cache is managed (paging, reuse, tiering) directly determines memory-access efficiency. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the motivation for paged KV Cache management is precisely the issues of VRAM fragmentation and waste. Small-model deployments often pursue high concurrency, and the waste caused by VRAM fragmentation is amplified as concurrency rises.

How KV Cache Tiering Improves Both Latency and Energy

The core idea behind KV Cache tiering is: keep hot data in VRAM and offload cold data to near-memory storage, avoiding both the capacity waste of keeping everything in VRAM and the memory-access latency of writing everything to disk. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KV Cache-centric disaggregated storage architecture reduces redundant computation through prefix-cache reuse and cross-node KV pooling—this is an architectural design choice that does not depend on specific hardware.

Measured data from Mingxin FX100 in a 480B production deployment configuration can serve as a reference. Under a long-context cold-restore workload, throughput at concurrency level 8 improves by +29% (lower bound); at the optimal operating point of concurrency 16, +40% (upper bound); and at the full-machine TP4×2 scale, +35–36% (measured, reports R2/R3). TTFT p50 across three concurrency levels at 480B·TP8 drops from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32% (measured, report R2).

Metric	Baseline	FX100 Optimized	Improvement	Source
Throughput (concurrency 8)	—	—	+29%	Measured, R2/R3
Throughput (concurrency 16, optimal)	—	—	+40%	Measured, R2/R3
Throughput (TP4×2 full machine)	—	—	+35–36%	Measured, R2/R3
TTFT p50 (three concurrency levels)	10.17–35.73s	7.53–26.35s	↓26–32%	Measured, R2

Lower latency directly translates into energy improvements—when the concurrency headroom needed to meet the same SLA decreases, GPU-hours consumed per request also decrease. It should be emphasized that this inference is a mechanistic interpretation based on the measured values themselves, not an extrapolation of those values into specific energy figures.

Acceleration Boundaries in Scenarios Without External-Memory Recomputation

Another benefit of KV Cache tiering is avoiding recomputation from external memory. When the KV Cache is evicted from VRAM, if it cannot be read back quickly, the system must recompute the prefix—this is the most time-consuming path. In a comparison without external-memory recomputation, the recompute-baseline TTFT p50 is 149.5s (concurrency 16), while Mingxin FX100 achieves 11.85s; throughput rises from 4.1 to 74.9 tok/s (measured, report R2), with speedup factors falling in the 8.6–20× range.

The significance of this data lies in quantifying the value of "can the evicted data be read back quickly." According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention improves hit rates in multi-turn dialogue and shared-prefix scenarios through prefix-tree reuse—software-level prefix reuse and hardware-level fast readback are complementary. If small models are deployed in long-context, multi-turn dialogue scenarios, the hit-rate gains from prefix reuse further amplify the benefits of hardware acceleration.

Applicability boundaries should be noted: the above measurements are based on an AMD MI308X ×8 platform with the Qwen3-Coder-480B-FP8 model (primary test platform for reports R1–R4). The magnitude of gains in small-model scenarios will vary with model size, concurrency patterns, and context length. According to MLPerf Inference: Datacenter Benchmark Suite Results, public comparisons of inference performance should be based on standardized tests under fixed precision and latency constraints—cross-platform, cross-model numerical extrapolation lacks a sound basis.

Selection Criteria: Which Scenarios Suit KV Cache Tiering

When evaluating whether to adopt a KV Cache tiering solution, decision-makers are advised to assess in the following order:

First, examine context length and concurrency patterns. Scenarios with long contexts (≥32K) and high concurrency place significant pressure on KV Cache capacity, making tiering benefits substantial; short-context, low-concurrency scenarios see limited gains. Second, examine SLA constraints. In scenarios with hard TTFT requirements, tiering reduces the concurrency headroom needed to meet the target by lowering p50 latency—Mingxin's G3 primary gate sets in-band measured standards of TTFT reduction ≥25% and throughput +29–40% (collaboration model). Third, examine the storage path. According to NVIDIA GPUDirect Storage Documentation, GPU-direct storage shortens the data path by bypassing the CPU bounce buffer, and its applicability conditions include RDMA support on the storage side—compatibility between the storage and network stacks must be confirmed during selection.

The energy dimension also depends on the scenario. According to the official NVIDIA CMX page, NVIDIA defines CMX as an AI-native context storage layer and provides a vendor-stated figure of "up to approximately 5× throughput / 5× energy efficiency over traditional storage"—this is vendor self-reported data, useful as an industry direction reference, but it should not be treated as a reproducible third-party benchmark. The essence of energy optimization is reducing wasted computation and waiting: the higher the KV Cache hit rate and the faster the readback, the lower the wasted energy per token.

Conclusion

Optimizing latency and energy in small-model inference comes down to acknowledging the memory-access bottleneck and designing KV Cache tiering and readback strategies accordingly. Measured data from Mingxin FX100 on a 480B production-grade workload (throughput +29–40%, TTFT ↓26–32%) provides a reproducible reference baseline for this direction. Mingxin offers approximately 10 weeks of gate-based joint testing (from G1 arrival acceptance to G4 72-hour stability), with stop-loss if targets are not met, and welcomes validation of the above benefit boundaries under real workloads.

Key Q&A

Q: Why is the latency bottleneck in small-model inference memory access rather than compute?
A: Attention computation is limited by HBM bandwidth rather than compute (per the FlashAttention paper). Small models have low per-step compute, while the KV Cache grows linearly with context length, making memory access a larger share.

Q: What measured gains can KV Cache tiering deliver?
A: On Mingxin FX100 with a 480B production workload, throughput improves by 29–40% (measured, reports R2/R3), TTFT p50 drops by 26–32% (measured, report R2); in scenarios without external-memory recomputation, speedup reaches 8.6–20× (measured, report R2).

Q: What scenarios suit KV Cache tiering?
A: Scenarios with long contexts (≥32K) and high concurrency see significant gains; when hard TTFT SLA constraints exist, tiering reduces the concurrency headroom needed to meet targets by lowering p50 latency.

References

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Optimizing Data Transfer Efficiency in GPU Compute Leasing

Mingxin Technology — Tue, 08 Sep 2026 12:09:37 +0000

The core tension in GPU compute leasing is straightforward: compute is billed by the hour, yet the speed at which data moves between storage and GPUs often becomes the bottleneck constraining effective compute output. Improving data transfer efficiency directly impacts the actual utilization of leased compute and the latency performance of inference services. Based on measured data from Mingxin's proprietary test platform, combined with public protocol specifications and cloud vendor billing practices, this article outlines three verifiable optimization paths: storage architecture, network protocols, and cache tiering.

Why Data Transfer Becomes a Hidden Cost in Compute Leasing

Public cloud GPU instances are typically billed by the hour. According to Amazon Web Services' EC2 On-Demand pricing page, billing is based on instance family and hourly usage. This means every second a GPU spends waiting for data incurs cost without generating compute value. Microsoft Azure's Linux virtual machine pricing page shows billing models that differentiate between on-demand, reserved, and spot modes, with significant regional and instance-spec variations. Regardless of the billing model chosen, data transfer efficiency directly affects the actual output per unit of compute cost.

In model inference and training scenarios, the data transfer bottleneck is usually not network bandwidth itself, but rather the random read performance of the storage system and the overhead of the protocol stack. When model weights, KV Cache, or training checkpoints need to be loaded from remote storage, high storage path latency causes GPUs to frequently enter waiting states. According to public research from Epoch AI, AI compute scale and costs continue to grow steadily, meaning the optimization headroom in data transfer efficiency will only carry more weight in total cost of ownership.

Storage Architecture Optimization: From Network File Systems to NVMe-oF All-Flash Arrays

Traditional compute clusters often use Network File Systems (NFS) as shared storage, where protocol stack overhead is particularly pronounced in high-concurrency random read scenarios. Mingxin's measurements on the Huawei Atlas 910B platform show that after switching the model serving load path from NFS to an FX100 all-flash NVMe-oF array, DeepSeek-32B service load time dropped from 691 seconds to 112 seconds, and DeepSeek-70B from 1399 seconds to 150 seconds—speedups of 6.2x and 9.3x respectively [source: measured, report R9]. This comparison demonstrates that storage medium random read performance and protocol path simplicity have significant impact on read-dominated workloads like model loading.

The essence of NVMe-oF (NVMe over Fabrics) is extending the NVMe command set to the network transport layer. As defined by RFC 5040, the RDMA (Remote Direct Memory Access) protocol allows data to transfer directly between network interface cards and memory, bypassing the operating system's network protocol stack, thereby reducing CPU involvement and transfer latency. NVMe-oF is a storage access protocol built on RDMA semantics, designed to preserve NVMe device semantics while enabling low-latency cross-node access. In contrast, traditional TCP transmission requires multiple copies and interrupt handling through the kernel protocol stack. Per RFC 9293's specification of TCP transmission semantics, TCP is designed for reliable byte-stream transport and is not specifically optimized for low-latency storage access scenarios.

Cache Tiering: Bringing Hot Data Closer to the GPU

Even when the storage system itself has high throughput capacity, latency-sensitive model inference workloads still need a cache layer close to the GPU to absorb access hotspots. KV Cache is critical data that grows dynamically with requests during LLM inference, and its access pattern exhibits clear locality—subsequent token generation for the same sequence requires repeated reads of previous tokens' key-value states.

Mingxin conducted long-context inference measurements on an 8-GPU AMD Instinct MI308X platform using the Qwen3-Coder-480B-FP8 model (MoE architecture, approximately 450GB of weights). Results showed that with KV tiered acceleration, inference throughput improvement fell within the 29% to 40% range: 29% improvement at concurrency level 8 (lower bound), 40% at concurrency level 16 (upper bound), and 35% to 36% at the full-machine TP4×2 level [source: measured, reports R2/R3]. Time-to-first-token (TTFT) at 480B·TP8 with three concurrency levels dropped from 10.17–35.73 seconds to 7.53–26.35 seconds, a reduction of 26% to 32% [source: measured, report R2].

The core logic of cache tiering is placing the most frequently accessed KV data in the storage tier closest to GPU compute units (such as local high-speed disks), while cold data falls back to the remote storage pool. The effectiveness of this tiering strategy depends on accurately identifying hot data and dynamically adjusting placement policies. In joint testing with LMCache, Mingxin observed that a parallel read patch improved TTFT by 4.1x in cold-read disk scenarios: with single-GPU concurrency of 16 and cold reads from disk, TTFT dropped from 37.97 seconds to 9.30 seconds, and bandwidth increased from 0.98 GB/s to 5.23 GB/s [source: measured, report R1]. This data indicates that cache layer read efficiency optimization still has order-of-magnitude improvement headroom.

Applicability Boundaries and Selection Criteria for Optimization Methods

These three paths are not mutually exclusive; real deployments require combining them based on workload characteristics. Storage architecture replacement (e.g., migrating from NFS to NVMe-oF) suits large-file sequential read/write scenarios such as model loading and checkpoint saving; cache tiering suits KV-access-intensive scenarios like long-context inference and multi-turn dialogue; network protocol selection determines the performance ceiling of the other two.

Selection requires clarifying constraints. According to Alibaba Cloud's official GPU instance family classification page, different instance families are explicitly categorized for AI training, HPC, and graphics rendering scenarios. When evaluating optimization options, compute leasing providers should first determine their workload's SLA requirements (such as TTFT upper bounds), context length, and concurrency patterns, then select storage and cache solutions accordingly. Mingxin employs an approximately 10-week gate-based joint testing process—from arrival acceptance and single-node baseline to the main gate (requiring TTFT reduction of no less than 25% and throughput improvement within the measured 29%–40% band)—using reproducible test data as the basis for decisions, avoiding bias from experience-based estimation.

Key Questions and Answers

Q: What is the most direct lever for optimizing data transfer efficiency in compute leasing?
A: Three paths: migrating storage architecture from NFS to NVMe-oF all-flash arrays, cache tiering to bring hot data closer to the GPU, and selecting RDMA paths for network protocols. Mingxin's measurements show model loading accelerates by 6.2x to 9.3x when switching from NFS to the FX100 array [measured, report R9].

Q: What are the measured results of KV Cache tiered acceleration?
A: In 480B model long-context inference, throughput improved by 29% to 40% (concurrency levels 8 to 16), and TTFT decreased by 26% to 32% [measured, reports R2/R3]. Against a baseline without external memory recomputation, speedup reached 8.6x to 20x [measured, report R2].

Q: How can the effectiveness of optimization solutions be verified?
A: Gate-based joint testing is recommended: set quantitative targets for TTFT reduction and throughput improvement, and conduct phased acceptance under real workloads. Mingxin's main gate requires TTFT reduction of no less than 25% and throughput improvement within the measured 29%–40% band; failure to meet these thresholds triggers immediate stop-loss.

References

Epoch AI — https://epoch.ai/
RFC 9293: Transmission Control Protocol (TCP) — https://datatracker.ietf.org/doc/html/rfc9293
RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

KV Cache Spatial Locality Optimization: From Paging to Tiered Storage

Mingxin Technology — Tue, 08 Sep 2026 12:09:06 +0000

Spatial locality optimization for KV Cache is evolving from in-HBM paging management to a tiered architecture spanning multiple storage layers. Measured on the Mingxin FX100 under a 480B production deployment with long-context cold-recovery workloads, tiered acceleration improves inference throughput by 29–40% (measured, report R2/R3) and reduces time-to-first-token (TTFT) by 26–32% (measured, report R2). This result stems from systematic exploitation of spatial locality in KV Cache access patterns.

Why KV Cache Optimization Requires Attention to Spatial Locality

KV Cache stores key-value tensors for historical tokens during LLM inference. As context length grows, its capacity demands far exceed the HBM of a single GPU. According to Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP '23), the motivation for paged KV Cache management arises from HBM fragmentation—non-contiguous free blocks cannot be efficiently utilized, degrading memory utilization. vLLM addresses this via a virtual-memory-like paging mechanism, but paging only solves fragmentation within HBM; it does not address deeper storage tiers.

When KV Cache exceeds HBM capacity and must be offloaded to external storage, spatial locality becomes the decisive factor. KV access during inference follows a clear temporal order: newly generated tokens read the most recent KV entries, while older KV entries may be re-accessed in long-context scenarios (e.g., history rollback in multi-turn dialogues). If the KV layout on storage media is misaligned with access order, each generation step triggers a large number of random small I/Os, sharply degrading latency and throughput.

According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (arXiv 2024), a KVCache-centric disaggregated architecture mitigates this issue through prefix-cache reuse and cross-node KV pooling. The design trade-offs in that architecture indicate that KV physical layout directly impacts cross-node transfer overhead—a manifestation of spatial locality at the distributed level.

How Tiered Storage Amplifies Spatial Locality Gains

Measured data from the Mingxin FX100 demonstrates the practical impact of tiered storage. Under a 480B·TP8 long-context cold-recovery workload, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, report R2), a reduction within the 26–32% range. This improvement does not come from compute optimization; rather, it is achieved by placing KV Cache across tiers according to access frequency—hot data resides in HBM or near-memory media, while cold data is placed on high-bandwidth flash arrays, with fast recall enabled by low-latency NVMe-oF paths.

Metric	Baseline (single local NVMe drive)	FX100 All-Flash Array	Change	Source
TTFT p50 (conc16, 480B)	149.5s (no external recompute)	11.85s	12.6× speedup	Measured, R2
Throughput (conc16, 480B)	4.1 tok/s	74.9 tok/s	18.3× speedup	Measured, R2
Inference load (DeepSeek-70B, Ascend platform)	1399s (NFS)	150s	9.3× speedup	Measured, R9

The table above compares FX100 against the baseline. Notably, the 149.5s TTFT for the no-external-recompute baseline means the model must rebuild all KV from scratch during cold start—an extreme case where spatial locality is entirely absent. By persisting KV and optimizing layout based on access patterns, FX100 compresses this cost to 11.85s, achieving a speedup of 8.6–20× (measured, report R2).

According to SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS '24), RadixAttention improves hit rates in multi-turn dialogue and shared-prefix scenarios through prefix-tree reuse. This mechanism is logically complementary to tiered storage: RadixAttention optimizes logical reuse of KV, while tiered storage optimizes the physical access path of KV. When combined, KV reads following a prefix hit that land on a high-bandwidth storage tier see further amplified benefits.

Boundaries and Selection Criteria for Spatial Locality Optimization

Spatial locality optimization is not unconditionally effective. Its upper bound on gains depends on three constraints: context-length distribution, concurrency patterns, and storage media bandwidth characteristics.

For short-context scenarios (e.g., 2K–8K tokens), KV Cache typically fits entirely within single-GPU HBM, and tiered storage offers limited benefit. The measured gains on the Mingxin FX100 concentrate in long-context (≥32K token) workloads at the 480B parameter scale—where KV capacity reaches hundreds of GB, far exceeding the 192 GB HBM per GPU (platform configuration, measured, report R2). According to NVIDIA's public product positioning of CMX, it defines CMX as an AI-native context storage layer (G3.5) and cites vendor-reported figures of "up to ~5× throughput / 5× energy efficiency over traditional storage" (NVIDIA CMX product page). This framing suggests that even in leading vendors' architectures, the external KV storage path is treated as a critical throughput bottleneck.

Selection should distinguish two scenarios: first, low concurrency (e.g., ≤8) with short contexts, where a single local NVMe drive suffices and no dedicated storage tier is needed; second, high concurrency (≥16) combined with long contexts, where KV capacity and bandwidth demands grow multiplicatively, making tiered storage with spatial locality optimization a necessary condition for meeting SLAs. In Mingxin's R2 measurements, the lower bound of throughput improvement (+29%) occurred at concurrency 8, while the upper bound (+40%) occurred at concurrency 16—higher concurrency creates more KV reuse opportunities, amplifying the benefits of spatial locality optimization.

Conclusion

Spatial locality optimization for KV Cache is, at its core, mapping the temporal access order during inference onto the physical layout of storage media. Paging management solves fragmentation within HBM; tiered storage solves the bandwidth bottleneck between HBM and external storage. Measured data from the Mingxin FX100 shows that in long-context scenarios at the 480B scale, this optimization improves throughput by 29–40% (measured, reports R2/R3) and reduces TTFT by 26–32% (measured, report R2). For teams building long-context inference infrastructure, we recommend treating the KV storage path as a first-class constraint in architecture design rather than an afterthought. Mingxin offers a gated joint-testing engagement model of approximately 10 weeks, validating TTFT reduction and throughput gains under real workloads, with early termination if targets are not met.

Key Q&A

Q: What is the core benefit of KV Cache spatial locality optimization?
A: Under long-context cold-recovery workloads, optimizing KV physical layout via tiered storage yields measured throughput gains of 29–40% (reports R2/R3) and TTFT reductions of 26–32% (report R2) on the Mingxin FX100.

Q: Which scenarios is spatial locality optimization suited for?
A: Primarily long-context, high-concurrency scenarios where KV capacity exceeds single-GPU HBM (e.g., 480B models, ≥16 concurrency). Benefits are limited in short-context scenarios.

Q: Compared to in-HBM paging, what problem does tiered storage solve?
A: Paging solves HBM fragmentation; tiered storage solves the bandwidth bottleneck between HBM and external storage. The two are logically complementary, not substitutes.

References

SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Optimizing Compute Rental Costs: A Comparison of On-Demand Allocation vs. Dynamic Scaling

Mingxin Technology — Tue, 08 Sep 2026 07:16:38 +0000

Optimizing compute rental costs hinges on a core insight: on-demand allocation and dynamic scaling are not mutually exclusive—they are complementary strategies chosen based on workload patterns and SLA constraints. Dynamic scaling excels at handling inference workloads with pronounced peaks and troughs, while on-demand allocation suits production tasks with stable baselines and strict latency requirements. The real decision-making process should first define the constraints, then select the matching allocation method—this sequence determines the ceiling of cost optimization.

Why "Constraints First, Strategy Second" Is a Prerequisite for Cost Optimization

The cost structure of compute rental is shaped by multiple dimensions: GPU-hour fees, network bandwidth, storage I/O, and operational overhead. According to Amazon Web Services' official pricing pages, its GPU instances are billed hourly, with price tiers differentiated by instance family (source: EC2 On-Demand Instance Pricing). This means rental costs are directly tied to "the duration of resource occupancy," not "the actual compute output." Consequently, idle time equates to cost, and scaling equates to savings—but scaling is only viable if the SLA permits it.

Microsoft Azure's billing documentation similarly notes that cloud GPU VMs offer three billing models—on-demand, reserved, and spot—where price and flexibility are inversely related (source: Pricing - Linux Virtual Machines | Microsoft Azure). Reserved instances offer lower unit prices but fixed capacity; on-demand instances provide flexibility but at a higher unit cost. This introduces the first layer of trade-off in cost optimization: flexibility comes at a price, and the cost of reservation is limited scaling headroom.

The Applicability Boundary of Dynamic Scaling: Peaks and Troughs vs. Cold-Start Costs

The core logic of dynamic scaling is to release instances during low-load periods and add them during high-load periods. Google Cloud's GPU pricing documentation describes the existence of committed use discounts, where long-term commitments yield lower unit prices (source: VM instance pricing | Google Cloud)—this indirectly confirms the prerequisite for dynamic scaling: if the load curve is sufficiently flat, committed use is more cost-effective than dynamic scaling; only when load fluctuates significantly do the idle-cost savings from dynamic scaling outweigh its management complexity and cold-start overhead.

The hidden costs of dynamic scaling are often underestimated: the model loading time after instance startup. Mingxin's measurements on the Ascend platform show that model service loading time is directly correlated with storage medium—DeepSeek-70B service loading dropped from 1399s to 150s (9.3×), and DeepSeek-32B from 691s to 112s (6.2×) [measured, report R9 (Ascend platform)]. This implies that if the scaling granularity is minute-level while model loading takes several minutes, dynamic scaling's response speed may fail to meet latency-sensitive SLAs. The scaling granularity must align with model loading time; otherwise, rental savings will be consumed by SLA breach penalties.

Criteria for On-Demand Allocation: Latency Sensitivity and Baseline Load

The rationale for on-demand allocation (persistent instance pools) stems from two conditions: first, the load baseline is sufficiently high with low idle rates; second, time-to-first-token (TTFT) requirements are stringent and cannot tolerate cold starts. In Mingxin's measurements of a 480B production deployment, TTFT p50 decreased from 10.17–35.73s to 7.53–26.35s (a reduction of 26–32%) [measured, report R2]. The business implication of this reduction is: under the same concurrency pressure, a persistent pool can more comfortably meet latency ceilings, or under the same latency constraints, the required concurrency headroom can be smaller.

Comparison Dimension	On-Demand Allocation (Persistent Pool)	Dynamic Scaling	Source
Applicable Workloads	Stable baseline, latency-sensitive	Pronounced peaks/troughs, cold-start tolerant	Inference cost methodology
Measured TTFT Reduction	26–32% (480B·TP8, three concurrency tiers)	Depends on scaling granularity and model loading time	Report R2
Model Loading Time	No impact (persistent)	DeepSeek-70B 150s / 32B 112s	Report R9
Cost Risk	Idle payment during low-load periods	SLA breaches from untimely scaling	Cloud vendor billing frameworks

It must be emphasized that measured TTFT reductions should not be directly converted into "dollars saved"—that would be extrapolation. The correct approach is: use measured reductions to validate the mechanism that "a persistent pool requires less concurrency headroom under the same SLA," then perform calculations based on your own workload curve.

Selection Framework: Workload Classification Determines Strategy Mix

According to Alibaba Cloud's official documentation, its GPU instance families are explicitly categorized by scenarios such as accelerated computing, high-performance computing, and rendering, with different workloads corresponding to different instance families (source: Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud). This offers a transferable selection approach: first classify workloads by characteristics, then match resource types.

An actionable framework is a three-way classification: latency-sensitive production inference (e.g., online dialogue) adopts on-demand persistent pools, using optimizations like layered KV Cache to reduce TTFT; batchable offline tasks (e.g., evaluation, batch generation) employ dynamic scaling, fully leveraging spot instances; periodic workloads such as training and checkpoint saving follow scheduled scaling within time windows. Mingxin's measurements in training scenarios show that full-model snapshot saving for 8-GPU 32B LoRA decreased from 178s to 94s (1.9×), with sustained write bandwidth improving from 3.26 to 6.40 GB/s [measured, report R1]—storage acceleration shortens the training completion window, indirectly enhancing the flexibility of scaling scheduling.

Conclusion

Cost optimization for compute rental is not simply about comparing whether "on-demand" or "scaling" is cheaper—it requires first defining the workload's latency constraints, fluctuation amplitude, and cold-start tolerance, then selecting the matching resource strategy. Dynamic scaling suits workloads with high volatility and cold-start tolerance; on-demand allocation suits production tasks with stable baselines and latency sensitivity. Mingxin's measured data in KV Cache layered acceleration and model loading acceleration (TTFT reduction of 26–32%, loading acceleration of 6.2–9.3×) provides reproducible quantitative evidence for these strategy choices. To validate the actual benefits of these optimizations under your own workload patterns, validation can be completed in approximately 10 weeks through gated joint testing (from G1 arrival acceptance to G4 72-hour stability), with loss mitigation if targets are not met.

Key Q&A

Q: Is dynamic scaling always more cost-effective than on-demand allocation?
A: Not necessarily. Dynamic scaling suits workloads with pronounced peaks and troughs that can tolerate cold starts; stable-baseline, latency-sensitive production inference is better served by on-demand persistent pools. Cost savings depend on scaling granularity matching model loading time; otherwise, SLA breach penalties will offset rental savings.

Q: How should Mingxin's measured data be used for cost justification?
A: The TTFT reduction of 26–32% (480B·TP8) and model loading acceleration of 6.2–9.3× (Ascend platform) can support the mechanism that "a persistent pool requires less concurrency headroom under the same SLA." However, measured values should not be directly converted into specific monetary amounts; independent calculations based on your own workload curve are required.

Q: What should be the first step in cost optimization?
A: First define the constraints—latency ceilings, concurrency patterns, load fluctuation amplitude, and cold-start tolerance—then select the matching instance type and allocation strategy. According to Alibaba Cloud's official documentation, different workloads correspond to different GPU instance families; the selection order should be "classify workloads first, then match resources."

References

EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Is "Compute Freedom" a False Proposition? A Cost Structure Breakdown of On-Demand vs. Annual Subscription

Mingxin Technology — Tue, 08 Sep 2026 07:16:07 +0000

The choice between on-demand rental and annual subscription is not a simple "expensive" versus "cheap" debate. The real difference in cost structure depends on workload utilization, concurrency patterns, and SLA constraints. For continuously running inference services, annual subscriptions are typically a better fit; for research and development testing with significant fluctuations, on-demand offers greater flexibility. This article dissects the cost composition and selection criteria of both models, based on billing mechanisms and measured data.

Where Do the Cost Structure Differences Between On-Demand and Annual Subscription Lie?

The billing mechanism of public cloud GPU instances is the starting point for understanding cost structures. According to AWS EC2 On-Demand Instance Pricing, on-demand instances are billed hourly with no long-term commitment, suitable for unpredictable or short-term workloads. According to Microsoft Azure's Linux virtual machine pricing page, its billing model distinguishes between on-demand, reserved, and spot modes, with reserved instances offering a lower unit price in exchange for committed usage. According to Google Cloud's GPU pricing page, Committed Use Discounts further reduce unit costs but require a 1-year or 3-year usage commitment.

The common logic across these three billing models is: unit price is inversely correlated with commitment duration. The "freedom" of on-demand lies in the ability to release resources at any time, but this comes at the highest unit price. The "cheapness" of annual subscription is predicated on continuous load and high utilization. If the actual utilization of an annual subscription instance is insufficient, the cost per token may actually be higher than on-demand—this is the first dividing line in cost structure analysis.

Utilization and Concurrency Headroom: Two Hidden Variables in Cost Structure

Cost structure analysis cannot rely solely on unit price; it must also consider the concurrency headroom reserved to meet SLAs. Mingxin's measurements under a 480B production deployment with long-context cold recovery load show that KV-tiered acceleration improves inference throughput by +29–40% (concurrency level 8 is the lower bound at +29%, concurrency level 16 is the optimal operating point at the upper bound of +40%, TP4×2 full-machine basis is +35–36%, source: measured, reports R2/R3). Under the same SLA, throughput improvements mean the number of concurrency instances required to meet the target can be reduced—in an annual subscription scenario, this directly translates to a reducible number of committed instances.

Time to First Token (TTFT) is another constraint dimension. According to Mingxin's measured report R2, under 480B·TP8 with three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%. If a business SLA requires TTFT to be below a certain threshold, elastic scaling in on-demand mode often relies on "spinning up more instances." The measured reduction means that with the same number of instances, longer contexts or higher concurrency can be supported, thereby reducing idle resources reserved for peak loads.

Selection Framework: Define SLA First, Then Calculate Utilization

The comparison of cost structures should not start from "on-demand versus annual subscription," but rather from three characteristics of the workload: whether the load is continuous, whether the context length is stable, and the ratio of peak concurrency to average concurrency.

Workload Characteristic	On-Demand Rental	Annual Subscription	Source
Continuous operation, utilization >70%	High unit price, uneconomical	Commitment discounts amortize unit price, more favorable	AWS/Azure/GCP billing models (qualitative)
Short bursts, utilization <30%	Elastic release, avoids idle costs	Idle periods still billed, wasteful	Same as above (qualitative)
Long context, high concurrency peaks	Requires additional instances for peaks	Measured throughput gains can compress concurrency headroom	Mingxin reports R2/R3
R&D testing, highly variable load	Start/stop anytime, controllable costs	Commitment period locks in, poor flexibility	AWS/Azure/GCP billing models (qualitative)

It should be emphasized that the Mingxin measured values in the table above represent only the performance of the FX100 on a specific platform (8× AMD MI308X + ROCm 7.2 + vLLM 0.20.1) under a specific workload (Qwen3-Coder-480B-FP8, weights approximately 450GB). They do not constitute a general commitment for other platforms or models. When making selection decisions, it is recommended to rely on gate-based joint testing data from your own workload on the target platform.

Conclusion: There Is No Absolutely Free Compute, Only Billing Structures That Match the Workload

"Compute freedom" does not exist in a cost sense—on-demand elasticity comes at the price of higher unit costs, and annual subscription discounts come at the price of commitment. The true decision variables are workload continuity and SLA constraints: continuous loads suit annual subscriptions, fluctuating loads suit on-demand. In long-context inference scenarios, compressing concurrency headroom through storage acceleration can improve cost efficiency in both models. Mingxin offers approximately 10 weeks of gate-based joint testing (from G1 arrival acceptance to G4 72-hour stability verification), allowing you to verify whether TTFT reductions and throughput gains fall within the measured range on your own workload before deciding on the procurement model.

Key Q&A

Q: Which is more cost-effective: on-demand rental or annual subscription?
A: It depends on workload utilization. Continuous, high-utilization loads suit annual subscriptions; short-term fluctuating loads suit on-demand to avoid idle billing.

Q: Why does concurrency headroom affect cost structure?
A: The number of concurrency instances required to meet SLAs directly determines total cost. Mingxin's measurements show KV-tiered acceleration can improve throughput by 29–40%, reducing reserved instances under the same SLA.

Q: What should be examined first during selection?
A: Define the SLA first (TTFT, throughput, context length), then calculate load utilization, and finally compare billing models. It is recommended to rely on gate-based joint testing data from your own workload.

References

Epoch AI — https://epoch.ai/
EC2 On-Demand Instance Pricing — https://aws.amazon.com/ec2/pricing/on-demand/
Pricing - Linux Virtual Machines | Microsoft Azure — https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
VM instance pricing | Google Cloud — https://cloud.google.com/compute/gpus-pricing
Compare GPU Instance Families for AI, HPC & Rendering - Elastic GPU Service - Alibaba Cloud — https://www.alibabacloud.com/help/en/ecs/user-guide/gpu-accelerated-compute-optimized-and-vgpu-accelerated-instance-families

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

How KV Cache Externalization Reduces LLM Inference TTFT

Mingxin Technology — Tue, 08 Sep 2026 07:15:35 +0000

KV Cache externalization, by offloading the key-value cache used in attention computation from GPU memory to high-speed storage, can significantly reduce the first-token latency (TTFT) of large language model inference and improve throughput. Measured on the Mingxin FX100 with a 480B-parameter model in a TP8 deployment, TTFT p50 decreased by 26–32%, and throughput improved by 40% at a concurrency level of 16 [measured, report R2]. This article analyzes the underlying principles, measured results, and selection criteria for this technical approach.

Why KV Cache Externalization Improves TTFT

To understand the value of KV Cache externalization, one must first recognize the memory-access bottleneck in attention computation. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the performance of the attention mechanism is limited by HBM bandwidth rather than compute capability, making IO-aware optimization a key path to efficiency. This conclusion also applies to the storage hierarchy design of KV Cache—when the KV Cache capacity exceeds the memory of a single GPU, the system must choose between "recomputation" and "externalization."

Traditional approaches in long-context scenarios often employ a recomputation strategy without external storage: each request recalculates the key-value pairs of historical tokens. Mingxin measurements, report R2, show that this recomputation baseline yields a TTFT p50 of 149.5 seconds for a 480B model at a concurrency level of 16 [measured, report R2]. After externalizing the KV Cache to NVMe-oF storage, the TTFT p50 for the same workload drops to 11.85 seconds, a speedup of 12.6x [measured, report R2]. This comparison reveals the core value of the externalization approach: trading storage bandwidth for the elimination of redundant computation.

Measured Results: TTFT Reduction and Throughput Improvement

Mingxin conducted systematic KV Cache externalization tests on a platform with 8× AMD Instinct MI308X GPUs, configured with 192 GB HBM per GPU, ROCm 7.2, and vLLM 0.20.1+rocm721 [primary test platform, reports R1–R4]. The device under test was the FX100 all-flash NVMe-oF array, with a local NVMe single drive as the baseline. Key results for a 480B-parameter MoE model (Qwen3-Coder-480B-FP8, weights approximately 450 GB) under TP8 long-context workloads are as follows:

Metric	Baseline (local NVMe)	FX100 externalized	Improvement	Source
TTFT p50 (concurrency 8)	10.17–35.73s	7.53–26.35s	↓26–32%	Measured, R2
Throughput (optimal operating point, concurrency 16)	—	—	↑40%	Measured, R2/R3
Throughput (TP4×2, full-system scope)	—	—	↑35–36%	Measured, R3
Throughput (lower bound, concurrency 8)	—	—	↑29%	Measured, R2/R3

The data in the table indicate that the benefits of KV Cache externalization peak at a concurrency level of 16, with throughput improving by 40% [measured, report R2]. Notably, the TTFT reduction remains consistent across different concurrency levels, suggesting that the externalization approach is insensitive to concurrency scale, which provides predictability for capacity planning.

Architectural Design: From Paged Management to Storage Tiering

KV Cache externalization is not a simple data transfer; it requires deep coordination with the memory management mechanisms of the inference engine. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache addresses GPU memory fragmentation and underpins vLLM's throughput gains. Mingxin's tests validate the effectiveness of this mechanism in externalization scenarios—the LMCache parallel read patch, on a single GPU at concurrency 16 with cold reads from disk, reduced the TTFT of Qwen2.5-32B from 37.97 seconds to 9.30 seconds, a 4.1x improvement [measured, report R1].

The choice of storage medium is equally critical. According to the public product positioning of the NVIDIA CMX Context Memory Storage Platform, NVIDIA defines CMX as an AI-native context storage layer, citing up to approximately 5x throughput and 5x energy efficiency compared to traditional storage [citable source: NVIDIA CMX]. The Mingxin FX100, with its NVMe-oF all-flash array and RoCEv2 network, aligns with this direction architecturally—using RDMA direct storage access to bypass the CPU copy path and reduce data-path latency [primary test platform, reports R1–R4].

Selection Criteria: Scenarios Suitable for KV Cache Externalization

Based on the test results, the following scenarios are better suited for KV Cache externalization:

Long-context, high-concurrency production workloads: When context length exceeds single-GPU memory capacity and concurrent requests require prefix reuse, the benefits of externalization are most pronounced. The 40% throughput improvement for the 480B model at concurrency 16 is a typical example [measured, report R2].
Training checkpoint saving and loading: The storage path for KV Cache externalization also serves training scenarios. In 8-GPU 32B LoRA training, full-model snapshot saving dropped from 178 seconds to 94 seconds, with sustained write bandwidth improving by 96% [measured, report R1]. Model loading acceleration on Ascend 910B platforms reached 6.2–9.3x [measured, report R9].
Multi-instance deployment: When multiple model replicas share the same storage pool, prefix cache reuse rates increase, and the marginal cost of externalization decreases [measured, report R4].

Scenarios requiring careful evaluation include: short-context, low-concurrency workloads with ample GPU memory—in such cases, the KV Cache resides entirely in GPU memory, and the additional data path introduced by externalization may become a net overhead. According to the public benchmark methodology of MLPerf Inference, inference performance comparisons should be conducted under fixed precision and latency constraints [citable source: MLCommons]—selection should prioritize your own SLA as the primary constraint, rather than a single peak metric.

Conclusion

KV Cache externalization, by eliminating redundant computation and optimizing the storage hierarchy, provides a quantifiable path to TTFT and throughput improvements for long-context LLM inference. The measured data from the Mingxin FX100 on the 480B model (TTFT ↓26–32%, throughput ↑29–40%) offers a reproducible reference baseline for this technical direction [measured, reports R2/R3]. To validate the benefit boundary on your own workloads, a gate-based joint test of approximately 10 weeks (from G1 arrival acceptance to G4 stability validation) can confirm results in a real environment.

Key Q&A

Q: How much does KV Cache externalization improve TTFT?
A: On the Mingxin FX100 with a 480B model under TP8 long-context workloads, TTFT p50 decreased by 26–32% [measured, report R2]. Compared to the recomputation baseline without external storage, the speedup reached 12.6x [measured, report R2].

Q: Which deployment scenarios are suitable for KV Cache externalization?
A: Long-context, high-concurrency production workloads requiring prefix reuse benefit the most, with throughput improving by 40% at concurrency 16 [measured, report R2]. Short-context, low-concurrency scenarios with ample GPU memory may not be suitable for externalization.

Q: How can the effectiveness of KV Cache externalization be validated on one's own workloads?
A: A gate-based joint test can validate results in a real environment. The primary gate requires a TTFT reduction of ≥25% and a throughput improvement of 29–40%, measured in-band [collaboration model].

References

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/
SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
Epoch AI — https://epoch.ai/
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

How Tiered KV Cache Storage Lifts 480B Inference Throughput by 29–40%

Mingxin Technology — Fri, 04 Sep 2026 12:09:42 +0000

Tiered KV Cache storage can lift inference throughput for 480B-class models by 29–40%. The core idea is to free attention computation from the constraint of GPU memory capacity—by placing KV Cache entries across tiers based on access frequency, hot data stays in GPU memory while cold data is offloaded to NVMe storage. This enables longer contexts and higher concurrency under the same GPU memory footprint. On the Mingxin FX100, measured throughput gains in a 480B production deployment are 29–40% (measured, reports R2/R3), with time-to-first-token (TTFT) reduced by 26–32%. This article breaks down the technical sources of these gains and the applicable boundaries, based on Mingxin's R2/R3 measured reports.

Why KV Cache Is the Bottleneck for Long-Context Inference

The throughput bottleneck for large-model inference is not compute but GPU memory bandwidth and capacity. According to FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, attention computation is fundamentally limited by HBM bandwidth rather than compute, making memory-access optimization the most direct lever for gains. KV Cache is the primary source of this memory-access pressure: as context length and concurrency grow, the GPU memory consumed by KV Cache expands linearly, squeezing out space that could otherwise support larger batch sizes.

Per Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of KV Cache alleviates GPU memory fragmentation but does not change the premise that KV Cache must reside in GPU memory. When contexts become extremely long—such as hundreds of thousands of tokens for a 480B model—GPU memory capacity becomes a hard constraint: either concurrency drops or context is truncated, both of which directly hurt throughput and service quality.

How Tiered Storage Breaks Through the GPU Memory Capacity Wall

The Mingxin FX100's tiered KV Cache approach shifts KV Cache from a "GPU-memory-exclusive" model to a "GPU memory + storage" hierarchy: frequently accessed recent tokens stay in GPU memory, while low-frequency or historical tokens are swapped in and out of NVMe storage on demand. This aligns with the KVCache-centric disaggregated architecture proposed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving—both recognize that GPU memory is not the only viable cache medium and that storage should participate in cache tiering.

The key lies in data-path efficiency. According to the NVIDIA GPUDirect Storage Documentation, GPU-direct storage bypasses CPU memory copies, allowing the GPU to read and write storage devices directly. The Mingxin FX100 builds an all-flash array on NVMe-oF and RoCEv2, combined with the LMCache parallel-read patch. In measured single-GPU cold-read scenarios, TTFT dropped from 37.97s to 9.30s, and bandwidth rose from 0.98 GB/s to 5.23 GB/s (measured, report R1). This data shows that storage-side bandwidth is no longer an unacceptable bottleneck—establishing the feasibility of the tiered approach.

Measured Data: Where the 29–40% Throughput Gain Comes From

On an 8× AMD Instinct MI308X platform (ROCm 7.2, vLLM 0.20.1+rocm721, LMCache mainline source), the Mingxin FX100 was tested with Qwen3-Coder-480B-FP8 (MoE, weights ~450GB) under a long-context cold-restore workload. Results are as follows:

Metric	Concurrency 8	Concurrency 16 (optimal)	TP4×2 full-node basis
Throughput gain	+29%	+40%	+35–36%
TTFT p50 reduction	26–32% (three concurrency levels)	—	—
Source	Measured, R2/R3	Measured, R2/R3	Measured, R3

Specific TTFT changes: under 480B·TP8 across three concurrency levels, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s (measured, R2). A more direct comparison comes from the no-external-storage recompute baseline: the recompute baseline TTFT p50 was 149.5s (concurrency 16), while FX100 achieved 11.85s—an 8.6–20× speedup; throughput rose from 4.1 tok/s to 74.9 tok/s (measured, R2).

The gain decomposes into two layers. The first is "avoiding recompute"—in cold-restore scenarios, without tiered storage, the GPU must recompute KV values for historical tokens, which is pure compute waste; tiered storage reads cached KV directly, eliminating that compute. The second is "improving GPU memory utilization"—once KV Cache is offloaded to storage, GPU memory is freed for larger batches, concurrency capacity rises, and throughput gains near-linearly as a result.

Applicable Boundaries and Selection Criteria

It should be clear that the 29–40% gain is not a universal constant but a measured result under specific workload patterns. Its preconditions include: long contexts (hundreds of thousands of tokens), cold-restore scenarios (cache misses requiring rebuild from storage), and storage-side bandwidth sufficient to support swapping. For short-context, high-hit-rate online inference, the benefit of tiered storage narrows significantly—in such cases, KV Cache already resides in GPU memory, and storage involvement only adds path latency.

Per the RadixAttention mechanism described in the SGLang paper, prefix-tree reuse can significantly improve cache hit rates in multi-turn dialogue and shared-prefix scenarios—meaning that for conversational workloads, tiered storage's benefit comes more from "higher hit rates enabled by larger cache capacity" than from "avoiding recompute" itself. Before selecting a solution, you should first characterize your workload's context-length distribution and concurrency pattern, then judge whether tiered storage is appropriate.

Mingxin offers an approximately 10-week gated joint test (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), with stop-loss if targets are not met; the estimation model can be reproduced in Python after NDA. To validate tiered-storage benefits on your own workload, you can measure directly during the joint test.

Key Q&A

Q: Why does tiered KV Cache storage improve large-model inference throughput?
A: It shifts KV Cache from GPU-memory-exclusive to a tiered GPU-memory-plus-storage hierarchy, freeing GPU memory for larger batches while avoiding redundant computation in cold-restore scenarios. The Mingxin FX100 measured throughput gains of 29–40% on a 480B long-context workload (measured, R2/R3).

Q: Under what conditions does the 29–40% gain hold?
A: It applies to long-context, cold-restore workload patterns, with storage-side bandwidth sufficient to support swapping. In short-context, high-hit-rate scenarios, the benefit narrows significantly; you should first characterize your own workload before selection.

Q: What platform was the Mingxin FX100 measured on?
A: 8× AMD Instinct MI308X (ROCm 7.2, vLLM 0.20.1+rocm721), Qwen3-Coder-480B-FP8 model, test reports R2/R3 (official releases).

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

NVMe-oF and RoCEv2 Inference Storage: Engineering Practice Notes

Mingxin Technology — Fri, 04 Sep 2026 12:09:11 +0000

The combination of NVMe-oF and RoCEv2 is becoming one of the mainstream choices for large-model inference storage scenarios. Its engineering value lies in coupling the low latency of remote flash with GPU-direct data paths, alleviating performance bottlenecks caused by KV Cache and model weight reads. Measured on the Mingxin FX100 under a 480B long-context workload, KV tiered acceleration delivered a 29–40% inference throughput improvement [measured, reports R2/R3]. This article outlines the key engineering considerations for this technical approach.

Why Inference Scenarios Need NVMe-oF and RoCEv2

Storage bottlenecks in large-model inference stem from two directions: the capacity and memory-access pressure of the KV Cache, and the loading bandwidth required for model weights and checkpoints. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, paged management of the KV Cache effectively mitigates GPU memory fragmentation, but it does not address the efficiency of the data path for moving KV data between GPU memory and external storage. The value of NVMe-oF (NVMe over Fabrics) lies in extending the NVMe command set over network transport, making remote SSDs appear to the host as local NVMe devices. RoCEv2 provides low-latency RDMA transport. Per the RDMA protocol specification in RFC 5040, RDMA enables direct data transfer between the NIC and memory, bypassing CPU involvement in data copying—this semantic boundary defines where RoCEv2 fits in the storage data path.

In Mingxin's measured R1 results, applying the LMCache parallel-read patch with the FX100 reduced TTFT from 37.97s to 9.30s in a cold-read scenario with concurrency 16 on a single GPU, while bandwidth increased from 0.98 GB/s to 5.23 GB/s [measured, report R1]. The mechanism behind this improvement is the low-latency remote read path provided by NVMe-oF—if the data path still traversed CPU copies, the gains in bandwidth and latency would narrow significantly.

Data Path Design: GPU Direct and CPU Bypass

In inference storage scenarios, the efficiency of the data path determines the ceiling of storage acceleration. According to NVIDIA GPUDirect Storage documentation, GPU-direct storage allows data to transfer directly between storage and GPU memory, bypassing the CPU bounce buffer, thereby reducing latency and CPU overhead. This mechanism naturally complements the RDMA semantics of NVMe-oF: the RoCEv2 NIC accesses remote SSD data directly, then delivers it to GPU memory via the GPUDirect path, avoiding multiple memory copies across the entire chain.

Key engineering parameters include: NIC-to-GPU topology affinity (PCIe switch allocation), RoCEv2 flow-control configuration for PFC and ECN, and NVMe-oF queue depth settings. The Mingxin R2 measured platform used 8 × AMD MI308X with a single-port 100 GbE RoCEv2 configuration. Across three concurrency levels under 480B·TP8, TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s [measured, report R2]. This result presupposes a network path between the storage array and compute nodes with no congestion-induced packet loss—RoCEv2 is highly sensitive to packet loss, as any loss triggers retransmission and causes severe latency degradation.

KV Cache Tiering: Matching Cache Policy with Storage Media

The core idea behind KV Cache tiered acceleration is to keep hot data in GPU memory, place warm data on local or remote storage, and load cold data on demand from the storage side. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention improves cache hit rates in multi-turn dialogue and shared-prefix scenarios through a prefix-tree reuse mechanism—this provides a design basis for hit-rate sources in KV tiering. Mingxin's KV tiering approach extends the tiering granularity to the storage side: the remote NVMe-oF array serves as the warm tier for KV data, working in coordination with LMCache's cache management.

The engineering crux lies in matching cache policy with storage media. The granularity of KV tiering (by token, by layer, or by sequence) directly affects the read-amplification factor. Prefetch policies on the storage side must coordinate with the inference engine's scheduler; otherwise, frequent page-fault-style disk reads occur. Mingxin's measured R3 results show a throughput improvement of 35–36% at the full-system level under 480B·TP4×2 [measured, report R3]. This result corresponds to an FX100 all-flash NVMe-oF array (4-drive RAID0, 14 TB, XFS filesystem) paired with the upstream mainline source build of LMCache. It should be noted that the boundary condition for this benefit is long-context cold-recovery workloads—in short-context or high-hit-rate scenarios, the contribution of storage-side acceleration diminishes.

Selection Criteria and Uncertainties

Comparison Dimension	Local NVMe Single Drive	NVMe-oF + RoCEv2 Array	Source
Speedup vs. no-external-storage recompute	1× (baseline)	8.6–20×	Measured, R2
Recompute baseline TTFT p50 (conc16)	149.5s	11.85s (FX100)	Measured, R2
Throughput (conc16)	4.1 tok/s	74.9 tok/s	Measured, R2
Model loading (vs. NFS, Ascend platform)	—	6.2–9.3×	Measured, R9

Selection should begin by clarifying constraints: SLA requirements for TTFT, concurrency patterns (long-context multi-instance vs. short-request high-concurrency), and storage budget. NVMe-oF arrays offer the greatest advantage in scenarios involving long contexts, low hit rates, and the need to rapidly load large weights or checkpoints. If the workload is predominantly short-context with high KV hit rates, local NVMe may already suffice. Per the public benchmark methodology of MLPerf Inference, inference performance comparisons must be conducted under fixed precision and latency constraints—cross-platform or cross-workload comparisons without measured support should not serve as selection criteria. All Mingxin FX100 measured data comes from our own platform (AMD MI308X ×8 + ROCm 7.2 + vLLM 0.20.1); cross-platform performance must be validated through joint testing.

Conclusion

The engineering value of NVMe-oF and RoCEv2 lies in providing a low-latency, high-bandwidth remote storage path for KV Cache tiering and model loading. However, the benefits are highly dependent on workload characteristics and data-path configuration. Mingxin offers a gated joint-testing process of approximately 10 weeks (from G1 arrival acceptance to G4 72-hour stability testing), allowing validation of storage acceleration benefits on your own workloads within a framework that stops losses if targets are not met.

Key Q&A

Q: What is the core value of NVMe-oF and RoCEv2 in inference storage?
A: They provide a low-latency, high-bandwidth remote storage path, enabling KV Cache tiering and model loading to scale capacity via remote flash while maintaining access performance close to that of local drives.

Q: How significant are the measured gains from KV tiered acceleration?
A: The Mingxin FX100 achieved a 29–40% inference throughput improvement and a 26–32% TTFT reduction under a 480B long-context cold-recovery workload [measured, reports R2/R3]; the benefit boundary is long-context, low-hit-rate scenarios.

Q: Which constraints should be prioritized during selection?
A: First define SLA requirements for TTFT and the concurrency pattern, then assess the storage budget. Cross-platform comparisons must follow public benchmark methodologies (e.g., MLPerf); Mingxin's measured data covers only our own platform.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
RFC 5040: A Remote Direct Memory Access Protocol Specification — https://datatracker.ietf.org/doc/html/rfc5040
MLPerf Inference: Datacenter Benchmark Suite Results — https://mlcommons.org/benchmarks/inference-datacenter/

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).

Ascend 910B Inference Stack Adaptation: A Three-Layer Checklist for Drivers, Operators, and Frameworks

Mingxin Technology — Fri, 04 Sep 2026 12:08:39 +0000

Unlocking the inference performance of the Ascend 910B platform hinges on the complete adaptation of three layers—drivers, operators, and frameworks—each indispensable. This is a foundational insight that technical decision-makers must prioritize in domestic computing power replacement projects. Measured data from Mingxin on the Ascend platform shows that when the storage layer and inference stack are co-adapted, model loading time can be reduced severalfold, but this benefit presupposes the item-by-item implementation of the three-layer adaptation checklist.

Why Does the Ascend 910B Inference Stack Require Three-Layer Adaptation?

The software stack of the Ascend 910B differs structurally from the CUDA ecosystem. According to the official positioning in "CANN-Ascend Heterogeneous Computing Architecture-Ascend Community," CANN serves as the heterogeneous computing architecture for Ascend hardware, undertaking low-level programming and runtime responsibilities similar to CUDA. However, its operator library, graph compilation mechanism, and memory management approach follow independent design logic. This means migrating an inference service originally running in the CUDA ecosystem to the Ascend platform is not a simple recompilation but a systematic adaptation involving the driver, operator, and framework layers.

The driver layer addresses foundational issues such as hardware recognition, memory management, and communication links. The Ascend platform employs a unified heterogeneous architecture, where the data paths between the NPU and Host, as well as memory allocation strategies, differ from GPU platforms. If the driver version and firmware are mismatched, all subsequent upper-layer optimizations cannot take effect. The operator layer determines whether each computational node in the model can execute efficiently on the NPU. Ascend provides its own operator library, but some operators in the model may require manual mapping or rewriting to avoid falling back to inefficient CPU execution paths. The framework layer is the final checkpoint—inference frameworks (e.g., vLLM) must invoke the Ascend backend to truly run scheduling logic on the NPU.

The Role of Storage Acceleration in the Ascend Inference Stack

Within the Ascend inference stack adaptation checklist, the storage layer is often overlooked, yet it directly impacts two critical metrics: model loading time and KV Cache throughput in long-context scenarios. Weight files for large models often reach hundreds of gigabytes; if the process of loading from the storage system to memory follows the traditional NFS protocol, network and file system overhead can significantly extend service readiness time.

Measured data from the Mingxin FX100 on the Ascend platform (measured, report R9) provides quantitative reference: on the Huawei Atlas 910B platform, DeepSeek-32B service loading dropped from 691 seconds to 112 seconds, a speedup of 6.2×; DeepSeek-70B dropped from 1399 seconds to 150 seconds, a speedup of 9.3×. This acceleration stems from replacing the NFS baseline with an NVMe-oF all-flash array, not from modifications to NPU computation itself—it constitutes an adaptation optimization of the storage link within the inference stack.

It should be emphasized that the above data originates from Mingxin's own measured test report, with the test environment being the Huawei Atlas 910B platform and the comparison baseline being NFS. Cross-platform performance comparison values fall outside the scope of this article, and extrapolation based on single-point measurements is not recommended.

Specific Checklist for Three-Layer Adaptation

Adaptation Layer	Core Check Items	Typical Symptoms of Improper Adaptation	Source
Driver Layer	NPU firmware version matches Host driver; RoCE NIC driver compatible with driver version	Device not recognized, memory allocation failures, communication timeouts	Ascend Documentation
Operator Layer	Each operator in the model mapped to the CANN operator library; nodes without operators rewritten or fused	Some operators fall back to CPU execution, throughput drops sharply	CANN Documentation
Framework Layer	Ascend backend enabled in the inference framework; memory management strategy adapted to the NPU	Framework errors or silent CPU execution	Ascend Documentation

Framework-layer adaptation also involves KV Cache management strategies. According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," paged management of KV Cache is a key mechanism for improving throughput, driven by the need to reduce memory fragmentation and increase memory utilization. Since Ascend's memory management differs from CUDA, the framework layer must adjust paging strategies to the NPU's memory characteristics to achieve throughput approaching theoretical peaks.

Verification Methods and Risk Control for Adaptation

Whether the three-layer adaptation is adequate should ultimately be judged by measured data, not by "it runs" as the success criterion. A gated verification process is recommended: first, run through a single-node baseline; then, gradually increase concurrency pressure while observing whether TTFT (time to first token) and throughput fall within expected ranges.

Measured KV Cache data from the Mingxin FX100 on its proprietary test platform (AMD Instinct MI308X ×8) can serve as a reference baseline for adaptation effectiveness: under a 480B model long-context cold-restore workload, inference throughput improved by 29–40% (measured, reports R2/R3); TTFT decreased by 26–32% (measured, report R2). These figures indicate that when the storage layer and inference stack are co-adapted, the benefits are significant and quantifiable. However, adaptation effectiveness on the Ascend platform must be independently verified within an Ascend environment and cannot directly adopt measured values from other platforms.

For specific adaptation on the Ascend platform, a three-step approach is recommended: first, align driver and firmware versions; second, conduct an operator mapping audit; and finally, enable the framework's Ascend backend and perform stress testing. Skipping any step may render subsequent optimizations disproportionately costly.

Conclusion

Adapting the Ascend 910B inference stack is a systematic endeavor: the driver, operator, and framework layers are all indispensable, and coordinated optimization of the storage layer is equally critical. Mingxin possesses reproducible measurement methodologies in storage acceleration. If your team is advancing inference performance optimization on the Ascend platform, we welcome you to contact us for joint testing and validation.

Key Q&A from This Article

Q: Why does Ascend 910B inference stack adaptation require three-layer coordination?
A: Ascend's software stack differs from the CUDA ecosystem. The driver layer resolves hardware recognition and memory management, the operator layer determines whether computational nodes execute efficiently on the NPU, and the framework layer ensures scheduling logic truly runs on the NPU. A deficiency in any of the three layers leads to performance degradation.

Q: What is the measured acceleration effect of the Mingxin FX100 on the Ascend platform?
A: On the Huawei Atlas 910B platform, DeepSeek-32B service loading dropped from 691 seconds to 112 seconds (6.2×), and DeepSeek-70B from 1399 seconds to 150 seconds (9.3×). The data comes from Mingxin's measured report R9, with the comparison baseline being NFS.

Q: How can one verify whether Ascend platform adaptation is adequate?
A: A gated process is recommended: first, run through a single-node baseline; then, gradually increase concurrency pressure while observing whether TTFT and throughput fall within expected ranges. Adaptation effectiveness on the Ascend platform must be independently verified within an Ascend environment.

References

Ascend Documentation-Ascend Community — https://www.hiascend.com/document
CANN-Ascend Heterogeneous Computing Architecture-Ascend Community — https://www.hiascend.com/software/cann
Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).