Cold restart of long-context sessions—reloading KV Cache and model weights after service restarts or cache invalidation—is the most underestimated performance bottleneck in agent and code-assistant scenarios. Measured data from Mingxin FX100 shows that with KV-tiered acceleration, inference throughput for a 480B model under long-context cold-restart workloads can improve by 29–40%, while time-to-first-token (TTFT) decreases by 26–32% (measured, reports R2/R3). This finding indicates that the recovery storm is not unsolvable; rather, it can be effectively mitigated through a redesign of the storage architecture.
What Is a "Recovery Storm" and Why Does It Only Affect Long Sessions
Agent and code-assistant workloads differ fundamentally from traditional short Q&A. A single complete code review or multi-round tool invocation can generate context spanning tens of thousands to hundreds of thousands of tokens. When a service instance restarts due to scaling, failover, or version updates, this context must be reloaded from persistent storage.
The issue lies in the fact that KV Cache size scales proportionally with context length. For a 480B-parameter model in long-context scenarios, the KV Cache can occupy tens of gigabytes of GPU memory. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, KV Cache memory management is itself a core challenge in LLM serving, and cold restart transforms this from a compute problem into an I/O problem. The bandwidth and latency of traditional network storage, such as NFS, quickly become bottlenecks under random-read workloads of this scale, causing prolonged service unavailability—this is the origin of the "recovery storm."
How Storage Acceleration Compresses Recovery Time from Minutes to Seconds
In measured report R2, Mingxin benchmarked cold-restart performance for a 480B model (Qwen3-Coder-480B-FP8, weights approximately 450GB) in a TP8 deployment, comparing the FX100 all-flash NVMe-oF array against a local NVMe single-disk baseline. Results show that under workloads without external recomputation, the FX100 achieves a speedup of 8.6–20× (measured, report R2). Specifically, under a concurrency of 16, the recomputation baseline TTFT p50 was 149.5 seconds, while the FX100 reduced it to 11.85 seconds; throughput increased from 4.1 tok/s to 74.9 tok/s (measured, report R2).
The key to this improvement lies in the tiered storage architecture. As analyzed in Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, a KVCache-centric disaggregated compute-storage architecture is an effective design for addressing prefix-cache reuse and cross-node KV pooling in LLM serving. Mingxin's approach aligns with this philosophy but goes further: by leveraging the NVMe-oF protocol to reduce storage latency to microsecond levels, GPUs can read KV Cache directly from the storage array without first copying data to CPU memory.
| Metric | Recomputation Baseline (no external storage) | After FX100 Acceleration | Speedup | Source |
|---|---|---|---|---|
| TTFT p50 (concurrency 16) | 149.5s | 11.85s | 12.6× | Measured, R2 |
| Throughput (concurrency 16) | 4.1 tok/s | 74.9 tok/s | 18.3× | Measured, R2 |
| Throughput improvement (480B long-context cold restart) | — | — | 29–40% | Measured, R2/R3 |
| TTFT reduction (480B·TP8, three concurrency levels) | — | — | 26–32% | Measured, R2 |
Code-Assistant Scenarios: The Experience Leap from "Waiting 2 Minutes" to "Waiting 2 Seconds"
For code assistants, the experiential impact of cold restart is direct and severe. If a user must wait more than 10 seconds after a service restart to see the first token, their workflow is interrupted. In measured report R1, Mingxin's GPU-memory efficiency test for a 14B model showed that the LMCache parallel-read patch improves TTFT by 4.1× (single GPU, concurrency 16, cold disk-read scenario, Qwen2.5-32B), reducing TTFT from 37.97 seconds to 9.30 seconds, with bandwidth increasing from 0.98 GB/s to 5.23 GB/s (measured, report R1).
This data is particularly significant for real code-assistant workloads. Code completion and repository-level Q&A typically involve extensive shared prefixes—for example, multiple files within the same codebase. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention's prefix-tree reuse mechanism is designed precisely to exploit this characteristic. When the storage layer can supply these prefix data at sufficient speed, the hit rate of prefix reuse translates into tangible latency gains.
Conclusion
The "recovery storm" problem in long-context cold restart is, at its core, a storage architecture issue. Mingxin's measured data demonstrates that by tiering and offloading KV Cache to high-performance NVMe-oF storage, recovery time can be reduced by an order of magnitude without sacrificing model accuracy. Mingxin (Tianjin) Semiconductor Equipment Co., Ltd. specializes in storage acceleration and domestic compute power. Its FX product line spans the full generational range from PCIe 3.0 to PCIe 6.0, providing end-to-end support from hardware to joint validation. To verify recovery performance in your actual workloads, please contact us to arrange gate-based joint testing.
Key Q&A Summary
Q: Where does the primary performance bottleneck lie in cold restart of long-context sessions?
A: The bottleneck is the I/O involved in reloading the KV Cache. The longer the context, the larger the KV Cache; traditional network storage lacks sufficient bandwidth under random-read workloads, resulting in recovery times of up to minutes.
Q: What improvements does the Mingxin FX100 deliver under long-context cold-restart workloads?
A: Per measured reports R2/R3, inference throughput improves by 29–40% for a 480B model, and TTFT decreases by 26–32%. In scenarios without external recomputation, the speedup reaches 8.6–20×.
Q: How does this acceleration affect the real-world experience of code-assistant applications?
A: Taking a 14B model cold disk-read as an example, TTFT drops from 37.97 seconds to 9.30 seconds (measured, report R1), meaning the wait time after a service restart shifts from unacceptable to smoothly interactive.
References
- SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
- NVIDIA CMX Context Memory Storage Platform — https://www.nvidia.com/en-us/data-center/ai-storage/cmx/
- Introducing NVIDIA BlueField-4-Powered Inference Context Memory Storage Platform for the Next Frontier of AI — https://developer.nvidia.com/blog/introducing-nvidia-bluefield-4-powered-inference-context-memory-storage-platform-for-the-next-frontier-of-ai/
- SNIA — Storage Networking Industry Association — https://www.snia.org/
- NVIDIA GPUDirect Storage Documentation — https://docs.nvidia.com/gpudirect-storage/index.html
Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).
Top comments (0)