ifnodoraemon

Posted on Oct 21 Originally published at blog.llmgo.top

Inside DeepSeek's Trillion-Token Engine: DeepInfer and deepseek-recipe Architecture from Silicon to Protocol Layer

#inferencearchitecture #ai #programming

[!NOTE] Executive Summary (TL;DR)

Silicon & Kernel Foundations: DeepInfer is co-optimized for NVIDIA SM100 (Blackwell with NVLink 5) and Huawei Ascend 950 NPUs, leveraging custom kernels—FlashMLA (latent attention decoding), DeepGEMM (FP8/FP4 block-scaled GEMM), DeepEP (zero-bubble All-to-All communication), and TileKernels (TileLang domain-specific JIT compilation)—to eliminate hardware abstraction overhead.

Cluster Topology & 1:1 PD Disaggregation: Breaking past single-node time-sharing limitations, the system establishes a symmetrical 1:1 ratio between 8-GPU compact Prefill nodes (compute-bound, high FLOPs density) and 64–128 GPU Wide-EP Decode supernodes (memory-bound, ultra-wide aggregate HBM bandwidth), supplemented by an isolated asynchronous Vision Encoder pipeline.

Triad Algorithmic Acceleration Suite: Introduces DSpark Confidence-Scheduled Speculative Decoding (Confidence Head predicting adaptive verification trees, unlocking 4%–7% net throughput), SWA Bounded Replay (eliminating redundant recomputation across sliding window boundaries in long contexts), and dynamic MoE load balancing (resident core experts + on-demand redundant replicas to eliminate cross-node stragglers).

Protocol Revolution & Unit Economics: Checkpoints officially discard traditional Jinja templates in favor of the Rust-native deepseek-recipe engine for zero-copy OpenAI/Anthropic protocol translation, <think> isolation, and tool-call parsing. The production report reveals single-cluster daily throughput of 155 Trillion Tokens, average prompt lengths of 77K, a 96% KV Cache hit rate, and $38/GPU/hour (~254 RMB/GPU/h) in off-peak revenue.

Deep-Dive Cross References: Inter-node RDMA transport is detailed in P/D Disaggregation & Distributed KV Cache; speculative decoding theory is explored in EAGLE Speculative Decoding; underlying attention mechanics are derived in MLA Matrix Absorption and MoE Expert Parallelism; high-performance networking is covered in InfiniBand Production Tuning.

Introduction: Revealing the Economics of Production Inference

In modern AI research, open source often stops at releasing weights and publishing papers. The underlying engineering that enables frontier labs to sustainably serve hundreds of millions of requests under viable unit economics has remained a guarded trade secret.

In October 2026, DeepSeek published its landmark technical report, Production Deployment Practices of DeepSeek V4.1 Flash at Scale, accompanied by the open-source release of its core production stack: the high-performance inference framework DeepInfer and the protocol adaptation layer deepseek-recipe.

The disclosures revealed staggering production numbers:

Single-Cluster Daily Throughput: Operating at 155 Trillion Tokens/day;
Monetization & Efficiency: Generating $38 per GPU per hour in revenue even at off-peak rates;
Extreme Long-Context Resilience: Processing average prompts of 77K Tokens while sustaining a 96% KV Cache hit rate, maintaining Time to First Token (TTFT) at ~400ms, and clocking generation speeds between 230 and 300 tokens/s.

These achievements stem from a ground-up reconstruction of the entire inference stack—spanning semiconductor microarchitectures, collective communication fabrics, specialized operator compilers, disaggregated prefill-decode clustering, and unified protocol parsing. This guide provides an exhaustive engineering dissection across all layers.

Architectural Blueprint: The Eight-Layer DeepSeek Stack

Before exploring micro-level mechanics, review the macro-level architecture spanning DeepInfer and deepseek-recipe:

flowchart TD
    subgraph Layer_Protocol["Layer 7: Protocol & API Intermediary (deepseek-recipe)"]
        REC_In["OpenAI Chat / Anthropic Messages / Tool Call Requests"] --> REC_Core["deepseek-recipe (Rust Native Engine)"]
        REC_Core --> REC_Parse["&lt;think&gt; Reasoning Isolation / Tool Calling Extraction / Prompt Formatting"]
    end

    subgraph Layer_Scheduling["Layer 6: Algorithmic Scheduling & Acceleration"]
        SCH_DSpark["DSpark Confidence-Scheduled Speculative Decoding<br/>(Confidence Head Adaptive Tree Search)"]
        SCH_SWA["SWA Bounded Replay<br/>(Sliding Window Boundary Anchor Replay)"]
        SCH_MoE["Dynamic MoE Load Balancing<br/>(Resident Core Experts + Dynamic Redundant Replicas)"]
        SCH_KV["72h+ Tiered KV Cache Governance<br/>(96% Hit Rate / DRAM-NVMe Tiered Eviction)"]
    end

    subgraph Layer_Disaggregation["Layer 5: 1:1 Physical PD Disaggregation"]
        PD_Prefill["Prefill Pool (8-GPU Compact Nodes)<br/>Compute-Bound / High FLOPs Density / Large Context Throughput"]
        PD_Decode["Decode Supernodes (64-128 GPU Wide-EP)<br/>Memory-Bound / Massive HBM Bandwidth / Breaking Single-Node Bandwidth Walls"]
        PD_Vision["Isolated Vision Encoder Module<br/>Asynchronous Patch-Level Pre-encoding"]
        PD_Prefill -- "GPUDirect RDMA One-Sided Zero-Copy KV Push" --> PD_Decode
    end

    subgraph Layer_Kernel["Layer 4: High-Performance Kernels & Compilers (The Deep* Matrix)"]
        K_FlashMLA["FlashMLA (Latent Space Attention Decoding)"]
        K_DeepGEMM["DeepGEMM (FP8 / MXFP4 Block-Scaled Matrix Multiplication)"]
        K_DeepEP["DeepEP (Zero-Bubble All-to-All MoE Communication)"]
        K_TileKernels["TileKernels (Cross-Platform JIT via TileLang)"]
        K_DeepSelect["DeepSelect (Fused Top-K Gating Router)"]
    end

    subgraph Layer_Hardware["Layer 3: Physical Silicon & Fabric Foundations"]
        HW_NV["NVIDIA SM100 (Blackwell)<br/>NVLink 5 (1.8TB/s) / FP4 Tensor Cores"]
        HW_Ascend["Huawei Ascend 950 NPU<br/>HCCS Mesh / DaVinci Cube+Vector Units"]
        HW_Net["8x 400G/800G NDR InfiniBand & Lossless RoCEv2 (Leaf-Spine)"]
    end

    Layer_Protocol --> Layer_Scheduling
    Layer_Scheduling --> Layer_Disaggregation
    Layer_Disaggregation --> Layer_Kernel
    Layer_Kernel --> Layer_Hardware

Layer 1: Physical Silicon and Interconnect Foundations

Inference efficiency is bound by semiconductor physics. DeepInfer's design philosophy rejects generic hardware abstractions in favor of direct alignment with accelerator microarchitectures.

1. Dual-Track Silicon: NVIDIA Blackwell SM100 vs. Huawei Ascend 950

DeepSeek's production deployments officially embrace both NVIDIA and Huawei compute platforms:

Hardware Dimension	NVIDIA Blackwell (SM100)	Huawei Ascend 950 NPU
Process & Packaging	TSMC 4NP Dual-Die CoWoS-L	7nm Advanced System-in-Package (SiP)
Die-to-Die Interconnect	10 TB/s NV-HBI Fabric	Ultra-Wide On-Package Bus
Intra-Node P2P Bandwidth	NVLink 5 (1.8 TB/s Bidirectional per GPU)	HCCS (1.2 TB/s Bidirectional per NPU)
Memory Subsystem	192GB–288GB HBM3e (8.0 TB/s Bandwidth)	128GB–192GB HBM3 (3.8 TB/s Bandwidth)
Compute Engines	5th-Gen Tensor Cores (Native FP4 / FP8)	DaVinci Architecture (Cube Matrix + Vector Units)
Driver & Toolchain	CUDA 13.x / PTX / TileLang JIT	CANN 8.x / Ascend C / TileLang JIT

graph LR
    subgraph Chip_Interconnect["Physical Bandwidth Hierarchy (Orders of Magnitude Disparity)"]
        L_SRAM["On-Chip SRAM<br/>~20-30 TB/s Ultra-Fast L1/L2"] --> L_HBM["On-Package HBM3e<br/>3.8 - 8.0 TB/s Memory Bandwidth"]
        L_HBM --> L_NVLINK["Intra-Node NVLink 5 / HCCS<br/>1.2 - 1.8 TB/s Bidirectional Link"]
        L_NVLINK --> L_NIC["Inter-Node 400G/800G RDMA<br/>50 - 100 GB/s Network Fabric"]
    end

2. Physical Interconnect Gaps and Distributed Parallelism Laws

The bandwidth disparity between on-package HBM (8.0 TB/s) and inter-node networks (50–100 GB/s) represents an 80x–100x performance drop. This physical reality establishes two core distributed rules:

Tensor Parallelism (TP) Must Remain Intra-Node: TP requires two high-frequency All-Reduce collective operations per transformer layer for every single token. Extending TP across physical nodes stalls Tensor Cores in network wait states over 95% of the time.
Expert Parallelism (EP) Requires Non-Blocking Fabrics: While MoE All-to-All collective communication exchanges lower per-token data volume than TP, it is hypersensitive to network tail latency and buffer jitter. Inter-node topologies mandate 8x 400G/800G InfiniBand (NDR) or rail-optimized lossless RoCEv2 fabrics (see InfiniBand Production Tuning).

Layer 2: High-Performance Operators and Custom Compilers (The Deep* Matrix)

Generic inference runtimes decompose complex models into generic CUDA kernels, causing excessive global memory round-trips. DeepInfer addresses this through a dedicated matrix of custom kernels:

flowchart TD
    subgraph Deep_Matrix["DeepInfer Custom Kernel Suite"]
        direction TB
        F_MLA["FlashMLA<br/>Latent Space Attention"]
        D_GEMM["DeepGEMM<br/>FP8/FP4 Micro-Scaling"]
        D_EP["DeepEP<br/>Zero-Bubble All-to-All"]
        T_LANG["TileKernels<br/>TileLang Cross-Platform JIT"]
        D_SEL["DeepSelect<br/>Fused Top-K Gating"]
    end

    F_MLA & D_GEMM & D_EP & D_SEL --> T_LANG
    T_LANG --> Target_NV["NVIDIA SASS Binary"]
    T_LANG --> Target_Ascend["Ascend Machine Code"]

1. FlashMLA: Dedicated Latent Space Attention Kernel

As analyzed in MLA Matrix Absorption, Multi-Head Latent Attention compresses keys and values into a low-rank latent representation $c_t^{KV}$, reducing per-token KV Cache footprint to less than one-sixth of traditional MHA.

FlashAttention-2/3 assumes standard multi-head formats and cannot absorb low-rank projection matrices in-flight. FlashMLA provides:

On-Chip Matrix Absorption: Executes the multiplication of query projections with KV decompression weights $W_U^{KV}$ directly inside GPU shared memory (SRAM), eliminating the need to materialize uncompressed key-value matrices in HBM.
TMA Hardware Alignment: Aligns tile sizes with NVIDIA Blackwell's Tensor Memory Accelerator (TMA) for asynchronous, register-bypassing data movement.

2. DeepGEMM: Fine-Grained Block-Scaled Matrix Multiplication

Large MoE checkpoints require low-precision quantization. However, coarse per-tensor quantization destroys numerical precision in reasoning tasks.

Block-Level Scaling: DeepGEMM applies fine-grained 1x128 or 128x128 block quantization, maintaining independent FP8 E8M0 scale factors per block.
Micro-Architecture Specialization: On Blackwell, DeepGEMM compiles directly to native MXFP4 (Microscaling FP4) and FP8 Tensor Core assembly. On Ascend 950, it maps to DaVinci 3D Cube mixed-precision execution pipelines without precision conversion penalties.

3. DeepEP: Zero-Bubble Communication-Computation Overlap

In distributed MoE execution, tokens must be dispatched to experts on disparate nodes via all_to_all_v collectives.

Micro-Batch Pipelining: DeepEP chunks incoming batches into micro-batches. While micro-batch $n$ executes expert forward passes on GPU Tensor Cores, micro-batch $n+1$ asynchronously exchanges tokens across the RDMA fabric, achieving near-perfect communication hiding.
Hierarchical Transport: DeepEP routes intra-node token transfers across NVLink while routing inter-node traffic directly over GPUDirect RDMA.

4. TileKernels and TileLang: Bridging NVIDIA and Domestic Silicon

To avoid maintaining separate CUDA and Ascend C codebases, DeepInfer integrates TileLang:

Engineers define compute and memory movement patterns using Pythonic Tile-level abstractions.
The TileLang compiler handles register allocation, shared memory bank conflict resolution, and instruction scheduling, emitting native SASS binaries for NVIDIA SM100 and optimized machine code for Huawei Ascend 950.

Layer 3: Physical Cluster Topologies & Disaggregated Serving (Disambiguating Macro 1:1 vs. Micro 8 and 64)

Co-locating compute-bound prefill operations with memory-bound decode operations on the same GPU creates severe resource interference. DeepSeek's production deployments address this via physical Prefill-Decode (PD) Disaggregation.

However, the juxtaposition of "Prefill and Decode maintain an exact 1:1 GPU card count ratio" alongside "Prefill runs on 8-GPU nodes while Decode runs on 64–128 GPU supernodes" frequently puzzles systems engineers: How can 8 and 64 equate to a 1:1 ratio? How does an 8-GPU node stream its KV Cache to a 64-GPU supernode without incurring catastrophic network broadcast storms?

This section resolves the paradox across four technical dimensions: macro-cluster capacity, micro-serving topologies, queueing theory proofs, and tensor resharding mechanics.

1. Conceptual Disambiguation: Macro Cluster Allocation vs. Micro-Pod Granularity

The confusion stems from conflating macro-cluster capacity allocation with micro-serving execution boundaries:

Architectural Dimension	Prefill Pool (Prompt Ingestion)	Decode Pool (Token Generation)	Architectural Relationship
Micro-Serving Unit (Pod)	Single 8-GPU Server (P-Node)	64–128 GPU Distributed Supernode (D-Supernode)	Micro granularity disparity: 8-card standalone node vs 8–16 physical nodes grouped into a supernode
Parallel Sharding	Intra-node Tensor Parallelism (TP=8)	Intra-node TP=8 + Inter-node Wide-EP=8/16	NVLink saturates attention locally; RDMA routes MoE expert tokens across nodes
Execution Bottleneck	Compute-Bound (High FLOPs Density)	Memory-Bound (HBM Bandwidth Constrained)	8 GPUs saturate Tensor Cores; Decode requires 64 GPUs to aggregate HBM bandwidth
Macro Cluster Scale (512-GPU Example)	256 GPUs (32 Independent 8-GPU Nodes)	256 GPUs (4 Independent 64-GPU Supernodes)	Macro total card ratio is exactly 1 : 1 ($32 \times 8 = 4 \times 64 = 256$)

flowchart TB
    subgraph Macro_Cluster["Macro 512-GPU Production Cluster (Exact 1:1 Card Ratio)"]
        direction TB
        subgraph P_Cluster["Prefill Compute Pool (256 GPUs = 32 Independent 8-GPU Servers)"]
            P0["P-Node 01 (8-GPU)"]
            P1["P-Node 02 (8-GPU)"]
            P_Dots["... (32 Standalone Servers) ..."]
            P31["P-Node 32 (8-GPU)"]
        end

        subgraph D_Cluster["Decode Memory Pool (256 GPUs = 4 Distributed 64-GPU Supernodes)"]
            D0["D-Supernode 1 (64-GPU Fabric<br/>8 Interconnected Servers)"]
            D1["D-Supernode 2 (64-GPU Fabric<br/>8 Interconnected Servers)"]
            D2["D-Supernode 3 (64-GPU Fabric<br/>8 Interconnected Servers)"]
            D3["D-Supernode 4 (64-GPU Fabric<br/>8 Interconnected Servers)"]
        end
    end

    GW["Global Session Gateway"] -->|"Dispatch Text Prompt"| P_Cluster
    P_Cluster -->|"400G/800G RDMA Mesh Fabric (M:N Dynamic KV Cache Ingestion)"| D_Cluster

Key Takeaway: The architecture does not pair one 8-GPU node with one 64-GPU node. Instead, 32 independent 8-GPU Prefill nodes serve prompts in parallel, streaming completed KV Caches across an M:N RDMA fabric into 4 distributed 64-GPU Decode supernodes. The aggregate GPU count ($32 \times 8 = 256$ vs. $4 \times 64 = 256$) yields a symmetrical 1:1 capacity ratio.

2. Queueing Theory & Roofline Proof: Why Cluster Card Ratios Converge to 1:1

A deeper question emerges: Prefill requires only tens of milliseconds, whereas autoregressive Decode takes multiple seconds (a 64x duration difference). Intuition suggests long tasks should require 10x to 50x more hardware. Why does the optimal ratio mathematically converge to 1:1?

The proof lies in the mathematical product of the 96% KV Cache hit rate and per-unit concurrency capacity.

(1) Input Workload Specifications (Production Telemetry)

Average prompt length: $L_{\text{prompt}} = 77,000$ tokens;
Production KV Cache hit rate: $H = 96\%$;
Effective un-cached tokens requiring GPU forward passes: $$L_{\text{eff}} = L_{\text{prompt}} \times (1 - H) = 77,000 \times (1 - 0.96) = 3,080 \text{ tokens}$$
Average generated output length: $L_{\text{gen}} \approx 800$ tokens;
Global request arrival rate: $\lambda$ (req/s).

(2) Prefill Node Throughput Bounds

In the Prefill stage (Compute-Bound):

An 8-GPU HGX node executes DeepSeek-V4.1-Flash across intra-node NVLink (TP=8), utilizing FP8/FP4 Tensor Cores for peak arithmetic intensity.
Processing $L_{\text{eff}} = 3,080$ tokens through full attention and FFN layers requires: $$T_P \approx 50 \text{ ms} = 0.05 \text{ s}$$
Under extreme 77K context lengths, transient activations and KV allocation consume tens of gigabytes per request. Physical HBM constraints limit instantaneous concurrency per 8-GPU node to $B_P \approx 1$ (maximum 2).
Consequently, the service rate of a single 8-GPU Prefill node is: $$\mu_{P_node} = \frac{B_P}{T_P} = \frac{1 \text{ req}}{0.05 \text{ s}} = 20 \text{ req/s}$$
Across the 32-node Prefill pool (256 GPUs), aggregate prefill capacity is: $$\mu_{P_pool} = 32 \times \mu_{P_node} = 32 \times 20 = 640 \text{ req/s}$$

(3) Decode Supernode Throughput Bounds

In the Decode stage (Memory-Bound):

A 64-GPU supernode (8 physical servers) shards trillion-parameter MoE weights across all 64 GPUs, activating over $1,000 \text{ TB/s}$ of aggregate HBM bandwidth per decoding step.
With generation velocity at $V_D = 250 \text{ tokens/s}$, emitting 800 tokens requires: $$T_D = \frac{L_{\text{gen}}}{V_D} = \frac{800 \text{ tokens}}{250 \text{ tokens/s}} = 3.2 \text{ s}$$
Although $T_D (3.2\text{s})$ is 64x longer than $T_P (0.05\text{s})$, the 64-GPU supernode possesses 64 GPUs of pooled HBM. Leveraging MLA latent vector compression, each supernode sustains high-concurrency continuous batching of $B_D = 64$ (up to 128) concurrent streams simultaneously!
Thus, the service rate of a single 64-GPU Decode supernode is: $$\mu_{D_node} = \frac{B_D}{T_D} = \frac{64 \text{ req}}{3.2 \text{ s}} = 20 \text{ req/s}$$
Across 4 Decode supernodes (256 GPUs), aggregate baseline capacity is: $$\mu_{D_pool} = 4 \times \mu_{D_node} = 4 \times 20 = 80 \text{ req/s}$$ (With continuous batching saturation at $B_D = 128 \sim 256$, saturated throughput reaches $160 \sim 320 \text{ req/s}$, matching Poisson arrival tails).

(4) Steady-State Equilibrium Formulation

To prevent queue accumulation, the system balances arrival and processing capacities:

$$\mu_{P_pool} \approx \mu_{D_pool} \iff \frac{N_P \text{ (GPUs)}}{8} \times \frac{B_P}{T_P} = \frac{N_D \text{ (GPUs)}}{64} \times \frac{B_D}{T_D}$$

Substituting physical values:

$$\frac{N_P}{8} \times \frac{1}{0.05} = \frac{N_D}{64} \times \frac{64}{3.2} \implies N_P \times 2.5 = N_D \times 0.3125 \times 8 \implies N_P \approx N_D$$

[!IMPORTANT] Physical Reality: 1:1 is Dictated by the 96% Cache Hit Rate

At 0% Cache Hit Rate: $L_{\text{eff}} = 77,000$ tokens, ballooning Prefill time from 50ms to 1,200ms (a 24x increase). Maintaining a 1:1 ratio would trigger immediate prefill queue collapse, demanding 10x more Prefill cards than Decode cards.

At 96% Cache Hit Rate: The compute workload is reduced by 96%, allowing a 50% GPU allocation on Prefill to feed a 50% GPU allocation on Decode. The 1:1 ratio is not arbitrary symmetry—it is the exact mathematical equilibrium between long-cache reuse and HBM bandwidth limits.

3. Micro-Level Tensor Resharding: Direct 1-to-1 Injection from 8 GPUs to 64 GPUs

Engineers often ask: If an 8-GPU node shards tensors via TP=8, and a 64-GPU supernode spans 8 physical servers (TP8 × EP8), does transferring KV Caches require an 8-to-64 network broadcast storm (All-to-All Resharding)?

The answer is no. Inter-node RDMA transfers execute as clean 1-to-1 point-to-point pushes between matching GPU ranks.

flowchart LR
    subgraph P_Node["Prefill 8-GPU Node (TP=8)"]
        P_GPU0["GPU 0 (Head Group 0)"]
        P_GPU1["GPU 1 (Head Group 1)"]
        P_GPUDots["..."]
        P_GPU7["GPU 7 (Head Group 7)"]
    end

    subgraph D_SuperNode["Decode 64-GPU Supernode (8 Interconnected Servers, TP=8 x EP=8)"]
        subgraph Machine0["Physical Server 0 (Assigned Attention Node for Session)"]
            D_GPU0["GPU 0 (Head Group 0)"]
            D_GPU1["GPU 1 (Head Group 1)"]
            D_GPUDots["..."]
            D_GPU7["GPU 7 (Head Group 7)"]
        end
        subgraph Machine1_7["Physical Servers 1 ~ 7 (Participate only during subsequent MoE MLP steps)"]
            M_Other["GPUs 8 ~ 63<br/>(Store Zero KV Cache for this session)"]
        end
    end

    P_GPU0 -->|"RDMA 1-to-1 Direct Write (&lt; 3ms)"| D_GPU0
    P_GPU1 -->|"RDMA 1-to-1 Direct Write (&lt; 3ms)"| D_GPU1
    P_GPU7 -->|"RDMA 1-to-1 Direct Write (&lt; 3ms)"| D_GPU7

Why No 64-Way Broadcast Is Needed (MLA First Principles):

KV Cache Is Consumed Exclusively by Attention: In Transformer architectures, KV Caches are accessed only during Multi-Head Attention computation and are decoupled from subsequent MoE FFN layers.
Supernode Hierarchical Decomposition: Within a 64-GPU supernode, inter-server communication handles Expert Parallelism (EP=8), while each physical server executes intra-node Tensor Parallelism (TP=8) locally.
Session Affinity Binding: Upon completing prefill, the global router pins the session's attention evaluation to a single physical server (e.g., Server 0) within the 64-GPU supernode.
One-Sided Direct DMA Injection: GPU $i \in [0, 7]$ on the Prefill node initiates a GPUDirect RDMA Write targeting GPU $i$ on the destination server. Network transfers operate strictly as point-to-point 1-to-1 pipelines, transferring megabytes of compressed MLA latent vectors in under 3ms with zero broadcast overhead. Subsequent MoE token dispatch occurs across the local network during the MLP phase and involves no KV Cache movement.

4. Isolated Vision Encoder Pipeline

For multimodal workloads, DeepInfer extracts image processing into a standalone tier:

Incoming images are pre-processed by dedicated Vision Encoder nodes into patch-level embeddings.
Visual tokens merge with text tokens at the gateway before ingestion by the Prefill pool.
This prevents lightweight convolutional or ViT kernels from disrupting LLM execution graphs.

Layer 4: Triad Algorithmic Acceleration Suite

Beyond hardware and kernel co-design, DeepInfer integrates three algorithmic breakthroughs: DSpark Speculative Decoding, SWA Bounded Replay, and Dynamic MoE Balancing.

1. DSpark Confidence-Scheduled Speculative Decoding

Standard speculative decoding pairs a lightweight draft model with a primary target model, verifying $K$ drafted tokens in a single target forward pass (see EAGLE Speculative Decoding).

However, conventional schemes suffer from the Fixed-Stride Penalty:
With a fixed stride ($K=5$), if the first drafted token is rejected, all four remaining tokens are discarded. During deep reasoning phases or complex syntax emission, rigid strides introduce significant waste.

flowchart TD
    subgraph Traditional["Traditional Speculative Decoding (Fixed K=5)"]
        T1["Token 1 (High Conf)"] --> T2["Token 2 (High Conf)"]
        T2 --> T3["Token 3 (Low Conf - Rejected)"]
        T3 --> T4["Token 4 (Wasted Math)"]
        T4 --> T5["Token 5 (Wasted Math)"]
        T_Val["Verification: Accepted 2, Discarded 3 (60% Wasted Compute)"]
    end

    subgraph DSpark["DeepInfer DSpark (Adaptive Verification Tree)"]
        D_Head["Draft Model + Confidence Head"]
        D_Head --> D1["Token 1 (Confidence 0.95 -> Expand)"]
        D1 --> D2["Token 2 (Confidence 0.91 -> Expand)"]
        D2 --> D3{"Token 3 Confidence 0.42 < Threshold ?"}
        D3 -- "Yes: Immediate Dynamic Pruning" --> D_Commit["Submit only 2 tokens (Zero Wasted Compute)"]
        D3 -- "No" --> D_Next["Continue Expansion"]
    end

DSpark Confidence Head Formulation

DSpark attaches a lightweight regression head to the draft model's final layer, predicting target acceptance probability:

$$\hat{P}{\text{accept}}(x{t+i}) = \sigma \left( W_c \cdot h_{t+i} + b_c \right)$$

where $h_{t+i}$ denotes the draft model's hidden representation at step $t+i$.

The scheduler dynamically bounds verification tree depth:

$$\text{Depth}(T) = \max \left{ m \in [1, K_{\max}] \;\middle|\; \prod_{j=1}^{m} \hat{P}{\text{accept}}(x{t+j}) \ge \tau_{\text{threshold}} \right}$$

Structured Syntax (JSON, Code Boilerplate, Prose): When cumulative confidence remains high, DSpark expands validation trees up to 8–12 tokens.
Complex Logic & Symbolic Math: As confidence dips, the verification tree shrinks to 1–2 tokens, preventing rollback overhead.
Production Impact: DSpark provides an additional 4% to 7% net throughput gain over baseline generation.

2. SWA Bounded Replay (Sliding Window Attention Boundary Anchoring)

DeepSeek-V4.1-Flash employs Sliding Window Attention (SWA) to sustain efficiency across long contexts. In conversational turns, SWA typically encounters Boundary Recomputation Penalties:

When the context window slides past length $W$, tokens outside the window lose cross-attention visibility. If historical context must be revisited or verified, the system is conventionally forced to re-execute prefill across earlier tokens.

SWA Bounded Replay Mechanics:

Boundary Anchoring: At fixed window intervals $W$, the kernel snapshots an Attention Sink State Fingerprint in SRAM.
Localized Replay: During context shifting, the engine recomputes a bounded slice (< 128 tokens) from the nearest anchor rather than reprocessing the entire 77K context.
Efficiency: SWA Bounded Replay eliminates over 85% of redundant prefill computation in multi-turn dialogues.

3. Dynamic MoE Expert Load Balancing

In multi-node expert parallelism, unmitigated routing skew creates severe Straggler Bottlenecks: if all tokens in a batch route to a small subset of popular experts, those GPU ranks saturate while others idle, blocking the entire cluster at the subsequent All-to-All barrier.

DeepInfer employs a three-part balancing strategy:

Resident Core Experts: Offline traffic analysis identifies top-frequency experts, which are replicated across every physical node. Requests hitting these experts execute locally, generating zero cross-node network traffic.
Dynamic Redundant Replicas: Telemetry monitors token arrival rates per expert in real time. When an uncommon expert experiences sudden traffic spikes, the cluster broadcasts weights via RDMA to instantiate temporary replicas on underutilized nodes within milliseconds.
Capacity Factor Truncation & Reranking: Enforces strict token processing caps per expert with residual bypass connections, bounding inter-GPU rank computation variance to within 5%.

4. Tiered 72h+ KV Cache Management: Sustaining a 96% Hit Rate

The production report's most striking metric is an online 96% KV Cache hit rate across 77K-token average inputs. This efficiency relies on a structured three-tier storage hierarchy:

graph TB
    subgraph Storage_Hierarchy["DeepInfer Tiered KV Cache Storage Architecture"]
        Tier1["Tier 1: GPU HBM Memory Pool<br/>Capacity: Tens of TBs | Latency: < 1 µs<br/>Active working set for high-concurrency sessions"]
        Tier2["Tier 2: Host DRAM Shared Memory Pool<br/>Capacity: Hundreds of TBs | Latency: ~10-20 µs<br/>Warm cache for sessions idle within 1-6 hours"]
        Tier3["Tier 3: Distributed NVMe-oF Local Storage<br/>Capacity: Multiple PBs | Latency: ~100-300 µs<br/>Cold archive for 7-day Chatbot / 3-day Agent TTLs"]
    end

    Tier1 <== "GPUDirect Swapping" ==> Tier2
    Tier2 <== "Asynchronous Write-Through" ==> Tier3

Policy-Based TTL Enforcement:
- Conversational Traffic (Chatbot): Configured with a 7-day TTL, ensuring multi-turn user follow-ups hit cache near-instantaneously.
- Agent Workflows: Configured with a 3-day TTL, preserving long tool execution traces and environment observations.
Radix Tree Prefix Indexing: Incoming token streams are matched against a global prefix radix tree. Requests route directly to the Decode node hosting the matched KV Cache blocks, executing prefix-aware placement.

Layer 5: Protocol and Gateway Layer (`deepseek-recipe` Deep Dive)

Production deployments require reliable API serialization. Alongside DeepInfer, DeepSeek open-sourced deepseek-recipe.

1. Why Jinja Templates Were Deprecated

Traditional Hugging Face repositories rely on chat_template.jinja rendered via Python. In massive production clusters, Jinja presents three primary bottlenecks:

CPU Parsing Overhead: Interpreting complex Jinja templates across hundreds of thousands of concurrent requests saturates gateway CPU cores.
State Machine Fragility: Reasoning models produce interleaved reasoning tags (<think>...</think>), structured function calling parameters, and multimodal references. String-based Jinja rendering frequently mishandles token boundaries and escaping.
Protocol Divergence: Bridging external OpenAI (Chat Completions / Responses) or Anthropic (Messages) schemas into Jinja templates requires fragile boilerplate.

Consequently, DeepSeek-V4.1-Flash removed Jinja templates, delegating all prompt preparation to deepseek-recipe.

2. Rust Native Core with Python Bindings

deepseek-recipe is authored in Rust with PyO3 bindings:

flowchart LR
    A["API Request (OpenAI / Anthropic / Custom)"] --> B["Python Gateway (FastAPI / vLLM Server)"]
    B --> C["deepseek-recipe Core (Rust Native Shared Library)"]
    C --> D["Zero-Copy Conversation AST"]
    D --> E["Token ID Serialization & &lt;think&gt; State Machine"]
    E --> F["Native Token Buffers Dispatched to Engine"]

3. Implementation Example: Protocol Bridging and Reasoning Extraction

Installable via pip install deepseek-recipe, the following example demonstrates production API bridging:

"""
production_gateway_recipe.py
Production gateway integration using deepseek-recipe for protocol unification,
reasoning trace separation, and function calling extraction.
"""

from typing import Dict, Any, List
import deepseek_recipe as ds_recipe

def handle_chat_completion_request(payload: Dict[str, Any]) -> ds_recipe.RenderedInput:
    """
    Transforms standard OpenAI API payloads into native token streams via deepseek-recipe.
    """
    # 1. Initialize structured conversation instance
    conversation = ds_recipe.Conversation()

    # 2. Configure thinking and reasoning behaviors
    enable_thinking = payload.get("enable_thinking", True)
    conversation.set_reasoning_mode(enable_thinking)

    # 3. Bind client tool declarations
    if "tools" in payload:
        for tool in payload["tools"]:
            conversation.register_tool(
                name=tool["function"]["name"],
                description=tool["function"].get("description", ""),
                parameters=tool["function"].get("parameters", {})
            )

    # 4. Ingest multi-turn message history
    for msg in payload["messages"]:
        role = msg["role"]
        content = msg["content"]

        if role == "system":
            conversation.add_system_message(content)
        elif role == "user":
            conversation.add_user_message(content)
        elif role == "assistant":
            # Preserve historical reasoning chains and tool calls cleanly
            reasoning_content = msg.get("reasoning_content", None)
            tool_calls = msg.get("tool_calls", None)
            conversation.add_assistant_message(
                content=content,
                reasoning_content=reasoning_content,
                tool_calls=tool_calls
            )
        elif role == "tool":
            conversation.add_tool_result(
                tool_call_id=msg["tool_call_id"],
                result=content
            )

    # 5. Compile to native token arrays and attention masks via Rust engine
    rendered_input = conversation.render_for_engine(
        tokenizer_model="deepseek-ai/DeepSeek-V4.1-Flash",
        add_generation_prompt=True
    )

    return rendered_input


def parse_engine_stream_output(raw_token_stream: List[int]) -> Dict[str, Any]:
    """
    Streams and demuxes raw token streams into <think> traces and content payloads.
    """
    parser = ds_recipe.StreamOutputParser(tokenizer_model="deepseek-ai/DeepSeek-V4.1-Flash")

    parsed_result = {
        "reasoning_chunk": "",
        "content_chunk": "",
        "detected_tool_calls": []
    }

    for token_id in raw_token_stream:
        event = parser.step(token_id)

        if event.is_reasoning():
            # Encapsulated within <think> blocks
            parsed_result["reasoning_chunk"] += event.text()
        elif event.is_content():
            # Primary user-facing generation
            parsed_result["content_chunk"] += event.text()
        elif event.is_tool_call_completed():
            # Fully parsed structured function invocation
            parsed_result["detected_tool_calls"].append(event.get_tool_call())

    return parsed_result

Robust Syntax Isolation: The Rust parser guarantees strict syntactic tokenization, ensuring synthetic prompt injections (e.g., forged <|fim_begin|> or <think> tags) are safely escaped.

Layer 6: Cluster Financials and Production Unit Economics

How does DeepSeek maintain profitability while offering competitive API pricing, achieving $38/GPU/hour in revenue? A formal financial model clarifies the dynamics.

1. Revenue and Throughput Modeling

Consider a baseline production deployment of 512 accelerators (256 Prefill GPUs + 256 Decode GPUs):

Revenue Rate per GPU: $\$38 \text{ / (GPU} \cdot \text{Hour)}$
Aggregate Daily Revenue: $$\text{Daily Cluster Gross} = 512 \text{ GPUs} \times 24 \text{ Hours} \times \$38 \approx \$466,944 \text{ / Day}$$
Annualized Gross Revenue: $$\text{Annualized Gross} = \$466,944 \times 365 \approx \$170.4 \text{ Million / Year}$$

2. Comprehensive Total Cost of Ownership (TCO) Breakdown

Operating 512 enterprise-grade GPUs (e.g., 64 8-GPU Blackwell / H100 nodes) incurs the following monthly cost model:

Cost Category	Monthly Allocation (USD)	Modeling Methodology
Server Hardware Amortization (CapEx)	$550,000 / month	64 nodes amortized over 36 months straight-line + enterprise support
Network Fabric & Optics Amortization	$65,000 / month	800G NDR InfiniBand leaf-spine switches and active optical cabling
Datacenter Power & Cooling (OpEx)	$130,000 / month	High-density liquid-cooled suites, 12kW–15kW per rack, PUE = 1.25
High-Bandwidth Transit & BGP Ports	$45,000 / month	Dedicated redundant transit ports supporting high egress volume
AI Infrastructure Engineering Allocation	$90,000 / month	Dedicated cluster operations and systems engineering personnel
Total Monthly Operating Cost	~$880,000 / month	Yields an average daily run-rate of $29,333 / day

3. Financial Margin Analysis: The Role of Cache Hit Rates

Comparing gross revenues against operating costs:

Daily Gross Revenue: $466,944 / day
Daily Operating Run-Rate: $29,333 / day
Daily Operating Margin: $437,611 / day (93.7% Gross Margin)

pie title Daily Financial Composition ($466,944 / Day)
    "Net Operating Margin" : 437611
    "Server Amortization (CapEx)" : 18333
    "Datacenter Power & Bandwidth" : 5833
    "Network Optics & Fabric" : 2167
    "AI Systems Personnel" : 3000

The True Driver: 96% KV Cache Hit Rate

While large language model serving is often treated as compute-bound selling of FLOPs, DeepSeek's operational model primarily monetizes memory bandwidth and cache reuse:

Without tiered 72-hour KV caching, a 77K prompt requires full quadratic forward passes across Tensor Cores, incurring massive compute cost.
With 96% of input tokens hitting cache, the prefill compute burden drops by 96%. The cluster reads stored states and passes execution directly to high-throughput Wide-EP Decode supernodes.
Takeaway: MLA's ultra-compact latent state combined with a 96% KV hit rate decouples serving revenue from raw compute expenses.

Layer 7: Cross-Framework Architectural Matrix

How does DeepInfer contrast with other prominent inference engines in production?

Architectural Dimension	DeepSeek DeepInfer	vLLM (v1 Engine)	SGLang (FlashInfer)	TensorRT-LLM
Primary Design Target	High-Scale Profitable Production	Developer accessibility and universal serving	Complex multi-turn Agent & long contexts	Peak single-node NVIDIA performance
P/D Disaggregation	Native Physical Disaggregation (1:1 Ratio)	Experimental community support	Mature distributed PD implementations	Supported via complex manual manifests
MoE Parallelism	Native Wide-EP (64–128 GPU Supernodes)	Intra-node focus, multi-node via Ray	Cross-node EP integrated with DeepEP	Static TP/EP groupings
Speculative Decoding	DSpark (Confidence Adaptive Tree Search)	Standard Speculative / EAGLE	Native EAGLE-2 Tree Decoding	Fixed-stride speculative decoding
Long-Context Cache	SWA Bounded Replay + 72h Tiering	PagedAttention blocks	RadixAttention prefix trees	Basic KV paging
Silicon Ecosystem	SM100 (Blackwell) + Huawei Ascend 950	Broad community coverage (NVIDIA-first)	Primarily NVIDIA, expanding ROCm	NVIDIA Platforms Only
API Protocol Layer	Custom deepseek-recipe (Rust Native)	Python Jinja template evaluation	Python abstraction layers	C++ native runtime / Triton

Layer 8: Production Troubleshooting and Failure Mode Playbooks

Based on multi-node supernode testing with DeepInfer, four critical failure modes and their mitigations are highlighted below:

1. Wide-EP Inter-Switch Congestion and Hash Polarization

Symptom: Scaling Decode supernodes to 128 GPUs across multiple leaf switches triggers a 70% collapse in All-to-All throughput, with escalating packet drop counters (rx_prio0_drop).
Root Cause: Dense All-to-All collective exchanges cause Equal-Cost Multi-Path (ECMP) routing to suffer from Hash Polarization, directing multiple elephant flows onto identical spine uplinks and causing localized buffer exhaustion.
Remediation: Enable Adaptive Routing (AR) on InfiniBand fabrics to dynamically steer packets around congested links. For RoCEv2 environments, enforce strict DSCP traffic class mappings and hardware-level PFC deadlock detection (PFC_DEADLOCK_DETECTION_EN=1).

2. DSpark Confidence Score Drift on Extended Reasoning Traces

Symptom: During complex, multi-step code synthesis, speculative acceleration collapses from 1.6x down to 0.9x (slower than unspeculated baselines).
Root Cause: On deep reasoning chains, hidden representations deviate semantically from the draft model's training distribution. The Confidence Head overestimates acceptance probabilities, issuing low-quality draft sequences that trigger costly rollbacks.
Remediation: Configure conservative dynamic rollback safety gates:

  --dspark-confidence-threshold 0.85 \
  --dspark-max-draft-tokens 4 \
  --dspark-abort-on-rollback-rate 0.30

If rollbacks exceed 30% over three consecutive execution steps, the scheduler temporarily falls back to pure autoregressive generation.

3. Asynchronous RDMA KV Ingestion Race Conditions

Symptom: Prefill nodes pushing KV Cache blocks via RDMA into Decode GPU memory occasionally result in corrupted generation or CUDA illegal memory access crashes.
Root Cause: RDMA Write operations transfer data directly into GPU memory without host CPU intervention. If the target node's attention kernel attempts to read a block before transmission completes, it reads partial state.
Remediation: Implement a Two-Phase Memory Gate Protocol: upon completing data transfer, the sending NIC writes an 8-byte completion flag to a designated synchronization mailbox. The target GPU kernel polls this memory flag via a lightweight spinlock before unblocking attention computation.

4. Cache Eviction Storms under Tiered Storage Saturation

Symptom: Periodic latency spikes where TTFT jumps from 400ms to 5 seconds across active sessions.
Root Cause: When Tier 1 (GPU HBM) and Tier 2 (Host DRAM) reach 95% capacity thresholds simultaneously, synchronous eviction triggers thousands of session offloads to local NVMe-oF arrays, exhausting random write IOPS and stalling the PCIe bus.
Remediation: Transition to Low-Watermark Gradual Eviction: lower the eviction threshold to 80% and run a background daemon utilizing token-bucket rate limiting to stage cold blocks to NVMe disks smoothly without I/O bursts.

Layer 9: End-to-End Production Implementation Blueprint: From Bare Metal to Cluster Acceptance

Architecture and algorithmic theory must translate into actionable commands and precise physical datacenter cabling. This section provides an end-to-end implementation manual guiding infrastructure teams from hardware procurement and lossless fabric configuration to cluster orchestration, prefix-affinity gateway deployment, and SLA acceptance verification.

flowchart LR
    P1["Phase 1: Hardware BOM<br/>& Lossless Cabling"] --> P2["Phase 2: Host OS, OFED Drivers<br/>& NUMA Initialization"]
    P2 --> P3["Phase 3: DeepInfer Kernel Build<br/>& Container Image Packaging"]
    P3 --> P4["Phase 4: Prefill Pool<br/>& Decode Supernode Launch"]
    P4 --> P5["Phase 5: deepseek-recipe Gateway<br/>& Prefix Affinity Routing"]
    P5 --> P6["Phase 6: End-to-End Benchmarking<br/>& SLA Monitoring Dashboards"]

Phase 1: Hardware Procurement, Rack Planning & Lossless Cabling

To host DeepSeek-V4.1-Flash across a standard production validation footprint, the recommended baseline cluster comprises 96 GPUs (12 8-GPU physical servers), partitioned into 4 Prefill nodes and 1 64-GPU Decode supernode:

1. Hardware Bill of Materials (BOM)

Compute Nodes (12x HGX H100/B200 or Huawei Ascend 950 8-way Servers):
- CPU: Dual-socket AMD EPYC 9654 (192 physical cores) or Intel Xeon Platinum 8480+;
- Memory: 2TB DDR5-4800 ECC Registered RAM (ensuring sufficient host shared memory);
- Local Storage: 4x 7.68TB NVMe PCIe 5.0 SSDs (configured in RAID 0 for Tier 3 KV swapping);
- GPUs: 8x NVIDIA H100 SXM5 80GB (fully meshed intra-node via NVLink at 900 GB/s);
- NICs: 8x NVIDIA ConnectX-7 400Gbps OSFP single-port HCAs (dedicated to inter-GPU RDMA traffic) + 2x 100Gbps NICs (management/public traffic).
Network Fabric (Leaf-Spine Lossless Topology):
- Leaf Switches: 2x NVIDIA Quantum-2 QM9700 64-port 400G NDR InfiniBand switches (or 400G/800G RoCEv2 equivalents);
- Spine Switches: 2x QM9700 spine switches, establishing a non-blocking 1:1 dual-uplink fabric.

2. Network Cabling & Switch Configuration

Ensure strict PCIe local affinity between GPUs and NICs, eliminating cross-socket traffic:

GPU 0 connects to the PCIe switch hosting NIC 0; GPU 1 connects to NIC 1, and so forth.
Lossless Fabric Switch Configuration (RoCEv2 Parameters):

  # Enable Priority Flow Control (PFC) on priority 3 and ECN marking on access ports
  interface Ethernet1/1-1/64
    flowcontrol priority 3 pfc
    congestion-control ecn minimum-threshold 150000 maximum-threshold 1500000 mark-probability 10

Phase 2: Host Operating System, Driver & NUMA Optimization

Apply uniform kernel and fabric tuning across all 12 physical nodes.

1. Kernel Network and Memory Configurations (`/etc/sysctl.d/99-ai-infra.conf`)

# Disable transparent hugepage compaction to protect pinned GPU memory allocations
vm.transparent_hugepage = never
# Expand socket read/write buffers and network backlog
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 67108864
net.core.wmem_default = 67108864
net.core.optmem_max = 67108864
net.core.netdev_max_backlog = 250000
# Enable ECN congestion notification
net.ipv4.tcp_ecn = 1
# Expand memory map limits
vm.max_map_count = 2621440

Apply via: sudo sysctl --system

2. Hardware Topology & GPUDirect RDMA Verification

Execute self-checks to ensure all 8 adapters operate at full bandwidth:

# 1. Verify InfiniBand/RoCEv2 adapter state and link speeds (400 Gbps Active)
ibstat | grep -E "(CA '|State:|Physical state:|Rate:)"

# 2. Verify GPU topology (All links must indicate NVLink NV18/NV9 interconnects)
nvidia-smi topo -m

# 3. Lock GPUDirect RDMA levels and export topology cache
export NCCL_NET_GDR_LEVEL=5
export NCCL_TOPO_DUMP_FILE=/etc/ai-infra/nccl_topo.xml
export NCCL_SET_STACK_SIZE=1

Phase 3: DeepInfer Kernel Compilation & Container Image Packaging

To avoid dependency conflicts in production, encapsulate custom kernels (FlashMLA, DeepGEMM, DeepEP, TileLang) in a multi-stage Docker build:

# syntax=docker/dockerfile:1.4
FROM nvidia/cuda:13.0.0-devel-ubuntu24.04 AS builder

WORKDIR /build

# Install compilation toolchains and Rust
RUN apt-get update && apt-get install -y --no-install-recommends \
    git cmake ninja-build python3-dev python3-pip curl pkg-config libssl-dev && \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

# 1. Build DeepInfer custom kernels
RUN git clone --recursive https://github.com/deepseek-ai/DeepInfer.git /build/DeepInfer && \
    cd /build/DeepInfer && \
    pip install --no-cache-dir -r requirements.txt && \
    python3 setup.py bdist_wheel

# 2. Build deepseek-recipe protocol parser
RUN git clone https://github.com/deepseek-ai/deepseek-recipe.git /build/deepseek-recipe && \
    cd /build/deepseek-recipe && \
    cargo build --release && \
    python3 -m pip install maturin && maturin build --release

# Production runtime image
FROM nvidia/cuda:13.0.0-runtime-ubuntu24.04
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip numactl infiniband-diags ibverbs-utils libibverbs1 && \
    rm -rf /var/lib/apt/lists/*

COPY --from=builder /build/DeepInfer/dist/*.whl /tmp/
COPY --from=builder /build/deepseek-recipe/target/wheels/*.whl /tmp/
RUN pip install --no-cache-dir /tmp/*.whl && rm -rf /tmp/*.whl

ENV PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:128"
ENV NCCL_NET_GDR_LEVEL=5
ENTRYPOINT ["python3", "-m", "deepinfer.entrypoints.server"]

Phase 4: Cluster Orchestration and Distributed Launch Scripts

Prefill nodes and Decode supernodes initialize with distinct topology roles.

1. Prefill Node Pool Launch Script (`launch_prefill_pool.sh`)

Execute across 4 Prefill nodes (IPs: 192.168.10.101 ~ 104):

#!/usr/bin/env bash
# Prefill node: Intra-node TP=8, RDMA one-sided write enabled
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export HOST_IP=192.168.10.101

numactl --interleave=all python3 -m deepinfer.entrypoints.server \
    --model-path /models/DeepSeek-V4.1-Flash \
    --role prefill \
    --host 0.0.0.0 \
    --port 8001 \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 1 \
    --max-model-len 131072 \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.90 \
    --kv-transfer-protocol rdma \
    --rdma-device-names mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

2. Decode 64-GPU Supernode Launch Script (`launch_decode_supernode.sh`)

Execute across 8 Decode physical servers (IPs: 192.168.10.111 ~ 118) with TP=8, EP=8:

#!/usr/bin/env bash
# On Decode Head Node (192.168.10.111) and 7 companion workers
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_GID_INDEX=3
export NCCL_ALGO=Tree

python3 -m deepinfer.entrypoints.server \
    --model-path /models/DeepSeek-V4.1-Flash \
    --role decode \
    --host 0.0.0.0 \
    --port 8002 \
    --cluster-nodes 8 \
    --tensor-parallel-size 8 \
    --expert-parallel-size 8 \
    --max-model-len 131072 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.92 \
    --dspark-enable \
    --dspark-confidence-threshold 0.85 \
    --dspark-max-draft-tokens 6 \
    --swa-bounded-replay \
    --kv-cache-ttl-hours 72 \
    --tiered-kv-storage-path /mnt/nvme_tier3_kv/

3. Kubernetes / KubeRay Manifest Configuration

In Kubernetes environments, mount /dev/shm backed by memory alongside RDMA device limits:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: deepseek-decode-supernode-64gpu
spec:
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
    template:
      spec:
        containers:
        - name: ray-head
          image: internal-registry.corp/deepinfer:v4.1
          resources:
            limits:
              nvidia.com/gpu: "8"
              rdma/hca: "8"
              memory: 256Gi
          volumeMounts:
          - mountPath: /dev/shm
            name: dshm
        volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 128Gi

Phase 5: Production Gateway, Protocol Bridging & Prefix Affinity Routing

To achieve the 96% KV Cache hit rate, the gateway enforces prefix-aware consistent hashing:

"""
production_affinity_gateway.py
Production gateway: prefix-aware consistent hashing, deepseek-recipe protocol translation,
and streaming reasoning demuxing.
"""

import hashlib
from typing import List, Dict, Any
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx
import deepseek_recipe as ds_recipe

app = FastAPI(title="DeepSeek Production Gateway")

# 4 Prefill nodes and 1 64-GPU Decode supernode endpoint
PREFILL_NODES = [f"http://192.168.10.{i}:8001" for i in range(101, 105)]
DECODE_SUPERNODE = "http://192.168.10.111:8002"

def compute_prefix_hash(messages: List[Dict[str, str]]) -> str:
    """
    Computes a 64-bit prefix hash over System Prompt and initial turns for affinity routing.
    """
    prefix_str = ""
    for msg in messages:
        if msg["role"] in ["system", "user"]:
            prefix_str += msg["content"]
            if len(prefix_str) > 2048:
                break
    return hashlib.sha256(prefix_str.encode("utf-8")).hexdigest()

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    payload = await request.json()
    messages = payload.get("messages", [])

    # 1. Consistent hash routing for prefill reuse
    p_hash = compute_prefix_hash(messages)
    selected_p_node = PREFILL_NODES[int(p_hash[:8], 16) % len(PREFILL_NODES)]

    # 2. Tokenize and format using deepseek-recipe Rust core
    conversation = ds_recipe.Conversation()
    conversation.set_reasoning_mode(payload.get("enable_thinking", True))

    for msg in messages:
        if msg["role"] == "system":
            conversation.add_system_message(msg["content"])
        elif msg["role"] == "user":
            conversation.add_user_message(msg["content"])
        elif msg["role"] == "assistant":
            conversation.add_assistant_message(
                content=msg["content"],
                reasoning_content=msg.get("reasoning_content", None)
            )

    rendered_input = conversation.render_for_engine(
        tokenizer_model="deepseek-ai/DeepSeek-V4.1-Flash",
        add_generation_prompt=True
    )

    # 3. Asynchronously trigger Prefill computation (auto-pushes KV Cache to Decode node via RDMA)
    async with httpx.AsyncClient(timeout=60.0) as client:
        p_resp = await client.post(
            f"{selected_p_node}/execute_prefill",
            json={
                "tokens": rendered_input.token_ids,
                "target_decode_node": DECODE_SUPERNODE,
                "session_hash": p_hash
            }
        )
        prefill_meta = p_resp.json()

    # 4. Stream response from Decode supernode with dynamic <think> demuxing
    async def token_generator():
        parser = ds_recipe.StreamOutputParser(tokenizer_model="deepseek-ai/DeepSeek-V4.1-Flash")

        async with httpx.AsyncClient(timeout=300.0) as client:
            async with client.stream(
                "POST",
                f"{DECODE_SUPERNODE}/stream_decode",
                json={"kv_session_id": prefill_meta["kv_session_id"]}
            ) as stream:
                async for chunk in stream.aiter_bytes():
                    token_ids = [int(x) for x in chunk.decode().split(",") if x]
                    for tid in token_ids:
                        ev = parser.step(tid)
                        if ev.is_reasoning():
                            yield f"data: {{\"choices\": [{{\"delta\": {{\"reasoning_content\": \"{ev.text()}\"}}}}]}}\n\n"
                        elif ev.is_content():
                            yield f"data: {{\"choices\": [{{\"delta\": {{\"content\": \"{ev.text()}\"}}}}]}}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(token_generator(), media_type="text/event-stream")

Phase 6: Cluster Acceptance Benchmarks & Observability

Before serving live user traffic, clusters must pass rigorous validation under simulated concurrency.

1. Automated Benchmark Execution

Run standard ShareGPT conversational benchmarks with synthetic 77K context lengths:

python3 -m deepinfer.benchmarks.benchmark_serving \
    --backend deepinfer \
    --host 127.0.0.1 \
    --port 8000 \
    --dataset-name sharegpt \
    --num-prompts 2000 \
    --max-concurrency 256 \
    --request-rate 50

2. Acceptance Verification Checklist

Sign-off requires satisfying all operational thresholds:

[x] KV Cache Hit Rate: Must stabilize at $\ge 90\%$ after 30 minutes of continuous load;
[x] Time to First Token (TTFT P95): Bounded at $\le 450 \text{ ms}$ under cache-hit conditions;
[x] Decode Throughput (TPOT): Average output velocity maintains $\ge 230 \text{ tokens/s}$;
[x] Zero Fabric Drops: Interface counters (ethtool -S <interface> | grep rx_prio3_drop) remain strictly 0.

Frequently Asked Questions (FAQ)

Q1: Why did DeepSeek build DeepInfer from scratch instead of contributing patches to vLLM or SGLang?

General-purpose frameworks like vLLM and SGLang are designed to support hundreds of model architectures and diverse consumer and enterprise deployment scenarios. This broad compatibility necessitates generalized abstraction layers and complex multi-device runtime wrappers.
DeepInfer was engineered with a focused objective: maximizing serving efficiency specifically for DeepSeek architectures (MLA, CSA+HCA, MoE), massive parameters, Wide-EP topologies, and target Blackwell/Ascend silicon. Removing extraneous abstractions yielded a lightweight codebase with zero memory overhead and direct kernel access, forming the operational foundation of its $38/GPU/h unit economics.

Q2: Why is deepseek-recipe implemented in Rust, and how can teams integrate it into existing API gateways?

Jinja templates are interpreted dynamically as raw text. When handling multi-turn tool calling, complex system prompts, and strict <think> token isolation, Jinja incurs high CPU overhead and risks token escaping errors. Rust provides zero-cost abstractions, deterministic memory safety without a garbage collector, and high-throughput AST manipulation.
For integration, the library is distributed via PyPI (pip install deepseek-recipe). Python gateways (e.g., built on FastAPI, Litestar, or LiteLLM) can replace prompt formatting modules by routing requests into deepseek_recipe.Conversation, achieving parity with official tokenization and reasoning separation without altering underlying network architectures.

Q3: How do Huawei Ascend 950 and NVIDIA Blackwell compare when running DeepInfer?

Compute Subsystem: NVIDIA Blackwell excels in raw dense FP4/FP8 GEMM throughput due to its 5th-Gen Tensor Cores and TMA subsystem. Huawei Ascend 950 provides strong matrix throughput via its DaVinci 3D Cube cores and Vector units, backed by competitive HCCS intra-node interconnects.
Software Alignment: DeepInfer uses TileLang to abstract hardware differences. TileLang compiles to optimized SASS for NVIDIA platforms and corresponding native assembly for Ascend platforms, optimizing for Ascend's L0/L1 on-chip memory buffers. In memory-bound MoE decode phases utilizing Wide-EP sharding, Ascend 950 clusters demonstrate token generation throughput that closely tracks tier-one accelerators.

Q4: Can enterprise private deployments replicate the 96% KV Cache hit rate?

Yes, provided traffic patterns and gateway routing are architected appropriately.
The 96% hit rate reported in production stems from two major workload characteristics:

Agent Workflows: Multi-turn agents repeatedly submit stable system prompts, tool specifications, and environmental state histories, forming large, reusable shared prefixes.
Enterprise Knowledge Bases (RAG): Multi-turn Q&A against static organizational documentation reuses identical reference contexts across queries. To achieve comparable metrics, teams must implement: (1) Radix Tree prefix caching with extended TTL windows (3–7 days), and (2) Session-affinity routing at the gateway layer, ensuring requests bearing identical prefix fingerprints route consistently to the same Decode worker nodes to avoid cache fragmentation.

Originally published at LLM Systems Architecture on blog.llmgo.top.