On October 20, 2026, DeepSeek announced DeepInfer, the inference framework it says serves DeepSeek-V4.1-Flash in production, and published the code under Apache 2.0. Eighteen months earlier, the same company had explained in writing why it would not release its engine. As The New Stack quotes that post, the software was “impractical for public deployment without significant modifications.”
Both statements can be true at once, and the space between them is where this release gets interesting. DeepInfer is a real production engine: kernels, a scheduler, a sampler, and multi-chip parallelism. It is also an offline inference release that leaves out the KV cache system, prefill and decode disaggregation, and load balancing across replicas. Those missing layers are the ones that bind most tightly to a specific cluster.
This post reads the repository, its examples, and the first round of reporting, and it marks which claims are DeepSeek’s and which readings are mine.
Quick take: DeepInfer is most useful as a reference implementation and as a day-one path to V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950. It is not a drop-in serving stack, DeepSeek’s speed figures have no independent verification yet, and the determinism features work on NVIDIA only.
What DeepSeek said in 2025
DeepSeek’s earlier open-source push went component by component. In late February 2025, its Open Source Week shipped one building block a day: FlashMLA, an MLA decoding kernel for Hopper GPUs; DeepEP, an expert-parallel communication library; and DeepGEMM, an FP8 GEMM library. The week also covered the EPLB expert-parallel load balancer, the DualPipe training schedule, and the 3FS file system.
Inference code was the thin spot. For V3, the model card offered the DeepSeek-Infer demo, described as a “simple and lightweight demo” for FP8 and BF16, alongside community engines such as SGLang and LMDeploy for serving it.
In April 2025 the same index gained a post on open-sourcing the inference engine. According to The New Stack’s account, DeepSeek had considered releasing the whole system and chose to contribute pieces to existing projects instead. Its reasons, in short: the engine had grown from an early vLLM version into a heavily customized system, a small research team would struggle to maintain another major open-source project, and the code was entangled with internal infrastructure.
The entanglement point matters most for what follows. The post tied it specifically to cluster management tooling.
DeepSeek also made a promise about hardware. It said it would synchronize inference engineering with partners ahead of future model launches, aiming for day-one availability across chip architectures. Keep that in mind for the Huawei section below.
What shipped on October 20
The repository supports one model, DeepSeek-V4.1-Flash, on two hardware families: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The default checkpoint is DeepSeek-V4.1-Flash-1017, the updated weights DeepSeek released days earlier.
Two companion facts shape the picture. DeepSeek credits vLLM as the base: DeepInfer was originally built primarily on it, with inspiration from SGLang and TensorRT-LLM. And a second repository, deepseek-recipe, ships alongside. It handles how API requests become model inputs and how outputs become responses, in Rust and Python, so self-hosters can match the behavior of DeepSeek’s official API.
The upstream loop DeepSeek described in 2025 is already visible. TNW reports that the vLLM project thanked DeepSeek and said it would keep improving its own V4.1-Flash support with ideas from DeepInfer. The repository had passed 8,000 stars by Tuesday afternoon, per the same report, and showed about 12,400 stars and 1,100 forks at the time of writing.
The scope is the argument
Here is the boundary, drawn from the README’s own list of what a complete inference service still needs.
The announcement called DeepInfer “fully open source.” The README is more careful. It describes offline inference with the core features needed for serving at scale, then lists three components it does not include: a KV cache system, VE/prefill/decode disaggregation (with the note that most of the required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode.
Set that against the entanglement objection. What the README leaves out is the layer that routes requests to replicas, decides where cached prefixes live, and splits phases across machines. Those parts depend on your network, your scheduler, and your traffic. The release boundary sits roughly where the coupling to infrastructure would be tightest.
That is my reading, and DeepSeek has not said so. The New Stack makes a similar observation and points out that the company has not explained how it overcame the earlier difficulties.
For planning, the line is simple. Offline batch inference, evals, and research on a cluster you control: DeepInfer is close to complete. A multi-tenant chat API with prefix caching and replica routing: budget significant engineering time for the layer above it.
Inside the engine
The README’s feature list is easier to judge with the switches next to it.
| Feature | What it does | How to turn it on |
|---|---|---|
| Parallelism | Attention DP and balanced CP, plus MoE expert parallelism with EPLB | --ep, --dp, and --cp flags in the offline example |
| Asynchronous scheduling | Overlaps CPU batch scheduling with device execution | On by default |
| DSpark with confidence-scheduled verification | Decodes with draft tokens and uses confidence estimates to schedule verification | DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets max draft length (default 5) |
| Decoder SWA bounded replay | Cuts decoder prefill work by replaying only a bounded suffix | DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 |
| Batch-invariant inference | Keeps results identical across batch compositions | DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) |
| Deterministic sampling | Identical outputs for identical input tokens and sampling parameters | DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 plus a fixed seq_seed (NVIDIA only) |
| Structured outputs | Constrains generation to a regex, JSON schema, or EBNF grammar | so_args passed to SamplingParams |
Read the last column twice. The DSpark recipe sets DEEPINFER_ENABLE_MTP_VARLEN=1, and the deterministic-sampling recipe sets the same variable to 0. If you need reproducible outputs for evals or regression tests, the two documented recipes pull in opposite directions. The README does not quantify the cost, so test before assuming they compose.
Bounded replay exposes a model-specific constant. The README says the retained suffix must cover the sliding-window-attention window plus the maximum backtracking length, and it gives 140 tokens as the default for V4.1. Treat 140 as a V4.1 setting, not a universal one. Anyone adapting the engine to another model has a number to derive rather than copy.
The kernel list is the 2025 strategy, assembled. FlashMLA, DeepGEMM, and DeepEP are the Open Source Week repositories. TileKernels, DeepSelect, and custom TileLang kernels in the tree join them. Read together, DeepInfer is where the formerly separate components meet a scheduler and a model runtime.
The vision encoder rides along
V4.1-Flash accepts images, and the engine handles them in-process. With --prompt_mode image, the examples feed raw images to the engine directly, with the ViT and the LLM in the same engine. CPU preprocessing runs in asynchronous MultimodalProcessor worker processes, off the step critical path.
That explains the odd “VE” in the list of missing components. VE is the vision encoder, and the examples benchmark it on its own. A sweep script loads only the VE-side weights, then times the GPU-side pad, ViT, downsample, and aligner path across image sizes and batch counts. Throughput is scaled by the number of LLM tokens each image produces, the same unit as the prefill curve, so the two are comparable.
So “VE/prefill/decode disaggregation” is a three-stage split with the encoder as a separable stage. What the release leaves to you is the orchestration around it.
The vision regression suite is refreshingly small: five inputs, made of two text-only samples, a single-image sample, a three-image sample, and a five-image four-shot sample. Every image comes from just two pictures, carrots and ribs.
Reading the benchmark charts without fooling yourself
The README shows two Pareto plots: prefill (non-cached input throughput against latency) and decode (output throughput against per-user speed). DeepSeek measured them on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points to run_sweep.sh for reproduction. TNW relays Bloomberg’s note that no one has independently benchmarked DeepInfer’s gains yet.
The examples README documents the harness in unusual detail, and one line changes how you should read the decode curve. Decode throughput is computed like this:
decode tokens/s/GPU = avg_SPS x decode_num_requests x (1 + nextn) / cp
The docs call this full-acceptance accounting: every request is credited with 1 + nextn tokens per step, regardless of the true acceptance rate. With the defaults of 64 requests and a draft length of 5, one step counts as 384 tokens.
Real speculative decoding rarely accepts every draft token, so a full-acceptance figure is a ceiling. The sweep script takes an acceptance-rate argument, and the docs’ own example sweeps a draft length of 3 and scales the decode plot by 0.75. Which scaling, if any, the published figure uses is not stated. Before comparing against vLLM or SGLang, check that setting.
The methodology has strengths worth copying. The harness separates prefill from decode, drops intervals that collide with MoE load-balance statistics flushes, and runs sweeps with the profiler off because it perturbs timings. It also warns that profiler-traced step times can be inflated by nearly 50% at batch size 1 and must not be used to derive steps per second.
Numerics are held to an exact standard. Perplexity baselines live in tests/models/offline_decode_ppl.json, keyed by platform and workload, and validation compares them sample by sample. A refresh_offline_decode_ppl.py script updates them once a numeric change is confirmed as expected, and the docs say to re-measure the short and long baselines on NVIDIA and Ascend separately after changing the vision samples. A --check_structured_output flag extends the same run to regex, JSON, and EBNF checks.
If you plan to compare DeepInfer with another engine, hold these constant. The list is mine, built from what the harness documents:
Hardware and scale-up domain. Match the layout behind the published figures, or state how yours differs.
Workload shape. Harness defaults are 32,768 context tokens with 4,096 non-cached. The documented nightly profile configuration uses 128k context with YoCo clipping.
Draft length and token accounting. State your draft length, and whether you count full acceptance or measured acceptance.
Raw quantities.
--result_jsonlappends the configuration plus window step count, token count, elapsed time, andavg_SPS, so you can recompute throughput under your own accounting.One tag per curve. The sweep tooling blocks plotting when model,
ep_size,--cp, context length, draft length, or redundant expert count differ within a tag. Borrow that discipline.
Nvidia and Huawei, side by side
The hardware list is the most strategic line in the README. Next to NVIDIA’s SM100 parts, which The New Stack identifies as Blackwell data-center GPUs, sits Huawei’s Ascend 950 series, with CANN 9.2 as its prerequisite. Older NVIDIA architectures such as Hopper are absent.
The New Stack places this inside a run of reporting. Bloomberg said in September that DeepSeek planned to deploy at least 160,000 Ascend 950DT chips at a new Inner Mongolia data center. Reuters later that month reported DeepSeek and Huawei collaborating on tools to make Ascend chips easier to program. TNW adds that DeepSeek published tooling for Huawei chips earlier in October as an alternative to CUDA.
Parity is incomplete. Batch-invariant inference and deterministic sampling are NVIDIA only, and the published performance numbers all come from NVIDIA GPUs. The README gives no Ascend numbers.
The examples do show Ascend taken seriously. They document debugging variables such as ASCEND_LAUNCH_BLOCKING=1, which runs operators synchronously so stack traces are readable, and, with the separate numeric baselines described earlier, that reads as a first-class target rather than an afterthought.
This matches the 2025 promise of day-one cross-chip support, delivered for one model on two chip families. It is a narrow delivery of a broad commitment. TNW reports that DeepSeek says future models and improvements will land in the same repository.
Running it today
The prerequisites are Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Start from an empty Python 3.12 environment.
export DEEPINFER_ACCELERATOR=nvidia # use "ascend" on Ascend
./build.sh
# first run downloads DeepSeek-V4.1-Flash-1017 into the Hugging Face cache
python examples/offline_inference.py --ep 8
The feature switches in the table above are environment variables, so export them in the same shell before launching.
The offline example sets its layout with flags. The documented variants are --ep 8, --ep 2 --dp 8, and --ep 2 --cp 8. Set DEEPINFER_HF_MODEL_DIR to run from a local copy of the weights.
The benchmark harness works differently, and the difference trips people up. There, ep_size equals WORLD_SIZE (the number of nodes) times the number of visible devices, and --cp must divide it. Redundant experts default to 32 through DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS, but that value is only a lower bound: the engine rounds the total up so the expert count divides evenly by ep_size.
The hardware requirement is the main gate: the README supports SM100 and Ascend 950 only.
Who should adopt it
| If you are | Verdict | Why |
|---|---|---|
| An inference-engine engineer | Read the code this week | A major lab’s account of what matters at scale, under Apache 2.0 |
| A hosting provider on Blackwell or Ascend 950 | Pilot it | You get the kernels and parallelism, but you supply KV cache, disaggregation, and routing, and you must verify the speed claims yourself |
| A team running offline batch jobs or evals on SM100 | Strong fit | Offline inference is what shipped, and batch invariance helps reproducibility on NVIDIA |
| An app team calling a hosted API | Skip for now | Nothing in the release changes how you call a model |
| A team on Hopper or older GPUs | Wait | The hardware is unsupported as released, so keep using vLLM or SGLang for now |
What to watch next
Four signals will show how far DeepInfer travels beyond the announcement.
Independent benchmarks on identical hardware and identical token accounting, against vLLM and SGLang.
The missing layers. The README calls them required for a complete service. Whether DeepSeek will ship them is unannounced.
What lands upstream. vLLM said it would use ideas from DeepInfer. Watch for merged changes rather than statements.
Ascend parity for batch-invariant inference and deterministic sampling.
The 2025 objections did not disappear. They moved to the edge of the release, where your own cluster begins. For anyone who builds or tunes inference engines, DeepInfer is a look at what a major lab says it runs in production. For teams that need a serving stack today, it is a release to watch rather than deploy.
Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.
Top comments (0)