On October 20, 2026, DeepSeek open-sourced DeepInfer — the inference framework it says powers its own large-scale production services — under the Apache 2.0 license. The repo is deepseek-ai/DeepInfer, and it crossed 10,000 GitHub stars within days of the announcement (as of writing, ~10.5k stars / 926 forks).
The one-line version from the official account:
"Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput..." — @deepseek_ai on X
There has been a wall of hot takes since (the Hacker News thread passed 1,400 points). Most of them skip the README, which is actually quite specific about what this release is and — more interestingly — what it is not. This post is a walkthrough of exactly that: what's in the box, what's deliberately left out, where it comes from, and who should actually deploy it.
What DeepInfer is
From the README, verbatim:
"DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale."
Two words in that sentence matter more than the rest:
- "production" — this is not a research playground. It's the same stack that serves DeepSeek's own API traffic, open-sourced as-is rather than rebuilt for external consumption.
- "offline inference" — the release ships offline (batch-style) inference. A full online serving service needs components DeepInfer does not include (more on that below).
It supports exactly one model family today: DeepSeek-V4.1-Flash (default checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 on Hugging Face). This is a model-hardware co-optimized production engine, not a universal deployment framework for every model on HF.
What's in the box
The feature list from the README, with the configuration knobs that go with them:
| Feature | How it's enabled |
|---|---|
| Optimized kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, custom TileLang kernels) | built in |
| Draft-token decoding + confidence-scheduled verification |
DEEPINFER_ENABLE_MTP_VARLEN=1; max draft length via DEEPINFER_USE_NEXTN_DECODE (default 5) |
| Parallelism: attention DP, balanced CP, MoE EP + EPLB | built in |
| Decoder SWA bounded replay (only replay a bounded suffix of the context) |
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140
|
| Asynchronous scheduling (overlap CPU batch scheduling with device execution) | enabled by default |
| Batch-invariant inference (identical results across batch compositions) |
DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only) |
| Deterministic sampling (identical outputs for identical inputs and seeds) |
DEEPINFER_BATCH_INVARIANT=1 + fixed seq_seed (NVIDIA only) |
| Structured outputs (regex / JSON schema / EBNF grammar constraints) | via SamplingParams(so_args={...})
|
Two of those deserve a note:
- Batch invariance + deterministic sampling kill a whole class of production headache. If you've ever been burned by "the same prompt gives a different answer depending on who else is in the batch," this is the guarantee that removes that bug class — the kind of property usually only achieved with bespoke deployment tricks. Note, though, that it's NVIDIA-only in this release.
- Confidence-scheduled verification is speculative decoding where the scheduler uses confidence estimates to decide how to spend verification budget — a scheduling refinement rather than a new algorithm, but the kind of thing that only gets built if you actually serve enormous volume.
What it's built on — and the hardware bet
The acknowledgments section settles the "is it a vLLM fork?" question:
"DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM."
So yes — vLLM lineage. What DeepSeek added on top is a stack of self-optimized kernels (FlashMLA, DeepGEMM, DeepEP are DeepSeek's own projects) and a scheduler designed around their model's architecture.
The hardware support is the political headline, though:
- NVIDIA SM100 GPUs (CUDA 13)
- Huawei Ascend 950 NPUs (CANN 9.2)
That second line is unusual for a top-tier open-source inference release: Ascend support sitting next to NVIDIA from day one, in a production engine. Requirements are strict on both sides — Linux, GCC ≥ 13.0, Python 3.12, and either CUDA 13 or CANN 9.2. This is not a laptop demo framework.
Benchmarks: transparency instead of a hero number
The README doesn't lead with a single "we're X% faster" number. Instead it publishes two Pareto plots — prefill throughput vs. latency, and decode throughput vs. per-user speed — measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, with a run_sweep.sh script so you can reproduce the configuration yourself.
That's the right shape for claims like these: a Pareto curve with a reproduction script is auditable in a way a bar chart never is. Whether the curves hold up on your interconnect is, of course, a different question — the sweep script exists precisely because the answer depends on your topology.
What's deliberately missing
This is the part of the README most coverage skipped. DeepInfer is open about requiring three subsystems for a complete inference service that are not in this release:
- A KV cache system
- VE/prefill/decode disaggregation (though the README notes most required engine functionality is already implemented)
- Load balancing across data-parallel replicas for both prefill and decode
In other words: this is an engine, not a turnkey server. If you were hoping to swap it in as the backend behind your OpenAI-compatible gateway tomorrow, you'll be building or bolting on those pieces. That's also a fair reading of DeepSeek's release philosophy: publish the hard, differentiating core; let the surrounding infrastructure stay open-ended.
One caveat worth repeating: batch invariance and deterministic sampling are NVIDIA-only in this release, and Ascend support, while present, carries exactly the same "read the fine print" weight as any first-version port.
Quickstart
Build from source (create an empty Python 3.12 environment first):
Top comments (0)