DeepSeek open-sourced its production inference engine today. I counted what's in it — and what's not.

DeepSeek open-sourced DeepInfer this morning — described in the repo as "the high-performance inference framework that powers DeepSeek's large-scale production services." It was the top story on Hacker News within hours (1,080 points when I checked). "Powers our production service" is exactly the kind of claim worth opening the box for, so I cloned it and counted.

The short version: the engine is real, substantial, and honest about its limits. The parts that make it a production service are, by DeepSeek's own admission, not in the box.

What is actually in the repo

A shallow clone today (HEAD 774ce24, Oct 20) gives you:

files total                     342
Python files                    270
lines (py/cpp/hpp/cu/h)         57,354
  deepinfer/ package            46,353
  csrc/                          5,908
  tests/                         3,117   (20 files)
  examples/                      3,018
  eval/                          1,694
DEEPINFER_* env vars            64
TODO/FIXME/NotImplemented       41 hits

57k lines is not a demo. There is a real scheduler, MoE expert-parallel load balancing (EPLB), speculative decoding via DSpark with confidence-scheduled verification, batch-invariant inference, deterministic sampling, and structured outputs via regex/JSON-schema/EBNF. The eval harness is also real: AIME 2026 (30 tasks × 32 samples), HLE (2,500 tasks), GPQA Diamond (198 × 8), Pass@1, with generation and scoring as separate stages. The datasets are not included — you fetch and convert them yourself.

The acknowledgments say DeepInfer "was originally built primarily on vLLM," with inspiration from SGLang and TensorRT-LLM. Whatever the lineage, the string vllm survives in only 23 Python files (24 mentions), so this is a rewrite/fork that has mostly been scrubbed into its own namespace, not a thin wrapper.

The support matrix is one model and two chips

This is the part the launch coverage tends to skip:

supported models     DeepSeek-V4.1-Flash          (exactly one)
supported hardware   NVIDIA SM100 GPUs
                     Huawei Ascend 950 NPUs
prerequisites        GCC >= 13, Python 3.12
                     CUDA 13 (NVIDIA) / CANN 9.2 (Ascend)
                     torch==2.13.0+cu132, flash-attn-4==4.0.0b34 (beta)

SM100 is Blackwell. If your fleet is H100s or A100s, this release is not for you, at least not yet. The one supported checkpoint is the same V4.1-Flash-1017 whose weights I summed from Hugging Face last week: 48 shards, 510,296,708,312 bytes. The "getting started" path is python examples/offline_inference.py, which starts by downloading half a terabyte.

The kernels are also not in the clone until you ask for them. Six directories under vendor/ — DeepEP, DeepGEMM, FlashMLA, DeepSelect, plus Ascend variants of DeepEP and DeepGEMM — are git submodules, and all six are empty in a default clone. The famous DeepSeek kernel stack is a git submodule update --init away, which is fine, but "open-sourced the engine" and "the engine, self-contained" are different claims.

What DeepSeek itself says is missing

Credit where due: the README has a section most launch posts would bury. "Required for a complete inference service (not included in this release)":

• A KV cache system
• VE/Prefill/Decode disaggregation (most engine functionality implemented, the system around it not shipped)
• Load balancing across data-parallel replicas for prefill and decode

The bundled deployment Q&A (doc/inference_deployment_qa.md, the most interesting document in the repo) explains why. In production, DeepSeek's design goals include retaining KV cache for more than 72 hours, on dedicated storage servers specced at 20 × 30.73 TB SSDs each, ≥40 GB/s read, working out to roughly 14.4 TB of logical KV cache per GPU. The Q&A states plainly that "the KV cache management and load balancing components of the inference system are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced."

So the two components that turn an inference engine into DeepSeek's cost curve are the two components that stayed home. What shipped is the engine room, not the ship.

The production numbers are in the Q&A, not the benchmarks

The repo's benchmark section is four PNG Pareto charts — prefill measured on 4 SM100 GPUs, decode on 64, "each within one scale-up domain." There is no CSV or JSON of the underlying data anywhere in the repo (I searched; the only JSON data file is a perplexity fixture in tests). You can rerun examples/sweep/run_sweep.sh if you happen to have 64 Blackwell GPUs. You cannot check their numbers from the repo alone.

The Q&A's self-reported production figures are more concrete, and worth quoting as claims rather than measurements:

cluster throughput        155 trillion tokens/day      (their figure)
revenue                   US$38 per GPU-hour at valley pricing  (their figure)
decode config             EP64-128 / DP64-128 (Wide-EP)
smallest *efficient* unit 64 prefill GPUs + 64 decode GPUs
interconnect budget       ~400 Gbps per 8 GPUs (cross-DC)
measured moe_lb           1.045-1.090 across configs   (their figure)

"Smallest efficient deployment unit: 128 GPUs" tells you who this release is for. It is a reference implementation of how a frontier lab serves a flagship MoE — genuinely valuable if you run, or plan to run, that class of system, and mostly a reading exercise if you don't.

Verdict

Believe "open-sourced the inference framework": 57k lines, real tests (3,117 lines of them, most GPU-bound), real eval harness, Apache 2.0. Discount "the framework that powers production" by exactly the three items DeepSeek lists as not included — KV cache, disaggregation plumbing, replica load balancing — because by the Q&A's own numbers (72-hour KV retention, 14.4 TB per GPU) those are where the production economics live. The most honest sentence in the launch is in their own README, and it's the exclusions list.

Sources: github.com/deepseek-ai/DeepInfer (cloned and counted today, HEAD 774ce24), its README and doc/inference_deployment_qa.md; checkpoint size from the Hugging Face file listing for deepseek-ai/DeepSeek-V4.1-Flash-1017, which I summed in an earlier post. All counts are mine; production figures are DeepSeek's, labeled as such.