- Book: System Design Pocket Guide: Interviews
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
DeepSeek's new open-source inference engine comes with a list of what it leaves out. Under the heading "Required for a complete inference service (not included in this release)," the DeepInfer README names three things: a KV cache system, VE/prefill/decode disaggregation, and load balancing across data-parallel replicas (README).
DeepSeek put the engine on GitHub on October 20, under Apache 2.0 (The Next Web). The README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services," and says this release "provides offline inference with the core features needed for serving at scale." By the evening its Hacker News thread had nearly 1,900 points and over 700 comments.
So you get DeepSeek's engine. You don't get DeepSeek's service. That gap decides whether this release is for you.
What DeepSeek actually shipped
The repo supports one model, DeepSeek-V4.1-Flash. Its default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, pulled from Hugging Face on first run (README). That is the checkpoint DeepSeek's API now serves as deepseek-flash (DeepSeek docs). Its model card lists 763B parameters and an MIT license for the weights.
Inside the engine, the README lists DeepSeek's own kernel libraries (FlashMLA, DeepGEMM, DeepEP and more), attention data parallelism, context parallelism, MoE expert parallelism, "DSpark and confidence-scheduled verification," asynchronous scheduling, and structured outputs that "constrain generation to a regex, JSON schema, or EBNF grammar." It says DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." The Next Web reports that vLLM thanked DeepSeek and plans to improve its own V4.1-Flash support with ideas from DeepInfer.
The hardware list is two lines long
DeepInfer runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, on Linux, with GCC 13 or later, Python 3.12, and either CUDA 13 or CANN 9.2 (README). SM100 is compute capability 10.0, which NVIDIA's table lists as the B200 and GB200. Older NVIDIA chips such as Hopper aren't on the list (Trending Topics).
If you do have the hardware, the build is short. These steps come from the README, inside an empty Python 3.12 environment:
export DEEPINFER_ACCELERATOR=nvidia # or: ascend
./build.sh
python examples/offline_inference.py
Set DEEPINFER_HF_MODEL_DIR to a local copy of the checkpoint to skip the download, and DEEPINFER_ENABLE_MTP_VARLEN=1 to turn on DSpark (README).
The performance charts are "measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain" (README). The README text gives no throughput numbers next to them. Those charts are DeepSeek's own, and "nobody outside the company has independently tested them yet" (The Next Web). Trending Topics notes there are no published benchmarks at all for the Huawei chips. I haven't run DeepInfer myself.
The three missing pieces are the service
Each item on the "not included" list is a system you'd have to build or borrow before real traffic hits this engine.
A KV cache system. The engine still holds KV state while it generates a response. What's missing is the system that stores that state and reuses it across requests. The attention state for a long, shared prefix (a system prompt or a tool list) is expensive to compute, and a cache system keeps it around so the next request with the same prefix skips that work. Without one, expect each request to pay for its full prompt.
Prefill/decode disaggregation. Reading the prompt and writing the answer stress a GPU in different ways, so large deployments often run them on separate pools. The benchmark setup above, 4 GPUs for prefill and 64 for decode, looks like that split. That's my reading, not DeepSeek's. For disaggregation, the README notes that "most required engine functionality is already implemented," but the wiring isn't in this release.
Load balancing across data-parallel replicas. Once you run more than one copy of the engine, something has to route each request to the right one. That router isn't here either.
Two ideas worth reading even without a Blackwell rack
The first is reproducibility. DeepInfer has batch-invariant inference, to "keep model results identical across different batch compositions," and deterministic sampling, to "produce identical outputs for identical input tokens and sampling parameters" (README). Both are NVIDIA-only for now. If you've ever rerun an eval and watched the score move with nothing changed, this is the problem being named, and batch composition is one source of that drift. Batch invariance is a switch, DEEPINFER_BATCH_INVARIANT=1. Deterministic sampling also needs DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed in SamplingParams, so you give up DSpark to get reproducible output.
The second is deepseek-recipe, MIT-licensed Rust libraries with Python bindings that render DeepSeek V4 and V4.1 prompts and parse model output back into API responses. It covers the Messages, Chat Completions and Responses request formats. Its README is clear about scope: "The caller must provide model inference, tool execution, and HTTP transport." It matches how the official API turns text into tokens and back (The Next Web), so if you serve V4.1-Flash on another engine, it's the obvious place to check your prompt formatting.
python3 -m pip install deepseek-recipe
Who should care this week
Hosts and inference providers. Cloud and hosting companies that already run DeepSeek's models can now use the same methods DeepSeek uses (The Next Web), and Bloomberg, as cited there, reported that companies running the models on their own servers could also cut costs. If you serve V4.1-Flash on B200-class hardware, read the kernels before you tune anything else.
Everyone on vLLM. The likelier route to you is indirect. If vLLM folds DeepInfer's ideas into its V4.1-Flash support, as The Next Web reports it plans to, you get some of the gain without changing engines.
API callers. Nothing in your code changes. Your requests to deepseek-flash already go to the production stack this engine powers.
Teams selling to the US government. The day before the release, the Department of Homeland Security published an order telling federal civilian agencies to remove DeepSeek products from their systems, and it covers third-party tools built on DeepSeek's models (The Next Web, Trending Topics). An Apache 2.0 license doesn't settle that question. This isn't legal advice: read the order with your counsel before DeepInfer goes anywhere near a federal deal.
Would you self-host V4.1-Flash on DeepInfer and build the cache and router yourself, or wait for vLLM to absorb the ideas? Tell me in the comments.
If this was useful
A KV cache, a prefill/decode split and a router across replicas are classic system-design trade-offs. The System Design Pocket Guide: Interviews walks through 15 real system designs step by step, trade-offs included, so you can see how teams usually settle them. I also build Hermes IDE, a free, source-available IDE for running Claude Code, Codex and other coding agents side by side on macOS, Windows and Linux (GitHub).
Top comments (0)