DeepSeek Open-Sourced DeepInfer. It's a vLLM Fork, and That's the Interesting Part

DeepSeek open-sourced DeepInfer, the inference engine that serves its production models. It hit the top of Hacker News with 2,000+ points and the repo passed 16k stars almost immediately.

Hype aside, what did they actually ship? I read the repo so you don't have to.

What it is

An Apache 2.0 inference framework for running DeepSeek models across lots of accelerators. Initial support is exactly one model: DeepSeek-V4.1-Flash. Hardware targets:

• NVIDIA SM100 GPUs
• Huawei Ascend 950 NPUs

The README is refreshingly honest about lineage: it was "originally built primarily on vLLM," with ideas borrowed from SGLang and TensorRT-LLM. So this is not a clean-sheet engine. It's a vLLM descendant with DeepSeek's kernel stack bolted on.

What's inside

The kernel list reads like a greatest-hits of DeepSeek's open-source drops:

• FlashMLA: attention kernels for multi-head latent attention
• DeepGEMM: FP8 GEMMs
• DeepEP: expert-parallel communication
• TileKernels and DeepSelect, plus custom TileLang kernels

On top of that:

• Attention DP / balanced CP, and MoE expert parallelism with load balancing (EP + EPLB)
• DSpark speculative decoding with confidence-scheduled verification
• Async batch scheduling with "zero-overhead" claims
• Decoder SWA bounded replay to cut prefill compute

The feature I actually care about: determinism

Two flags matter more than any throughput chart:

# Same output regardless of what else is in the batch
DEEPINFER_BATCH_INVARIANT=1

# Deterministic sampling with a fixed seed
DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0

Batch-invariant inference means your output doesn't change because some other user's request landed in the same batch. If you've ever tried to debug an eval that flakes at temperature 0, you know why this is gold. Most engines give you "mostly deterministic." This one says it has a mode that guarantees it. Verify that claim on your workload before you bet on it.

Structured output (regex, JSON schema, EBNF) is also built in.

Running it

Requirements are not casual: Linux, GCC 13+, Python 3.12, CUDA 13 (or CANN 9.2 for Ascend).

export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

Point it at local weights with DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017.

What's missing (read this before you get excited)

This is not a drop-in serving stack. The README says the release is offline inference with the core features for serving at scale, and explicitly leaves out:

• KV cache management
• Prefill/decode disaggregation
• Load balancing across data-parallel replicas

Those are exactly the pieces that make or break production serving. DeepSeek ships a companion "deepseek-recipe" to reproduce their API's token processing, but you're still assembling the rest yourself.

Benchmarks are Pareto charts (prefill on 4 SM100s, decode on 64) in the repo's images folder. I found no independent head-to-head against vLLM or SGLang yet. Treat any "X times faster" tweet as unverified until someone reproduces it. Those two engines have traded the lead for two years, so a single chart proves little.

Who should care

• Running DeepSeek V4.1 Flash at scale on SM100 or Ascend? Try it now. It's the vendor's own stack.
• Running Llama or Qwen on H100s? Nothing here for you today. One model, one hardware family.
• Building inference infra? Read the source. A production-grade reference for MoE expert parallelism and speculative decoding under Apache 2.0 is a free education.

My take

The value isn't a vLLM killer. It's that a frontier lab published the serving layer, not just weights. Weights without the engine are half a release, because the engine is where the cost per token lives. The stated goal is to help the community cut compute costs and energy use, and the API-pricing pressure that follows is the real story for anyone paying for tokens.

Watch for two things: someone running independent benchmarks, and the missing disaggregation pieces landing. Until then, it's a seriously impressive reference implementation, not a turnkey replacement.

Sources: the DeepInfer GitHub repo, The Decoder's coverage, and the Hacker News thread.