DEV Community

TechPulse
TechPulse

Posted on

DeepSeek Open-Sources DeepInfer, Its Production Inference Engine

#deepseek #ai #inference #opensource

DeepSeek open-sourced DeepInfer on October 20, 2026, the high-performance inference framework that runs its large-scale production services for the DeepSeek-V4.1-Flash model. The release, under the Apache 2.0 license on GitHub, gives developers a look at a frontier lab's real serving stack, including optimized kernels, speculative decoding, and dual support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

What Happened

DeepSeek announced the open-source release of DeepInfer on X, stating it powers DeepSeek-V4.1-Flash at scale with high throughput and low latency. The company aims to help the community lower compute costs, reduce energy use, and make AI more affordable. Alongside the engine, it released deepseek-recipe, an MIT-licensed component for token encoding and decoding that matches the official API behavior.

The framework was initially built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. It supports offline inference for DeepSeek-V4.1-Flash only at this stage, with plans to add future models. Key features include optimized kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect), parallelism for attention and mixture-of-experts, DSpark speculative decoding with confidence-scheduled verification, asynchronous scheduling, batch-invariant inference, deterministic sampling, and structured outputs.

Technical Details and Limitations

DeepInfer runs on Linux with Python 3.12, GCC 13 or later, and either CUDA 13 (for NVIDIA) or CANN 9.2 (for Ascend). It is not a complete serving solution: the release excludes the KV cache system, full prefill/decode disaggregation, and load balancing across machines. Production clusters reportedly handle 155 trillion tokens per day, with high cache hit rates, but those figures are self-reported.

Benchmarks published by DeepSeek for V4.1-Flash at long contexts show high throughput on the supported hardware, though independent verification is limited so far. The dual hardware support is notable amid efforts by Chinese firms to reduce reliance on NVIDIA while still serving global customers.

Why It Matters

Inference costs remain a major barrier for enterprises deploying large models, especially for agentic workloads that consume many tokens. By open-sourcing its production engine, DeepSeek pressures Western providers on pricing and gives developers tools to self-host efficiently. The move also positions the company as a leader in open-source AI infrastructure, even as U.S. restrictions on DeepSeek products for federal systems highlight ongoing geopolitical tensions.

For developers and businesses, this offers a practical path to lower serving costs if they have access to the supported GPUs or NPUs. It is not a drop-in replacement for a full API but a significant building block.

Sources

Top comments (0)