DeepSeek V4.1 Flash on 8 B300: vLLM vs DeepInfer, 600K+ tokens/s

Kirill Bespalov — Wed, 21 Oct 2026 15:04:21 +0000

DeepSeek has open-sourced DeepInfer, the engine behind its own production service. We put it head to head with vLLM on a single 8×B300 node running DeepSeek-V4.1-Flash-1017, and pushed both to the limit on output tokens and on prefill.

TL;DR: on one node vLLM is 2.0× faster on decode (1K → 1K), 4.3× at 8K → 1K, and 9× on prefill. It peaks at 658K prefill tokens/s and 55–65K output tokens/s.

The engines

Both run the same model with mostly the same kernels (FlashMLA, DeepGEMM, DeepEP) and scale it the same way: attention replicated per GPU, experts sharded across GPUs. The difference is scope.

	vLLM	DeepInfer
Models	Hundreds	DeepSeek-V4.1-Flash only
Hardware	NVIDIA, AMD, TPU…	NVIDIA SM100, Huawei Ascend 950
Interface	OpenAI-compatible server	Python offline API (`LLM.generate`), no server
Install	Docker / pip	Build from source (CUDA 13.1+)
Highlights	Prefix caching, structured outputs, LoRA, disaggregated prefill	DSpark speculative decoding, batch-invariant inference, deterministic sampling
Built for	Any scale	4 GPUs per prefill group, 64 per decode group

Setup

GPUs: 8× NVIDIA B300 SXM6, 275 GB HBM3e each, NVLink 5
Model: DeepSeek-V4.1-Flash-1017, 476 GB (FP4 experts, FP8 the rest, two Engram tables)
vLLM: 0.31.1rc1, B300 recipe, DEP8, 256 seqs per rank, DSpark with 5 draft tokens
DeepInfer: commit 8b16149, EP8 × DP8, DSpark with 5 draft tokens, offline API
Load: vllm bench serve, random-token prompts, fixed output length; DeepInfer got the same prompts in-process

Random prompts are the worst case for speculative decoding (19% draft acceptance at 1K, 12% at 8K), so every decode number here is a floor.

How DEP8 lays the model out

The model is MoE: 384 routed experts + 1 shared per layer, 6 per token. vLLM's recipe runs it as DEP8: each GPU has its own attention and KV cache and holds 48 experts; tokens travel to their experts over NVSwitch every MoE layer.

Memory is not the bottleneck: weights take 46 GiB per GPU and the KV cache gets 160 GiB (room for ~52M tokens per rank). The real limit is 256 running sequences per rank, 2,048 per node.

vLLM at full load

Prefill (one output token per request): 658K tokens/s with 8K prompts, about 82K per GPU.

Decode with 2,048 concurrent requests: 65.0K output tokens/s with short prompts, 55.0K at 1K → 1K (29 ms per token).

In → Out	Concurrent	Output tok/s	Total tok/s	TPOT
128 → 1K	2,048	64,990	73,114	24.8 ms
1K → 1K	2,048	54,985	109,969	28.8 ms
8K → 1K	2,048	29,500	265,496	52.8 ms
2K → 128	2,048	19,753	335,801	74.4 ms

One gotcha: our first run made only 37.2K tok/s at 1K → 1K because CUDA graphs were captured only up to 1,536 tokens. Under load, steps mix decode with prompt chunks and every DP rank pads to the largest, so those steps ran without graphs. Capturing graphs up to 8,192 tokens brought it to 55.0K.

DeepInfer on one node

Getting it running took longer than benchmarking it: source build with NVCC 13.1+, and it needs an RDMA device even on a single node (NCCL GPU-initiated networking for Engram rows).

In → Out	Concurrent	DeepInfer	vLLM	vLLM ÷ DeepInfer
1K → 1K	2,048	27,906	54,985	2.0×
1K → 1K	4,096	26,294	55,506	2.1×
8K → 1K	2,048	6,933	29,500	4.3×
32K → 1 (prefill)	512	64,847	589,924	9.1×

DeepInfer's own benchmark agrees (66K prefill tok/s per node), so it's not our harness.

Why the gap? DeepSeek benchmarks DeepInfer with prefill on 4 GPUs and decode on 64 GPUs, each in one NVLink domain. That's a rack, not a server, with prefill and decode on separate machines. The pieces that hold such a deployment together (KV cache system, P/D disaggregation, load balancing) aren't in the open-source release. On one node DeepInfer has to do everything at once, which is exactly what vLLM's recipe is built for. So read this as "DeepInfer on one node", not a verdict on its kernels.

Which engine, when

API for users on one or a few nodes: vLLM. 2× decode, 9× prefill, and a ready OpenAI-compatible server.
Any model other than DeepSeek-V4.1-Flash: vLLM.
Reproducible offline generation (RL rollouts, evals, synthetic data): try DeepInfer for batch invariance and deterministic sampling, but measure throughput on your shapes first.
Rack-scale NVLink (GB200/GB300 NVL72): look at both. DeepInfer is how DeepSeek serves at that scale, but you'd build the KV cache, disaggregation and load balancing yourself.
Huawei Ascend 950: DeepInfer.

For most teams with a node or two, the answer is vLLM. DeepInfer is worth following as the reference for how DeepSeek runs its own model, but on a single node it isn't the faster engine.

Originally published at warmctl.io, with the full setup, startup-time breakdown and raw numbers.