DEV Community

Deduped
Deduped

Posted on Originally published at dedupedai.com

How to run DeepSeek's DeepInfer: build, deterministic outputs and JSON schema

#ai #deepseek #machinelearning #llm

DeepSeek open-sourced DeepInfer, the inference framework behind its large-scale production services. This release does offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.

This guide covers the three things most engineers will want first: getting a build running, making outputs repeatable for regression tests, and forcing output into a JSON schema. Everything below comes from the repo's README.

Before you start: what this release is and is not

It is offline inference with optimized kernels, attention DP and MoE expert parallelism, draft-token decoding, asynchronous scheduling, batch-invariant inference, deterministic sampling and structured outputs.

It is not a complete serving stack. The README lists what you would still need: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Plan accordingly.

Prerequisites: Linux, GCC 13.0 or newer, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend).

Step 1: Build from source

Create and activate an empty Python 3.12 environment first. Then:

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
Enter fullscreen mode Exit fullscreen mode

Step 2: Run offline inference 

python examples/offline_inference.py
Enter fullscreen mode Exit fullscreen mode

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on the first run. To use a local copy instead:

export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
Enter fullscreen mode Exit fullscreen mode

The repo's examples/README.md has the details of each example.

Step 3: Make outputs repeatable for regression tests

Before and after comparisons are useless if outputs drift with batch composition or sampling randomness. DeepInfer has two switches for that, both NVIDIA only:

export DEEPINFER_BATCH_INVARIANT=1
export DEEPINFER_ENABLE_MTP_VARLEN=0
Enter fullscreen mode Exit fullscreen mode

Then pass a fixed seq_seed to SamplingParams in your script. Same input tokens plus same sampling parameters now give the same output. A simple check: run the same prompt file twice and diff the two outputs. An empty diff is a pass.

Note the tradeoff: DEEPINFER_ENABLE_MTP_VARLEN=1 is what turns on the draft-token decoding speedup, so deterministic runs give that up.

Step 4: Force output to match a JSON schema

Structured outputs constrain generation to a regex, a JSON schema or an EBNF grammar. Pass them through SamplingParams:

so_args={"schema_type": ..., "schema": ...}
Enter fullscreen mode Exit fullscreen mode

Start from examples/_structured_output.py in the repo, which shows how the constraint is built, then parse the returned text with json.loads and check the required keys.

Step 5 (optional): Reproduce the published benchmarks

The README's prefill and decode charts were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. The sweep script is at examples/sweep/run_sweep.sh. On a smaller setup your numbers will not match the charts directly, so compare shapes, not absolutes.

Use cases with copy-paste agent prompts

DeepInfer is released under the Apache License 2.0.

Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com

Top comments (0)