TL;DR: We integrated DeepInfer with GPUStack through a lightweight serving project, deployed DeepSeek-V4.1-Flash on a single server with eight NVIDIA B200 GPUs, and compared the resulting service with vLLM under a matched DP8+EP8 setup. In this run, DeepInfer-server delivered higher reported output throughput and shorter inter-token latency, while vLLM achieved lower time to first token and better P95 request latency.
Efficient inference technology creates business value only when it can be deployed in a real service.
DeepSeek's open-source DeepInfer gives developers a way to explore its inference technology. Teams that want to run models on their own infrastructure still need to answer several engineering questions: How do we expose an API? How should the runtime be configured? How do we manage GPUs? And how does the engine perform under a real workload?
To answer those questions, we built a lightweight serving project called DeepInfer-server and integrated it with GPUStack's custom inference backend. The result was a working text-generation service for DeepSeek-V4.1-Flash on a single server with eight NVIDIA B200 GPUs.
This article explains the integration, walks through the deployment process, and describes how we compared it with vLLM on the same hardware.
1. What Does DeepInfer Bring?
DeepInfer is DeepSeek's open-source high-performance inference framework. The current public release provides offline inference and the core engine capabilities required for large-scale serving. It supports DeepSeek-V4.1-Flash and integrates computation and communication components including FlashMLA, DeepGEMM, DeepEP, and DeepSelect. It also supports multiple parallelism strategies, speculative decoding, and asynchronous scheduling. See the DeepInfer repository.
These components optimize different parts of the inference path: computation executes faster, data moves between GPUs more efficiently, and scheduling makes better use of the hardware. What users ultimately care about is how many requests the same resources can serve and how long users have to wait.
Turning an engine into a production-facing service, however, also requires request handling, protocol conversion, an HTTP service, and deployment management.
The components in this integration have the following responsibilities:
| Component | Role in this integration |
|---|---|
| DeepInfer | Runs model inference and manages low-level computation, communication, and parallel execution |
| deepseek-recipe | Converts chat requests into model input and parses generated text, reasoning content, and tool calls |
| DeepInfer-server | Connects the components and provides an HTTP API, streaming responses, and service lifecycle management |
| GPUStack | Manages model deployments, GPU allocation, runtime configuration, and model access and testing |
deepseek-recipe is DeepSeek's protocol-conversion and codec component. It translates between API requests, conversation formats, model token sequences, and responses. Model inference and HTTP transport are supplied by the caller. See the deepseek-recipe repository.
Why Build DeepInfer-server?
DeepInfer's current public entry point is primarily designed for offline inference. To start it like vllm serve and make it callable by applications, the engine must be connected to an HTTP request-processing path. deepseek-recipe handles protocol conversion and output parsing, but the caller must still supply model inference and HTTP transport. Installing both components in the same container does not automatically create a complete online inference service.
A GPUStack custom-backend definition tells the platform which image and command to run and which arguments to pass. It does not automatically generate an API around an offline engine. This integration therefore needs a connecting layer that organizes the components into a service that can start, report its state, and receive requests.
DeepInfer-server provides that serving layer:
- Connect requests to inference. It receives Chat Completions requests, uses deepseek-recipe to convert them into model input, and calls DeepInfer to generate output.
- Return output to applications. It converts generated tokens into text, reasoning content, or tool calls; returns standard JSON or SSE streaming responses; and reports token usage.
- Manage the service lifecycle. It initializes inference processes, handles request dispatch, cancellation, timeouts, and errors, and exposes model-list and health-check endpoints.
- Integrate with the deployment platform. It provides a container image and common startup arguments so that GPUStack can pass the model path, port, and GPU configuration and then start and manage the service.
Applications can therefore call the model through the familiar OpenAI-compatible Chat Completions API, while GPUStack can reuse its custom-backend deployment workflow. DeepInfer still performs the underlying inference, and deepseek-recipe still handles protocol conversion and parsing.
The current wrapper focuses on single-node, multi-GPU text inference. It is an independent demonstration project for validating the integration. It does not reproduce DeepSeek's complete production architecture for KV-cache management, disaggregated prefill and decode, or load balancing, and it should not be treated as equivalent to DeepSeek's official online service.
2. Deploying DeepInfer Through a GPUStack Custom Backend
GPUStack's custom inference backend mechanism lets users define a container image, startup command, default parameters, and environment variables so that a new inference service can participate in the normal deployment workflow. Template variables can supply values such as the model path and service port at deployment time. See the GPUStack custom backend documentation.
This allows teams to evaluate new inference engines on the same platform while reusing existing model-management and deployment operations.
The integration includes the following backend definition:
gpustack-deepinfer-server-backend.yaml
Step 1: Prepare the Runtime Environment
Our demonstration used a single server with eight NVIDIA B200 GPUs and the following model:
deepseek-ai/DeepSeek-V4.1-Flash-1017
Before deployment, prepare the complete model weights, the DeepInfer-server image, a compatible GPU driver and container runtime, and sufficient host memory.
The configuration must also expose the correct RDMA devices and set up the communication environment. The NIC and network parameters in the accompanying YAML come from our validation environment and must be adapted to other servers.
Step 2: Import the Custom Backend
On GPUStack's inference-backend management page, create a custom backend and import the DeepInfer backend YAML definition.
The definition contains the service startup method, common arguments, health-check path, and JIT-cache directory. At deployment time, GPUStack supplies the model path, service port, model name, and number of allocated GPUs.
Step 3: Create the Model Deployment
When creating the model deployment, select the model weights and choose the DeepInfer-server inference backend created in the previous step.
To keep the deployment aligned with the benchmark described later, we explicitly set --expert-parallel-size=8, --data-parallel-size=8, and --context-parallel-size=1 in the model's backend parameters. These values override the DP=1 default in the supplied YAML. Verify the final startup configuration before launching the service.
| Setting | Example value |
|---|---|
| GPU count | 8 GPUs in one server |
| Parallel configuration | EP=8, DP=8, CP=1 |
| Maximum context length | 32,768 tokens |
| Maximum running sequences | 16 per DP rank |
| Batched-token budget | 32,768 per DP rank |
| GPU memory utilization | Default 0.85 |
| Startup timeout | 3,600 seconds |
EP, DP, and CP refer to expert, data, and context parallelism. The current wrapper requires DP × CP = EP. We use CP=1 here, so the context is not split across GPUs. max-num-seqs=16 is the limit for each DP rank, giving the DP8 service a total admission limit of 128 sequences.
The first startup loads the weights and may compile and warm up kernels. Use GPUStack to monitor the startup logs and wait for model initialization to finish before running inference tests. A persistent JIT cache helps reuse compiled artifacts.
Step 4: Verify Inference
After the model is ready, send a text request from the GPUStack Playground to verify streaming output. The same service can then be integrated with existing applications through its API.
At this point, the full path from model weights and inference engine to API access is working. Teams can continue testing real prompts, monitoring resource utilization, and evaluating the value of further optimization.
3. How Does It Compare with vLLM on the Same Eight B200 GPUs?
Before looking at the numbers, we need to define the scope of this comparison.
This test is a service-integration and preliminary performance validation. Neither deployment was fully tuned for the hardware, and both differ from DeepSeek's production inference architecture. Because the public DeepInfer release does not expose a TP path corresponding to vLLM's tensor parallelism, we configured DeepInfer with CP1, DP8, and EP8 and vLLM with TP1, DP8, and EP8. This makes the parallel strategies more comparable, but does not represent the best possible configuration for either engine.
DeepInfer's documented parallelism includes Attention DP or balanced CP and MoE EP with EPLB. Our single-node combined service does not reproduce its production disaggregated prefill/decode architecture, complete KV-cache management, or load balancing. See the DeepInfer documentation.
Even with DP8+EP8 on both sides, their kernels, communication paths, Engram access, and schedulers are implemented differently. DeepInfer-server also adds request coordination and protocol processing written for this demonstration, and that overhead is included in the end-to-end results. These numbers do not represent the performance of DeepSeek's official service and should not be used to infer the performance ceiling of either engine.
First, the Interactive Chat Experience
After deployment, we started with a single-request chat in the GPUStack Playground. In the screenshots, the vLLM deployment produced approximately 91.4 tokens/s, while DeepInfer produced approximately 106.2 tokens/s in that interaction.
A single chat result depends on the prompt, context, answer length, and the UI's measurement method. This screenshot illustrates the interactive experience; it is not a single-request performance ranking and should not be compared directly with the concurrent-output throughput below. For the benchmark, both engines used DP8+EP8, an 8K input, a 1K output, and 16 concurrent requests.
Test Environment and Configuration
We used the same eight-B200 server and the same model weights. GPUStack's Benchmark feature issued 100 requests with a target concurrency of 16.
| Item | DeepInfer-server | vLLM |
|---|---|---|
| Hardware | 8 × NVIDIA B200 on the same Worker | 8 × NVIDIA B200 on the same Worker |
| Model | DeepSeek-V4.1-Flash-1017 | DeepSeek-V4.1-Flash-1017 |
| Version | DeepInfer commit 9bd9cf2; service image tag latest
|
0.32.0 |
| Parallel strategy | DP8, CP1, EP8 | DP8, TP1, EP8 |
| Maximum context length | 32,768 tokens | 32,768 tokens |
max-num-seqs |
16 per DP rank | 16 per DP rank |
max-num-batched-tokens |
32,768 per DP rank | 32,768 per DP rank |
| GPU memory utilization | 0.85 | 0.85 |
| Speculative decoding | Disabled | Disabled |
| Dataset | Random | Random |
| Target input/output length | 8,192 / 1,024 tokens | 8,192 / 1,024 tokens |
| Target client concurrency | 16 | 16 |
| Number of requests | 100 | 100 |
Raw Results in GPUStack
The following table places the main metrics side by side:
| Metric | DeepInfer-server: DP8+EP8 | vLLM: DP8+EP8 |
|---|---|---|
| Successful/total requests | 100 / 100 | 100 / 100 |
| Mean output throughput (tokens/s) | 1,392.44 | 1,112.20 |
| Mean time to first token, TTFT (s) | 4.28 | 2.04 |
| P95 time to first token, TTFT (s) | 16.33 | 8.27 |
| Mean inter-token latency, ITL (ms) | 8.74 | 11.98 |
| P95 inter-token latency, ITL (ms) | 8.93 | 13.70 |
| Mean end-to-end request latency (s) | 13.23 | 14.30 |
| P95 end-to-end request latency (s) | 25.46 | 20.03 |
| Actual mean concurrency | 14.72 | 14.58 |
Output throughput, TTFT, ITL, and request latency come from the exported report's output_tokens_per_second_mean, time_to_first_token_*, inter_token_latency_*, and request_latency_* fields. The table converts TTFT from milliseconds to seconds. Output throughput retains the report's mean statistic and is not converted into whole-run duration. ITL is not mixed with the report's time_per_output_token or per-SSE-chunk intervals.
4. What Do These Results Tell Us?
In these two DP8+EP8 runs, DeepInfer-server reported higher mean output throughput and shorter mean inter-token latency, while vLLM delivered a shorter time to first token and lower tail latency.
DeepInfer-server's reported mean output throughput was approximately 25.2% higher than vLLM's corresponding run. Its mean ITL was approximately 27.0% lower, and its mean end-to-end request latency was approximately 7.5% lower. These are differences between two specific test runs; they are not yet proof of a stable performance advantage.
At the same time, DeepInfer-server's mean TTFT was 4.28 seconds, about 2.09× vLLM's 2.04 seconds. Its P95 request latency was 25.46 seconds, approximately 27.1% higher than vLLM's 20.03 seconds. The wait before an answer begins and the experience of the slowest requests both leave room for improvement.
The result demonstrates why an inference service should be evaluated across throughput, time to first token, sustained generation speed, and tail latency. Interactive applications benefit from beginning a response quickly, while batch-generation workloads also care about how much output can be completed per unit of time. No single metric fully describes the service experience.
Both DP8+EP8 configurations completed all 100 requests, validating service availability under this test load. One hundred successful requests are not equivalent to production stability certification, and the Random dataset does not evaluate answer quality.
For GPUStack, the experiment demonstrates a reusable workflow: integrate a new engine through a custom backend, deploy and test it on one platform, and use measured data to choose the runtime that fits the workload. We plan to continue testing fixed prompts and controlled conditions at different concurrency levels, and to separate the overhead of the serving wrapper from engine execution.
5. Continuing the Work to Make Efficient Inference Easier to Use
DeepInfer's open-source release gives the community more inference-optimization techniques that can be studied and validated.
In its deployment Q&A, DeepSeek explains that its KV-cache management and load-balancing components are tightly coupled to internal infrastructure and have not yet been fully open sourced. DeepSeek also states that inference code and performance optimizations for future DeepSeek models will continue to be released through DeepInfer. See the DeepSeek inference deployment Q&A.
This creates room for future integration work. The timing and scope of any additional open-source components should still be based on DeepSeek's official announcements.
GPUStack will continue following DeepInfer and the related computation, communication, and codec components. As components become available and verifiable, we will evaluate further integration through real deployments and tests.
Our goal is to help new inference technology reach developers and enterprises faster: lower the barrier to experimentation, make performance gains reproducible, make infrastructure cost measurable, and enable more teams to run large models effectively and affordably.
Making low-cost inference broadly accessible requires sustained technical optimization—and it also requires making those optimizations easier to deploy and use.
GPUStack: github.com/gpustack/gpustack
DeepInfer-server: github.com/gpustack/DeepInfer-server
Top comments (0)