DEV Community: GPUStack

Introducing GPUStack v2.3: Optimizing LLM Inference with PD Disaggregation, Shared KV Cache, and Elastic Scheduling

GPUStack — Thu, 29 Oct 2026 10:29:43 +0000

The open-source edition of GPUStack v2.3.0 is now officially available!

Automated PD disaggregation orchestration, shared KV cache pools, vGPU partitioning, intelligent benchmarking, and scheduled autoscaling give model services more flexible resource configuration options. They also enable teams to deploy, validate performance, and continuously optimize services through a unified workflow.

These capabilities address practical challenges that arise when models move into production: once Prefill and Decode are separated, how should instances work together? After a service scales out, how can it reuse existing caches? And how many resources should be provisioned for traffic peaks?

GPUStack v2.3 connects the management of resources, instances, caches, and traffic, allowing teams to focus more on model and business optimization while reducing repetitive work in multi-node deployment and day-to-day operations.

A Layer of Systems Engineering Above the Inference Engine

Inference engines such as vLLM and SGLang continue to improve model execution efficiency while adding capabilities such as distributed inference, PD disaggregation, and routing. These advances form an important foundation for the model services delivered by GPUStack.

When enterprises adopt these capabilities, they often need to support multiple models, different inference backends, and heterogeneous GPUs at the same time. Beyond optimizing individual components, they must coordinate resource allocation, instance deployment, request routing, and configuration changes.

PD disaggregation, for example, requires teams to organize Prefill and Decode instances and determine the right ratio between them, route requests through the corresponding processing stages, and maintain coordination among components as replica counts change. Each of these factors affects whether the service can continue to operate reliably.

GPUStack incorporates this engineering work into the model service management workflow: users configure a service, and the platform organizes the underlying runtime components. Version 2.3 further connects deployment, benchmarking, and runtime management, making advanced inference architectures easier to put into production and continuously tune.

Automated PD Disaggregation Orchestration: Configuring Resources Separately for Each Inference Stage

Processing an LLM request primarily involves two stages: Prefill, which processes the input context, and Decode, which generates output tokens one at a time.

These stages have different workload characteristics, and their resource requirements vary with input length, output length, and request volume. Deploying them separately makes it possible to tune resources and replica ratios independently for different production workloads.

Version 2.3 introduces automated orchestration for PD-disaggregated deployments, supporting the independent deployment, scheduling, and scaling of Prefill and Decode. Users configure the resources and replica counts for both roles, while GPUStack coordinates instance deployment, request routing, and collaboration as replica counts change.

This allows teams to optimize input processing and output generation separately while reducing the effort required to maintain multi-instance configurations manually. After deployment, teams can also benchmark the service against actual hardware and request patterns to validate and adjust the P/D ratio.

Resources and replicas are configured separately for Prefill and Decode, while the platform organizes them into a single model service.

Shared KV Cache: Reducing Repeated Context Computation

System prompts, knowledge base materials, and multi-turn conversation histories are often repeated across requests. Reprocessing the same content on every instance results in redundant computation.

Version 2.3 introduces a shared KV cache pool that, with compatible models and configurations, enables cache reuse across replicas and nodes. Multiple instances can reuse previously computed context and reduce repetitive Prefill work.

GPUStack integrates shared caching solutions such as LMCache, Mooncake, and XSKY MeshFusion into model service management. Teams can configure caching at the service level and evaluate its effectiveness using cache hit rates and time to first token.

Cache reuse must also be considered alongside request distribution and instance load. When a service scales from one instance to multiple instances, the results of prior context computation become a reusable resource. Teams can assess the value of caching as they scale services and adjust routing.

Configuring a shared cache around a model service makes repeated context reusable across instances.

vGPU Partitioning: Allocating Resources to Match Workloads

Small models, development environments, and validation jobs do not necessarily need exclusive access to an entire physical GPU. Coarse-grained resource allocation can leave a portion of compute capacity idle for extended periods.

Version 2.3 introduces vGPU partitioning, supporting model and GPU instance deployment using both software-based and hardware-based resource partitioning. This allows different workloads to consume resources on demand. The available partitioning methods and isolation capabilities depend on the hardware and the corresponding solution.

Partitioning support for eight types of heterogeneous GPUs.

In Kubernetes environments, K8s workloads can directly request full, shared, or partitioned GPU resources provided by GPUStack Operator KDP.

This extends the resource services introduced in v2.2: the same compute management system supports both model inference and development or experimentation. From full GPUs to partitioned resources, teams can allocate compute according to workload requirements and give smaller workloads an appropriately sized resource profile. Instance monitoring then helps teams observe actual utilization and continue refining their configurations.

Both model deployments and GPU instances can use partitioned resources, allowing teams to choose an allocation method suited to each workload.

Intelligent Benchmarking and Scheduling: Aligning Service Capacity with Business Workloads

Once a model is running, teams still need to understand how many requests the current configuration can handle and what kind of response experience it can deliver.

A single throughput figure cannot adequately answer this question. As concurrency increases, both throughput and latency change. Differences in input and output lengths can also invalidate earlier test results.

Version 2.3 introduces intelligent performance benchmarking. A single test covers multiple concurrency levels and produces load curves and a recommended operating point. This helps teams explore the model's performance limits and determine service capacity based on the latency their applications can tolerate.

Use throughput and latency across multiple load levels to select an operating point that meets application response-time requirements.

With capacity data in hand, teams can then plan traffic distribution and resource provisioning.

Version 2.3 enhances gateway routing with load-balancing strategies such as session affinity and least requests, as well as model selection through a decision service. The former manages request distribution across instances, while the latter selects an appropriate model for each request, enabling teams to configure traffic strategies based on service characteristics.

For predictable peaks and troughs, the platform also provides time-based autoscaling with tidal scheduling. It adjusts model replica counts according to configured time windows, adding instances before recurring traffic peaks and reducing resource consumption during off-peak periods.

Adjust replica counts by time window to provision resources ahead of predictable traffic peaks.

For example, a long-context knowledge base service can use shared caching to reuse repeated materials, run benchmarks to identify an operating point that meets response-time requirements, and then configure routing and a pre-peak scale-out plan. Every step revolves around the same service, and the configuration and observations from one step inform adjustments in the next.

From “How many requests can the service handle?” to “How should requests be distributed, and when should resources be provisioned?”, performance data becomes the basis for day-to-day scheduling.

OTA and Deployment Management: Keeping Pace with Technology While Preserving a Recovery Path

Models and inference backends evolve rapidly, and production services need a controlled way to adopt these changes.

Version 2.3 introduces OTA updates for the model catalog and inference backends. Without upgrading or restarting GPUStack, teams can update the model catalog, available inference backend versions, and the community backend marketplace online, gaining timely access to new deployment options.

For individual services, teams can use YAML import and export to save and reuse deployment configurations, apply validated configurations to subsequent deployments and environment migrations, and use deployment version history and rollback to retain a record of configuration changes and restore a previous version when necessary.

Reuse configurations through YAML, trace changes through version history, and restore a specific configuration version when necessary.

OTA updates bring new capabilities to the platform promptly, while configuration reuse and version management allow teams to validate and adopt them at their own pace. From introducing new deployment options to reusing validated configurations and restoring previous versions, service updates now have a clearer operational path.

Turning Advanced Inference Capabilities into Model Services Built for Sustainable Operations

The value of an inference optimization is ultimately realized in a production service: are resources allocated appropriately, is the deployment architecture maintainable, has capacity been validated, and are application requests handled as expected?

GPUStack v2.3 connects PD disaggregation, shared caching, resource partitioning, and elastic scheduling through a unified service management workflow. Teams can choose an inference architecture based on application needs, validate its actual capacity, and continuously adjust resource and traffic configurations.

This is GPUStack's approach to systematic inference engineering: turning advanced inference capabilities into model services that can be delivered at scale and operated sustainably.

Try GPUStack v2.3 with a real-world workload and explore the deployment and operating model that best fits your needs. We also invite you to share your practical experience with the community.

Can GPUStack Double GLM Inference Performance on 4 DGX Spark Nodes? From 24.3 to 51.3 tok/s

GPUStack — Thu, 22 Oct 2026 07:28:12 +0000

As GPU prices continue to rise, hardware that once seemed difficult to justify deserves another look. NVIDIA DGX Spark is one such system. Its price-to-performance positioning initially made us hesitant, but under today's procurement constraints, a small inference cluster built from several Spark systems is worth evaluating seriously.

We connected four DGX Spark systems, managed them as one cluster with GPUStack, and served GLM-5.3-Flash-NVFP4 through vLLM. After two days of network tuning, MTP speculative-decoding work, and multi-node troubleshooting, we obtained the following results.

Under our test conditions, single-stream decode throughput increased from 24.3 to 51.3 tok/s, or 2.11× the baseline. Aggregate output throughput at four concurrent requests increased from 72.0 to 104.1 tok/s.

This article focuses on three questions: what network tuning actually improved, why MTP initially failed to start, and which GPUStack multi-node deployment issues are worth avoiding in advance.

All numbers in this article come from this specific four-node cluster. The performance results apply to the model, configuration, and test cases described here. Interface names, paths, and parameters must be adapted to your own environment.

1. Test Environment: Four Spark Nodes with Cross-Node TP=4

Hardware and Network

Item	Configuration
Compute nodes	4 × NVIDIA DGX Spark with the GB10 Grace Blackwell Superchip
Unified memory	128 GB per node
Network adapter	ConnectX-7; the logical interfaces used in this deployment are listed in the appendix
Switch	MikroTik CRS804-4DDQ, four 200G access ports, RouterOS 7.19.6
Rail 1	`192.168.100.0/24`
Rail 2	`192.168.101.0/24`

One detail about the “dual-rail” wording matters: according to our deployment records, the two logical interfaces on each node shared one 200G physical link; they were not two independent 200G links. Each of the four nodes connected to one physical switch port.

The topology also had an important limitation: the switch did not expose usable DCB/PFC/ECN configuration for this setup, so our tuning was concentrated on the NIC side. We therefore describe the configuration and results for this topology without claiming that it formed an end-to-end lossless network.

Software and Model

Item	Configuration
Orchestration	GPUStack; Server on `node01`, with four Workers registered in one cluster
Inference engine	vLLM with the B12X attention backend
Container image	`eugr/spark-vllm-b12x:latest`
Model	GLM-5.3-Flash-NVFP4
Quantization and cache	NVFP4 weight quantization and FP8 KV cache
Parallelism	TP=4 across four nodes; PP=1

GPUStack handled model configuration, Worker scheduling, and instance management. The lower-level inference and communication work centered on vLLM, MTP, and NCCL.

2. Fixing the Network First: Making Better Use of RoCE with NCCL

Cross-node tensor parallelism makes inter-node communication unavoidable. We started with MTU consistency, NIC QoS, and NCCL interface selection.

2.1 Use a Consistent MTU

We set the host MTU to 9000 and persisted it through netplan. On the switch, we used the platform-specific setting mtu=9216; the recorded effective value was 9000.

The important part is to verify the effective configuration and connectivity across the entire path. Changing only the host configuration is not enough.

2.2 Configure NIC-Side QoS

The following commands show one interface from our deployment:

mlnx_qos -i enp1s0f0np0 --trust dscp
mlnx_qos -i enp1s0f0np0 -f 0,0,0,1,0,0,0,0

This configuration used DSCP trust mode and enabled PFC for priority 3.

One easy-to-miss detail caused confusion during early testing: mlnx_qos -p configures priority mapping, while -f controls PFC. Always read the configuration back after applying it and confirm that the effective state matches the intended one.

2.3 Let NCCL Use Both Logical RDMA Interfaces

NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_IB_TC=104
NCCL_IB_QPS_PER_CONNECTION=4
NCCL_IB_SPLIT_DATA_ON_QPS=1

In this setup, NCCL_IB_TC=104 corresponded to DSCP 26. Verify that it maps to the intended priority and queue under your NIC QoS configuration.

What Did the Network Tuning Improve?

In our recorded four-node, 256 MB AllReduce test, bandwidth increased from 4.0 GB/s under TCP fallback to 22.9 GB/s, or approximately 5.7×. The single-rail comparison measured 11.4 GB/s, while the dual-rail configuration reached approximately 2.0× that result.

These are bandwidth values from our NCCL test records. They should not be interpreted as physical port line rate.

Time to first token (TTFT) also improved substantially in the long-input workload, as shown in Section 6. Network tuning did not double every metric, however. In our records, its direct effect on decode throughput was only about 2%–5%.

Network tuning removed the communication bottleneck. The much larger improvement in single-stream decoding came after MTP was working.

3. Getting MTP Working: The Key to Doubling Single-Stream Decode Throughput

GLM-5.3-Flash includes an MTP, or Multi-Token Prediction, module. We attempted to use it for speculative decoding, but this environment exposed two separate problems: weight loading and KV-cache layout compatibility.

3.1 MTP Weights Were Loaded with the Wrong Quantization Format

In this checkpoint, the MTP layer was layer 45, and its name did not explicitly contain mtp. Its expert weights were actually BF16, but the quantization configuration's ignore list did not exclude the layer. The draft side therefore allocated it as NVFP4, resulting in a 256/512 dimension mismatch.

We fixed the issue by adding quantization-exemption rules to the model configuration on all four nodes.

First, back up the configuration file:

cd /models/GLM-5.3-Flash-NVFP4
cp -p config.json config.json.bak-ignore45

Then append the following four strings to both of these arrays:

quantization_config.ignore
text_config.quantization_config.ignore

[
  "model.language_model.layers.45",
  "model.language_model.layers.45*",
  "model.layers.45",
  "model.layers.45*"
]

These are entries to append. Do not replace the existing ignore arrays with this snippet. Layer 45 is specific to the checkpoint used in our test; verify the correct layer for other weight versions.

After updating all four nodes, we deleted the corresponding model instance in GPUStack to trigger instance recreation and weight reloading. This interrupts the instance, so schedule it during an acceptable maintenance window.

After the change, the MTP draft path loaded the layer as BF16 and the dimension mismatch disappeared. To roll back, restore the backup on every node and recreate the instance.

3.2 The KV-Cache Layouts Were Incompatible

Once the weights loaded, a second problem appeared between the FlashInfer backend used by the main model and the KV-cache layout used by the MTP draft path.

The layout constraints conflicted. Some layouts could not split a 2,560-token management block into 256-token kernel blocks, while layouts that could be split were rejected by the parser. Initialization could not proceed.

We initially built a locally patched image to relax the layout declaration. The configuration that ultimately worked was simpler: use B12X as the attention backend for both the main model and the draft path. With this configuration, the management and kernel blocks no longer required the problematic split.

--attention-backend=B12X
--speculative-config={"method":"mtp","num_speculative_tokens":3,"moe_backend":"humming","attention_backend":"B12X"}

We then repeated the test with a base image that did not contain our local layout patch. The same configuration ran successfully and showed no meaningful performance difference in our records, so we removed the patch.

The lesson was straightforward: when a backend compatibility issue appears, first check the support boundaries of the available backends before committing to a local patch.

3.3 Should the Speculative Depth Be 3 or 5?

Once MTP was running, we still had to decide how many tokens to predict per step.

The reference recipe used num_speculative_tokens=5. In our four-node TP=4 environment, we compared depths 3 and 5. Each configuration ran three times, and the table reports the median:

Test case	Depth 3	Depth 5	Relative change at depth 5
Single-stream, long-output decoding	51.9 tok/s	48.7 tok/s	−6.2%
Four-request aggregate output	110.0 tok/s	96.7 tok/s	−12.1%

The result was clear: depth 3 was faster in this test.

Our working hypothesis is that the additional computation from deeper speculation was not offset by an increase in accepted tokens. Confirming that explanation would require per-layer acceptance-rate and timing data; throughput alone is not enough to prove the cause.

We therefore selected depth 3. A reference recipe provides a starting point, but workload-specific A/B testing should determine the final setting.

Performance with MTP Enabled

The following results come from the same benchmark suite. They compare the no-MTP baseline with the combined MTP × 3 and dual-rail configuration:

Test case	No-MTP baseline	MTP × 3 + dual rail	Final/baseline
Single-stream, short-output decoding	24.4 tok/s	50.0 tok/s	2.05×
Single-stream, long-output decoding	24.3 tok/s	51.3 tok/s	2.11×
Four-request aggregate output	72.0 tok/s	104.1 tok/s	1.45×
Long-input decode throughput	24.4 tok/s	51.9 tok/s	2.13×

These numbers represent the benefit of the combined configuration. The entire gain should not be attributed to one parameter. The speculative-depth A/B test above came from a different run, which explains the small differences in absolute values.

For functional regression, all 17 test prompts passed, including character-counting cases. This provides a basic functional check, but it is not equivalent to a full output-distribution equivalence study.

4. A Silent Startup Hang: Gloo Selected the Loopback Address

One failure was particularly difficult to diagnose from the dashboard. All four vLLM rank processes stopped during initialization, GPU utilization remained at 0%, and the logs contained no explicit error.

We captured the call stacks with py-spy and found the processes waiting in:

gloo::transport::tcp::Pair::waitUntilConnected

The relevant Worker processes were listening only on 127.0.0.1.

Further investigation showed that /etc/hosts inside the instance container mapped the hostname to a loopback address. When Gloo selected an address from the hostname, it attempted to use that loopback address for cross-node communication, preventing initialization from completing.

The fix was to explicitly select the network interface for Gloo:

GLOO_SOCKET_IFNAME=enp1s0f0np0

Setting only NCCL_SOCKET_IFNAME did not resolve this failure. GLOO_SOCKET_IFNAME had to be configured separately.

We retained this variable in the final cluster configuration. For similar initialization hangs, check the call stacks, listening addresses, and hostname resolution inside the container together.

5. Six Operational Lessons from a Multi-Node GPUStack Deployment

GPUStack placed all four nodes under one configuration and instance-management workflow, which made it easier to repeat tests across different combinations. The following operational lessons were worth keeping.

This section describes observations from our deployment. API paths, log locations, and container behavior may vary by GPUStack version.

5.1 Read the Configuration Back After an Update

We used three main API operations:

GET    /v2/models/<model_id>
PUT    /v2/models/<model_id>
DELETE /v2/model-instances/<instance_id>

They read model configuration, update it, and delete an instance to trigger recreation. The paths are illustrative; actual calls also require the appropriate authentication and request body.

A useful habit is to run GET after PUT and verify that every critical field was saved as intended.

5.2 Verify the Effective Custom Image Separately

In our deployment, we repeatedly observed image_name becoming None after a PUT, after which follower nodes used an unexpected image.

We handled this by reading the configuration back, restoring the field when necessary, resubmitting it, and verifying the image actually running on every node. This is a troubleshooting record from our environment, not a claim that every GPUStack version behaves the same way.

5.3 Use Local Logs to Track Follower Progress

In this deployment, the Web dashboard logs mainly reflected the leader instance. When a follower remained in “Preparing model files,” we inspected the log inside its Worker container:

docker exec gpustack-worker cat /var/lib/gpustack/log/serve/INSTANCE_ID.0.log

Confirm the actual path and filename suffix for your instance. Restarting the affected Worker recovered one failure in our tests, but doing so affects services on that Worker. Preserve the logs and assess the impact first.

5.4 Scope Residual-Container Cleanup Precisely

After deleting an instance, old containers occasionally remained on follower nodes and affected the next follower health check. A new instance could also begin creating containers almost immediately, making broad cleanup filters dangerous.

We separated discovery from deletion: list candidate containers first, verify that they belong to the old instance, and only then delete the exact container ID.

# List candidates first. The name filter only helps locate them;
# verify the instance identity separately.
docker ps -a --filter 'name=<OLD_INSTANCE_UNIQUE_SUFFIX>' \
  --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}'

# After verification, replace CONTAINER_ID with the exact old container ID.
docker rm -f CONTAINER_ID

5.5 Verify the Container Runtime Requirements for RDMA

The instance containers we inspected ran in privileged mode, used host networking, and had access to /dev/infiniband, providing the runtime conditions needed for RoCE inside the container.

Those conditions alone do not prove that RDMA communication is working. We still used the container configuration, NCCL logs, and communication benchmarks together to verify the effective transport.

5.6 Use a Custom Image When the Backend Requires It

Under GPUStack's inference-backend configuration, we specified:

Image: eugr/spark-vllm-b12x:latest
Entry command: vllm serve

This provided the required B12X-enabled vLLM environment. We then managed the test combinations through model parameters and environment variables. As with every other configuration change, verify the image and launch arguments on the running instances.

6. Final Configuration and Results

We settled on the following configuration:

Four nodes with TP=4 and PP=1.
B12X as the attention backend for both the main model and MTP draft path.
MTP speculative depth 3.
Two logical RDMA interfaces for NCCL and an explicitly selected Gloo interface.
gpu-memory-utilization=0.85 and max-model-len=500000.

We did not retain kv_cache_memory_bytes=8G from the reference configuration, nor did we raise gpu-memory-utilization to 0.87. We preferred additional operating headroom over maximizing allocated memory.

Performance Results

Application-level results are the median of three bench.py runs. NCCL bandwidth is listed separately so that communication bandwidth is not confused with model output throughput.

Application metric	Baseline	Final result
Single-stream, long-output decoding	24.3 tok/s	51.3 tok/s, 2.11×
Four-request aggregate output	72.0 tok/s	104.1 tok/s, 1.45×
TTFT in the long-input case	1.73 s	0.12 s, approximately 93% lower

The TTFT result applies to this particular test case. Because the article does not fully specify input length, cache-hit state, and warm-up conditions, it should not be extrapolated to arbitrary long-input requests or attributed to one network parameter alone.

Communication metric	Baseline	After tuning
Recorded NCCL AllReduce bandwidth	4.0 GB/s	Approximately 19–23 GB/s

We also raised max-model-len from the previously recorded 32K tokens to 500,000 tokens. This is a configuration limit, not evidence that we completed end-to-end validation of one 500K-token request or four concurrent 500K-token requests.

Restart Stability

Three scripted restart tests all succeeded on the first attempt. Time to readiness was approximately 12.5–13.5 minutes, including about five minutes for weight loading, seven minutes for B12X kernel tuning and related initialization, and the remaining time for other startup steps.

These three runs were sufficient for our configuration sign-off, but longer-duration stability still requires continued observation.

Final Takeaways: Give Every Important Choice a Comparison

Two lessons stood out after the two-day tuning effort.

First, select a backend combination that is actually compatible. B12X resolved the MTP layout issue in our environment and allowed us to remove the local patch used during troubleshooting.

Second, use controlled comparisons to choose parameters. Single rail versus dual rail, speculative depth 3 versus 5, and no MTP versus the final combined configuration answered different questions. Testing them separately made it easier to identify where each gain came from.

GPUStack gave us one multi-node management plane for the experiment, while the final performance came from the network, inference backend, and model configuration working together. If you are experimenting with your own Spark cluster, we hope this record provides a useful troubleshooting path.

Share your hardware configuration, benchmark method, or GPUStack multi-node deployment experience in the comments.

Appendix: Deployment Configuration

The values below reproduce our environment. Verify your GPUStack and model versions, checkpoint layout, interface names, and available memory before using them.

Basic Model Configuration

Setting	Value used in this test
Model source	`local_path`
Local path	`/models/GLM-5.3-Flash-NVFP4`
Distributed inference	Cross-Worker distributed inference enabled
Backend version label	vLLM (`0.31.0-spark-custom`, preserved from the deployment record)
Image	`eugr/spark-vllm-b12x:latest`
Replicas	1

`backend_parameters`

Each line below is one backend parameter. Pass the JSON value for --speculative-config as one complete argument.

--distributed-executor-backend=mp
--tensor-parallel-size=4
--trust-remote-code
--max-model-len=500000
--gpu-memory-utilization=0.85
--max-num-seqs=4
--max-num-batched-tokens=8192
--kv-cache-dtype=fp8
--block-size=256
--mamba-cache-mode=align
--no-enable-flashinfer-autotune
--enable-auto-tool-choice
--tool-call-parser=glm47
--reasoning-parser=glm47
--attention-backend=B12X
--speculative-config={"method":"mtp","num_speculative_tokens":3,"moe_backend":"humming","attention_backend":"B12X"}

Environment Variables

Enter these as separate key-value pairs in GPUStack:

VLLM_USE_AOT_COMPILE=1
VLLM_USE_MEGA_AOT_ARTIFACT=1
VLLM_USE_V2_MODEL_RUNNER=1
CUTE_DSL_ARCH=sm_121a
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
VLLM_SSM_CONV_STATE_LAYOUT=DS
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_IB_TC=104
NCCL_IB_QPS_PER_CONNECTION=4
NCCL_IB_SPLIT_DATA_ON_QPS=1
NCCL_SOCKET_IFNAME=enp1s0f0np0
GLOO_SOCKET_IFNAME=enp1s0f0np0

We additionally enabled these variables during troubleshooting:

NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS=INIT,NET

Pre-Launch Checklist

Apply the layer 45 quantization exemptions required by this checkpoint on all four nodes, and retain a backup of the original configuration.
Read back and verify the effective MTU, DSCP trust, priority mapping, and NIC-side PFC settings. Confirm that persistent configuration matches runtime state.
Confirm that the RDMA and Gloo interface names match the actual devices on every node.
This configuration does not pin NCCL_IB_GID_INDEX; NCCL selects it automatically. Check the selected value in the startup logs.
After recreating the instance, verify the effective image and parameters on every node, then run communication, health, and real-inference checks.

References

DeepInfer vs. vLLM on 8 B200: Which Is Faster?

GPUStack — Thu, 22 Oct 2026 07:21:49 +0000

TL;DR: We integrated DeepInfer with GPUStack through a lightweight serving project, deployed DeepSeek-V4.1-Flash on a single server with eight NVIDIA B200 GPUs, and compared the resulting service with vLLM under a matched DP8+EP8 setup. In this run, DeepInfer-server delivered higher reported output throughput and shorter inter-token latency, while vLLM achieved lower time to first token and better P95 request latency.

Efficient inference technology creates business value only when it can be deployed in a real service.

DeepSeek's open-source DeepInfer gives developers a way to explore its inference technology. Teams that want to run models on their own infrastructure still need to answer several engineering questions: How do we expose an API? How should the runtime be configured? How do we manage GPUs? And how does the engine perform under a real workload?

To answer those questions, we built a lightweight serving project called DeepInfer-server and integrated it with GPUStack's custom inference backend. The result was a working text-generation service for DeepSeek-V4.1-Flash on a single server with eight NVIDIA B200 GPUs.

This article explains the integration, walks through the deployment process, and describes how we compared it with vLLM on the same hardware.

1. What Does DeepInfer Bring?

DeepInfer is DeepSeek's open-source high-performance inference framework. The current public release provides offline inference and the core engine capabilities required for large-scale serving. It supports DeepSeek-V4.1-Flash and integrates computation and communication components including FlashMLA, DeepGEMM, DeepEP, and DeepSelect. It also supports multiple parallelism strategies, speculative decoding, and asynchronous scheduling. See the DeepInfer repository.

These components optimize different parts of the inference path: computation executes faster, data moves between GPUs more efficiently, and scheduling makes better use of the hardware. What users ultimately care about is how many requests the same resources can serve and how long users have to wait.

Turning an engine into a production-facing service, however, also requires request handling, protocol conversion, an HTTP service, and deployment management.

The components in this integration have the following responsibilities:

Component	Role in this integration
DeepInfer	Runs model inference and manages low-level computation, communication, and parallel execution
deepseek-recipe	Converts chat requests into model input and parses generated text, reasoning content, and tool calls
DeepInfer-server	Connects the components and provides an HTTP API, streaming responses, and service lifecycle management
GPUStack	Manages model deployments, GPU allocation, runtime configuration, and model access and testing

deepseek-recipe is DeepSeek's protocol-conversion and codec component. It translates between API requests, conversation formats, model token sequences, and responses. Model inference and HTTP transport are supplied by the caller. See the deepseek-recipe repository.

Why Build DeepInfer-server?

DeepInfer's current public entry point is primarily designed for offline inference. To start it like vllm serve and make it callable by applications, the engine must be connected to an HTTP request-processing path. deepseek-recipe handles protocol conversion and output parsing, but the caller must still supply model inference and HTTP transport. Installing both components in the same container does not automatically create a complete online inference service.

A GPUStack custom-backend definition tells the platform which image and command to run and which arguments to pass. It does not automatically generate an API around an offline engine. This integration therefore needs a connecting layer that organizes the components into a service that can start, report its state, and receive requests.

DeepInfer-server provides that serving layer:

Connect requests to inference. It receives Chat Completions requests, uses deepseek-recipe to convert them into model input, and calls DeepInfer to generate output.
Return output to applications. It converts generated tokens into text, reasoning content, or tool calls; returns standard JSON or SSE streaming responses; and reports token usage.
Manage the service lifecycle. It initializes inference processes, handles request dispatch, cancellation, timeouts, and errors, and exposes model-list and health-check endpoints.
Integrate with the deployment platform. It provides a container image and common startup arguments so that GPUStack can pass the model path, port, and GPU configuration and then start and manage the service.

Applications can therefore call the model through the familiar OpenAI-compatible Chat Completions API, while GPUStack can reuse its custom-backend deployment workflow. DeepInfer still performs the underlying inference, and deepseek-recipe still handles protocol conversion and parsing.

The current wrapper focuses on single-node, multi-GPU text inference. It is an independent demonstration project for validating the integration. It does not reproduce DeepSeek's complete production architecture for KV-cache management, disaggregated prefill and decode, or load balancing, and it should not be treated as equivalent to DeepSeek's official online service.

2. Deploying DeepInfer Through a GPUStack Custom Backend

GPUStack's custom inference backend mechanism lets users define a container image, startup command, default parameters, and environment variables so that a new inference service can participate in the normal deployment workflow. Template variables can supply values such as the model path and service port at deployment time. See the GPUStack custom backend documentation.

This allows teams to evaluate new inference engines on the same platform while reusing existing model-management and deployment operations.

The integration includes the following backend definition:

gpustack-deepinfer-server-backend.yaml

Step 1: Prepare the Runtime Environment

Our demonstration used a single server with eight NVIDIA B200 GPUs and the following model:

deepseek-ai/DeepSeek-V4.1-Flash-1017

Before deployment, prepare the complete model weights, the DeepInfer-server image, a compatible GPU driver and container runtime, and sufficient host memory.

The configuration must also expose the correct RDMA devices and set up the communication environment. The NIC and network parameters in the accompanying YAML come from our validation environment and must be adapted to other servers.

Step 2: Import the Custom Backend

On GPUStack's inference-backend management page, create a custom backend and import the DeepInfer backend YAML definition.

The definition contains the service startup method, common arguments, health-check path, and JIT-cache directory. At deployment time, GPUStack supplies the model path, service port, model name, and number of allocated GPUs.

Step 3: Create the Model Deployment

When creating the model deployment, select the model weights and choose the DeepInfer-server inference backend created in the previous step.

To keep the deployment aligned with the benchmark described later, we explicitly set --expert-parallel-size=8, --data-parallel-size=8, and --context-parallel-size=1 in the model's backend parameters. These values override the DP=1 default in the supplied YAML. Verify the final startup configuration before launching the service.

Setting	Example value
GPU count	8 GPUs in one server
Parallel configuration	EP=8, DP=8, CP=1
Maximum context length	32,768 tokens
Maximum running sequences	16 per DP rank
Batched-token budget	32,768 per DP rank
GPU memory utilization	Default 0.85
Startup timeout	3,600 seconds

EP, DP, and CP refer to expert, data, and context parallelism. The current wrapper requires DP × CP = EP. We use CP=1 here, so the context is not split across GPUs. max-num-seqs=16 is the limit for each DP rank, giving the DP8 service a total admission limit of 128 sequences.

The first startup loads the weights and may compile and warm up kernels. Use GPUStack to monitor the startup logs and wait for model initialization to finish before running inference tests. A persistent JIT cache helps reuse compiled artifacts.

Step 4: Verify Inference

After the model is ready, send a text request from the GPUStack Playground to verify streaming output. The same service can then be integrated with existing applications through its API.

At this point, the full path from model weights and inference engine to API access is working. Teams can continue testing real prompts, monitoring resource utilization, and evaluating the value of further optimization.

3. How Does It Compare with vLLM on the Same Eight B200 GPUs?

Before looking at the numbers, we need to define the scope of this comparison.

This test is a service-integration and preliminary performance validation. Neither deployment was fully tuned for the hardware, and both differ from DeepSeek's production inference architecture. Because the public DeepInfer release does not expose a TP path corresponding to vLLM's tensor parallelism, we configured DeepInfer with CP1, DP8, and EP8 and vLLM with TP1, DP8, and EP8. This makes the parallel strategies more comparable, but does not represent the best possible configuration for either engine.

DeepInfer's documented parallelism includes Attention DP or balanced CP and MoE EP with EPLB. Our single-node combined service does not reproduce its production disaggregated prefill/decode architecture, complete KV-cache management, or load balancing. See the DeepInfer documentation.

Even with DP8+EP8 on both sides, their kernels, communication paths, Engram access, and schedulers are implemented differently. DeepInfer-server also adds request coordination and protocol processing written for this demonstration, and that overhead is included in the end-to-end results. These numbers do not represent the performance of DeepSeek's official service and should not be used to infer the performance ceiling of either engine.

First, the Interactive Chat Experience

After deployment, we started with a single-request chat in the GPUStack Playground. In the screenshots, the vLLM deployment produced approximately 91.4 tokens/s, while DeepInfer produced approximately 106.2 tokens/s in that interaction.

A single chat result depends on the prompt, context, answer length, and the UI's measurement method. This screenshot illustrates the interactive experience; it is not a single-request performance ranking and should not be compared directly with the concurrent-output throughput below. For the benchmark, both engines used DP8+EP8, an 8K input, a 1K output, and 16 concurrent requests.

Test Environment and Configuration

We used the same eight-B200 server and the same model weights. GPUStack's Benchmark feature issued 100 requests with a target concurrency of 16.

Item	DeepInfer-server	vLLM
Hardware	8 × NVIDIA B200 on the same Worker	8 × NVIDIA B200 on the same Worker
Model	DeepSeek-V4.1-Flash-1017	DeepSeek-V4.1-Flash-1017
Version	DeepInfer commit `9bd9cf2`; service image tag `latest`	0.32.0
Parallel strategy	DP8, CP1, EP8	DP8, TP1, EP8
Maximum context length	32,768 tokens	32,768 tokens
`max-num-seqs`	16 per DP rank	16 per DP rank
`max-num-batched-tokens`	32,768 per DP rank	32,768 per DP rank
GPU memory utilization	0.85	0.85
Speculative decoding	Disabled	Disabled
Dataset	Random	Random
Target input/output length	8,192 / 1,024 tokens	8,192 / 1,024 tokens
Target client concurrency	16	16
Number of requests	100	100

Raw Results in GPUStack

The following table places the main metrics side by side:

Metric	DeepInfer-server: DP8+EP8	vLLM: DP8+EP8
Successful/total requests	100 / 100	100 / 100
Mean output throughput (tokens/s)	1,392.44	1,112.20
Mean time to first token, TTFT (s)	4.28	2.04
P95 time to first token, TTFT (s)	16.33	8.27
Mean inter-token latency, ITL (ms)	8.74	11.98
P95 inter-token latency, ITL (ms)	8.93	13.70
Mean end-to-end request latency (s)	13.23	14.30
P95 end-to-end request latency (s)	25.46	20.03
Actual mean concurrency	14.72	14.58

Output throughput, TTFT, ITL, and request latency come from the exported report's output_tokens_per_second_mean, time_to_first_token_*, inter_token_latency_*, and request_latency_* fields. The table converts TTFT from milliseconds to seconds. Output throughput retains the report's mean statistic and is not converted into whole-run duration. ITL is not mixed with the report's time_per_output_token or per-SSE-chunk intervals.

4. What Do These Results Tell Us?

In these two DP8+EP8 runs, DeepInfer-server reported higher mean output throughput and shorter mean inter-token latency, while vLLM delivered a shorter time to first token and lower tail latency.

DeepInfer-server's reported mean output throughput was approximately 25.2% higher than vLLM's corresponding run. Its mean ITL was approximately 27.0% lower, and its mean end-to-end request latency was approximately 7.5% lower. These are differences between two specific test runs; they are not yet proof of a stable performance advantage.

At the same time, DeepInfer-server's mean TTFT was 4.28 seconds, about 2.09× vLLM's 2.04 seconds. Its P95 request latency was 25.46 seconds, approximately 27.1% higher than vLLM's 20.03 seconds. The wait before an answer begins and the experience of the slowest requests both leave room for improvement.

The result demonstrates why an inference service should be evaluated across throughput, time to first token, sustained generation speed, and tail latency. Interactive applications benefit from beginning a response quickly, while batch-generation workloads also care about how much output can be completed per unit of time. No single metric fully describes the service experience.

Both DP8+EP8 configurations completed all 100 requests, validating service availability under this test load. One hundred successful requests are not equivalent to production stability certification, and the Random dataset does not evaluate answer quality.

For GPUStack, the experiment demonstrates a reusable workflow: integrate a new engine through a custom backend, deploy and test it on one platform, and use measured data to choose the runtime that fits the workload. We plan to continue testing fixed prompts and controlled conditions at different concurrency levels, and to separate the overhead of the serving wrapper from engine execution.

5. Continuing the Work to Make Efficient Inference Easier to Use

DeepInfer's open-source release gives the community more inference-optimization techniques that can be studied and validated.

In its deployment Q&A, DeepSeek explains that its KV-cache management and load-balancing components are tightly coupled to internal infrastructure and have not yet been fully open sourced. DeepSeek also states that inference code and performance optimizations for future DeepSeek models will continue to be released through DeepInfer. See the DeepSeek inference deployment Q&A.

This creates room for future integration work. The timing and scope of any additional open-source components should still be based on DeepSeek's official announcements.

GPUStack will continue following DeepInfer and the related computation, communication, and codec components. As components become available and verifiable, we will evaluate further integration through real deployments and tests.

Our goal is to help new inference technology reach developers and enterprises faster: lower the barrier to experimentation, make performance gains reproducible, make infrastructure cost measurable, and enable more teams to run large models effectively and affordably.

Making low-cost inference broadly accessible requires sustained technical optimization—and it also requires making those optimizations easier to deploy and use.

GPUStack: github.com/gpustack/gpustack

DeepInfer-server: github.com/gpustack/DeepInfer-server

GPUStack Jev: Decision Routing — Mechanism, Configuration, and Validation

GPUStack — Mon, 12 Oct 2026 09:36:06 +0000

GPUStack v2.3.0rc2 introduced a preview of an advanced routing capability worth testing hands-on: selecting a model through the Jev decision service.

When multiple models are available in GPUStack, an application would normally need to specify which one to call. With decision routing enabled, the application can always use the same entry point. The gateway then chooses a generation model based on the task content and the policy defined by the administrator.

This article follows that path end to end. We first introduce decision models, then deploy two private models in GPUStack, connect DeepSeek and Jev, and finally use tasks of different difficulty levels to verify the resulting model selections.

01 What Is a Decision Model?

Generation models answer questions. Decision models provide evidence for choosing a model.

For example, “Translate Good morning into Chinese” and “Design a cross-region payment system” require different capabilities. A smaller model may be sufficient for the first task, while the second may require a stronger model to reason about failures and consistency.

Jev receives the request content, candidate models, and their capability descriptions. It returns structured selection evidence through the TypeSafe Choice interface. The GPUStack gateway uses that evidence to choose a model and then forwards the request to the selected generation service. The final answer still comes from Qwen3 or DeepSeek.

The key to configuring decision routing is therefore to describe clearly which tasks each candidate model is suited for. A decision model does not establish business-quality standards on behalf of the administrator. The closer the capability descriptions are to real workloads, the easier the selections are to validate.

02 What Problem Does Jev Solve in the Gateway?

GPUStack model routing can place multiple targets behind one entry point. Fixed weights answer the question “How much traffic should each model receive?” and are useful for canary releases and traffic distribution. Jev decision routing goes one step further and asks “Which model is appropriate for this request?”

For an assistant that handles both text and code, we want the following division of work:

Task type	Candidate in this policy
Short translation, field extraction, factual questions, and basic arithmetic	Private Qwen3-8B
Constrained algorithm implementation, SQL, and localized code debugging	Private Qwen3-32B
Complex concurrent code and cross-region system design	External DeepSeek

The application uses one route name, while the administrator maintains this division of work in GPUStack. Candidates can be added or task boundaries adjusted without changing how the application calls the route.

This is the gateway-level value of a decision model: centralize task classification in the routing policy so that different models handle the requests best suited to them. Whether this improves quality or reduces cost still has to be validated against real workloads. This article first checks whether task routing follows the configured policy.

03 Prepare the Models and External Services

First, prepare the three candidates that will generate answers. Then connect Jev as the service responsible for making the decision.

Deploy Qwen3-8B and Qwen3-32B-W8A8 through GPUStack. Both models should show one available replica out of one. Send an inference request to each model to confirm that both can answer successfully.

Next, add DeepSeek under Providers. Select the DeepSeek type, enter the API key, and test connectivity. The generation model used in this test is deepseek-flash.

Add Jev as another provider. Select TypeSafe Decision Service (Jev), set the endpoint to https://api.typesafe.ai, and enter the TypeSafe API key. This test uses jev-latest as the decision model.

At this point, the responsibilities are separate. Qwen3 and DeepSeek are route targets and generate answers. Jev is the decision service and provides evidence that helps the gateway choose a target.

04 Configure a Decision Route

Create a route named demo-smart and add the two private deployments and the DeepSeek model as regular route targets. The application only needs to request demo-smart.

Under Basic Information, select Policy, enable Decision Service Routing, choose the Jev provider routing-jev, and enter jev-latest as the decision model.

In this test, all three ordinary traffic weights are set to 0, while the decision contribution weight is set to 20 so that the effect of Jev is easy to observe. The decision contribution weight is a relative influence on the score. It is not a traffic percentage. Session stickiness and the least-outstanding-requests strategy are not combined with it in this test.

Next, define the Decision Instructions and Model Evaluation Criteria. Together, they answer two questions: What is the overall selection principle, and what tasks is each candidate suited for?

Setting	Core content used in this test
Decision instructions	Prefer the least expensive candidate that can complete the task. Send simple tasks to 8B, medium-complexity tasks to 32B, and complex tasks to DeepSeek.
8B capability description	Suitable for short translation, extraction, factual questions, short summaries, and basic arithmetic. The service context budget is 8,192 tokens.
32B capability description	Suitable for SQL, algorithm implementation, and localized debugging. The service context budget is 16,384 tokens.
DeepSeek capability description	Suitable for complex reasoning, concurrent code with tests, and distributed architecture design.

The keys in the capability descriptions must match the actual candidate model names: route-local-qwen3-8b, route-local-qwen3-32b, and deepseek-flash. Do not use route aliases or provider names because they cannot be mapped to the candidates.

This capability split is a policy to be validated. The assumption that 8B is the lower-cost tier, 32B is the middle tier, and DeepSeek is the higher tier is also part of the configuration. This test does not calculate hardware amortization or API charges.

05 Validate Three Model Choices Through One Route

Every test in this section uses the same route, demo-smart. The nine test inputs are nine questions, not nine routes. Only the question content changes. The model field remains unchanged while we observe whether the gateway follows the configured division of work and selects 8B, 32B, or DeepSeek.

Step 1: Write Down the Expected Results

Start with three representative inputs. A short translation is expected to go to 8B, constrained interval-merging code to 32B, and cross-region payment-system design to DeepSeek. Recording the expectations first prevents us from explaining the choice only after seeing the answer.

Select demo-smart in the GPUStack Playground to confirm that the route can complete a conversation.

Step 2: Send the Request and Retain the Headers and Answer

Use the API for the formal test so that the selected model can be traced. First, create a GPUStack inference key and save the following content as request.json:

{
  "model": "demo-smart",
  "messages": [
    {
      "role": "user",
      "content": "Translate ‘Good morning’ into Chinese. Return only the translation."
    }
  ],
  "max_tokens": 128,
  "temperature": 0,
  "stream": false,
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "thinking": {
    "type": "disabled"
  }
}

Send the request to the gateway, saving the response headers to response.headers and the answer to response.json:

curl -sS http://localhost:9080/v1/chat/completions \
  -D response.headers \
  -o response.json \
  -H "Authorization: Bearer $GPUSTACK_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @request.json

For the next test, change only the question in messages and increase max_tokens for code or longer answers. Keep model set to demo-smart. The interval-merging test uses a 1,024-token output budget, and the system-design test uses 1,536 tokens. Replace the gateway address with the address of your own deployment.

Step 3: Find the Actual Model by Request ID

Read x-gpustack-request-id from the response headers, then use it to locate the corresponding Higress log entry:

REQUEST_ID=$(awk 'tolower($1)=="x-gpustack-request-id:" {gsub("\r", "", $2); print $2}' response.headers)
test -n "$REQUEST_ID" || { echo 'response header does not contain a request ID'; exit 1; }

kubectl --context gateway-cluster \
  -n inference-gateway logs deployment/higress-gateway --tail=500 \
  | rg --fixed-strings "$REQUEST_ID"

Replace the context and namespace with your own gateway configuration. In the matching log entry, inspect upstream_cluster to identify the forwarding target and ai_log.model to confirm the model name. In this test, model-2-6.static maps to 8B, model-3-7.static maps to 32B, and provider-1.dns maps to DeepSeek. These identifiers change with the deployment.

The evidence is the request with one request ID, its response, and the matching access log. An HTTP 200 status only confirms that the request succeeded. It does not prove which model was selected.

Step 4: Compare the Measured Results

The next three figures show the request, response headers and body excerpt, and matching gateway log fields from the archived API test. Each figure retains the request ID. The entry point remains the same, while the model that generates the answer changes.

Short translation → 8B. The request asks for a short translation of “Good morning.” The model returned the expected translation, while the log records route-local-qwen3-8b and upstream model-2-6.static. The figure localizes the body excerpt into English for this edition while preserving the request ID and routing evidence.

Constrained code → 32B. The task asks for interval merging without modifying the input and requires invalid intervals to be rejected. The API returned Python code, while the log records route-local-qwen3-32b and upstream model-3-7.static.

Cross-region system design → DeepSeek. The task asks for an analysis of payment callbacks, inventory deductions, network partitions, and recovery reconciliation. The API returned a design explanation, while the log records deepseek-flash and upstream provider-1.dns.

In addition to these three representative inputs, we tested factual questions, extraction, arithmetic, SQL, and other tasks. All nine inputs went through demo-smart:

Test input	Expected model	Actual model
Capital of France	8B	8B
Short translation of Good morning	8B	8B
Extract a city from text	8B	8B
Calculate 2+2	8B	8B
Constrained interval-merging code	32B	32B
PostgreSQL aggregation query	32B	32B
Analyze and fix an asynchronous-lock deadlock	32B	32B
Cross-region order-payment system design	DeepSeek	DeepSeek
Asynchronous LRU cache with tests	DeepSeek	DeepSeek

All nine requests returned HTTP 200 with stop as the finish reason, and every actual model matched the expectation. This verifies that the configured task split can be implemented behind one entry point. The translation, interval-merging, and system-design tests were each repeated three times as non-streaming requests, and the model selection remained stable. Streaming versions of all three tasks also selected the same models.

Selecting the Right Model Is Not the Same as Accepting Its Answer

To determine whether this division of work had a concrete basis, we sent the same translation, extraction, arithmetic, and interval-merging tasks to all three candidates with identical input and output budgets.

The code task required correct interval merging, no mutation of the input, and a ValueError for an invalid interval. The generated code was executed in an isolated environment.

Acceptance check	8B	32B	DeepSeek
Short translation, city extraction, and 2+2	All correct	All correct	All correct
Merge results, empty input, and touching endpoints	Passed	Passed	Passed
Does not modify the original input	Failed	Passed	Passed
Raises `ValueError` for an invalid interval	Failed	Passed	Passed

The 8B answer modified the original input and omitted the invalid-interval check. The 32B and DeepSeek answers met both constraints. This provides specific evidence for routing constrained algorithm implementations to 32B, but it does not establish a general ranking of model capability.

The asynchronous-cache code generated by DeepSeek passed its three included tests, but the logs still contained an unhandled Future exception. Outputs from a stronger model still require independent validation.

What Happens When the Jev Decision Service Fails?

Model selection depends on Jev returning valid decision evidence. If Jev is unreachable, times out, or returns an invalid result, the decision plugin does not publish a ranking opinion. The gateway continues with the remaining selection and forwarding process. This is fail-open behavior.

In an isolated failure-test route, we configured an unreachable Jev endpoint while keeping all three generation candidates healthy. The log showed publishing no rank. DeepSeek happened to answer, and the final response was HTTP 200. This demonstrates that processing can continue after a decision failure. It does not mean that failures always route to DeepSeek.

Another input of approximately 79 KB did not succeed. Jev returned 400 and published no ranking opinion. The request then entered the 32B target, which also returned 400 because its context budget was insufficient. When no decision ranking is available, the gateway cannot guarantee selection of a model that meets the task requirements.

A separate translation prompt that instructed the system to ignore the policy and choose DeepSeek still went to 8B, but its answer did not satisfy the request. This again shows that model selection and answer quality must be evaluated separately.

06 Return to the Gateway and Follow the Complete Call Path

After the configuration and validation steps, the division of responsibilities between GPUStack and the gateway becomes easier to understand.

GPUStack manages candidate models, providers, and routing policies, then synchronizes the configuration to Higress. When a request reaches the gateway, plugins prepare the candidate information, the decision plugin calls Jev for selection evidence, and the gateway chooses an upstream target before forwarding the request to a private model or DeepSeek.

The Jev call and the generation call are two different paths. Jev participates in model selection. Qwen3 or DeepSeek generates the answer. If the decision call does not produce valid evidence, the request may continue, but selection of the expected model is no longer guaranteed.

A Deeper Look at How the Plugin Uses Jev Results

To understand how the gateway processes Jev’s evidence internally, read the next figure from top to bottom. The plugin prepares the candidates and task descriptions, calls Jev, converts the returned model probabilities into candidate scores, and lets the load-balancing finisher aggregate the scores and select an upstream target.

The diagram is based on commit 790256e of the decision-plugin source code. The probabilities and confidence value in the diagram illustrate the calculation and are not measured responses from the preceding tests. Confidence adjusts the influence of the decision opinion. It is not the probability that an answer is correct.

For the application, the entry point always remains demo-smart. For the administrator, the work is to maintain the capability boundaries of the candidate models and continuously validate selections against representative tasks. This is the value of decision routing: put the “which model should handle this request” decision into the gateway so that multi-model specialization becomes a configurable and testable policy.

This validation covered 36 requests: 9 distinct inputs through the same demo-smart route, 12 same-task baselines, 6 additional repetitions, 3 streaming requests, and 6 failure or policy-control cases. Routing and transport met their respective criteria, while answer quality was evaluated separately. The article focuses on the capability introduced in v2.3.0rc2 and uses a development environment with Server baseline f0c237f and UI baseline 5a4fc16. It does not include sustained-concurrency testing or a cost-benefit analysis.

References

Can an NVIDIA A800 80GB Run Qwen-Image-2.1? A GPUStack Deployment Guide

GPUStack — Thu, 08 Oct 2026 10:08:29 +0000

On September 20, 2026, the Qwen team open-sourced Qwen-Image-2.1, a model that unifies text-to-image generation and image editing. Its visual generation component is a 7B-parameter, 32-layer Single-Stream DiT, while text and conditioning-image encoding are handled by Qwen3-VL-8B. The team describes it as a compact image model designed to balance efficiency and general-purpose capability.

Here are the key specifications reported by the project and community testing:

Attribute	Specification
Intended use	Unified text-to-image and image-editing model
Visual generation component	7B parameters; 32-layer Single-Stream DiT (Block-Causal attention)
Text/conditioning encoder	Qwen3-VL-8B (text instructions and reference images use the same encoding path)
Core pipeline parameter count	Approximately 16B (the ModelScope repository lists 16.22B)
VAE	64-channel RGBA autoencoder with 16× spatial compression
Resolution	Native 2K; 2048 × 2048 by default; seven aspect ratios, up to 2752 × 1536
Inference	Flow Matching with the Euler scheduler; 40 steps recommended. Service defaults may differ.
Reference images	The model supports up to 10; the current vLLM-Omni recipe allows up to 4 per request.
Transparency	Native RGBA output, with the alpha channel generated directly in latent space
Checkpoint size	Approximately 33 GB (ModelScope repository)
License	Qwen Research License (limited to non-commercial research and evaluation)

The four upgrades highlighted by the Qwen team are a more compact and efficient architecture (the DiT shrinks from 20.4B parameters in the previous generation to 7B), native transparent-image generation (RGBA layers can be generated and edited directly), more flexible multi-image and local editing (the model supports up to 10 reference images, subject to backend limits), and improvements to text rendering and portrait lighting details. On transparency, the ComfyUI blog puts it this way: “No other major open model does this.”

1. Deploy Qwen-Image-2.1 on GPUStack

Qwen-Image-2.1 is a diffusion (DiT) model, not an autoregressive LLM, so it cannot be run with conventional vLLM alone. It uses vLLM-Omni, the vLLM project's runtime for diffusion and multimodal generation models. The Qwen team announced support on the model's release day (Day 0).

The workflow in GPUStack is similar to deploying an LLM: first add a vLLM version that points to a vLLM-Omni image under Inference Backends. The rest of the deployment can be completed in the web interface.

Test environment: NVIDIA A800-SXM4-80GB; driver 570.172.08; CUDA 12.8.

1) Add a vLLM-Omni version under Inference Backends

Important: Add the image version under Inference Backends first. It will then appear in the Backend Version dropdown on the deployment page.

In the left-hand menu, open Inference Backends, find the vLLM card, and click Edit. Under Version Configuration, click Add Version.

Version: qwen-image21
Image: vllm/vllm-omni:qwen-image21 (the image tag used in this guide). Verify that this image was built with the Qwen-Image-2.1 integration. At the time this guide was checked, the vLLM recipe said support had not yet been included in a versioned release package and that the corresponding branch should be used. An image tag alone does not prove that it is an official release.
Framework: Select CUDA.
Keep the default image entrypoint (ENTRYPOINT), vllm serve. Add --omni to the default command template, for example: {{model_path}} --omni --host {{worker_ip}} --port {{port}} --served-model-name {{model_name}}.

Save the configuration. You can then select qwen-image21 on the deployment page.

⚠️ Image and startup command: Qwen-Image-2.1 requires the diffusion implementation in vLLM-Omni and must be launched with vllm serve <model-name> --omni. Make sure the image contains the required model code and that GPUStack actually passes --omni to the process. A standard vLLM image or a configuration that omits this argument will not run the example as shown.

2) Create a deployment and select the model and backend

Return to Deployments and click Deploy Model in the upper-right corner. Enter the following settings:

Source: Select ModelScope (this guide uses ModelScope; choose a model source based on your network environment and organizational policy) and enter the repository ID Qwen/Qwen-Image-2.1.
Backend: Select vLLM.
Backend Version: Select the qwen-image21 version you just added.

Submit the deployment and wait for the instance to reach Running. If the model fails to load, check the image version, the --omni startup argument, and access permissions for the model repository.

2. Verify and call the service

1) Generate an image through an OpenAI-compatible API

In --omni mode, vLLM-Omni registers two OpenAI-compatible endpoints:

/v1/images/generations: standard image-generation endpoint
/v1/chat/completions: chat endpoint that supports multimodal input

The /v1/chat/completions endpoint uses a multimodal message structure: messages[].content is an array, and prompt text goes in a text item. To generate an image, also set modalities: ["image"] in the request body:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <API_KEY>" \
  -d '{
    "model": "qwen-image-2.1",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "A golden retriever running through a grassy field, backlit, cinematic lighting."}
        ]
      }
    ],
    "modalities": ["image"],
    "extra_body": {
      "size": "1024x1024",
      "num_inference_steps": 40,
      "true_cfg_scale": 1.0,
      "seed": 42
    }
  }'

To edit an image using a reference, add an image_url item to content, while keeping modalities: ["image"] and the image-generation parameters:

    "content": [
      {"type": "text", "text": "Extract the main subject from the reference image as a layer with a transparent background."},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,<REFERENCE_IMAGE_BASE64>"}}
    ]

Request parameter notes:

model: Use the same model name that you specified when creating the deployment.
messages[].content: Use the array structure; put the prompt in a text item and the reference image in an image_url item.
image_url.url: A data URI such as data:image/png;base64,<content> can be used. The current vLLM-Omni recipe limits each data-URI part to 1 MiB. For larger reference images, use the /v1/images/edits file-upload endpoint.

The generated image in the response is at choices[0].message.content[0].image_url.url. For /v1/images/generations, the image is at data[0].b64_json.

For /v1/images/generations, pass the resolution, number of steps, and guidance scale directly in the request body:

curl -s http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <API_KEY>" \
  -d '{
    "model": "qwen-image-2.1",
    "prompt": "A corgi wearing a knitted beanie, sitting at a wooden table by the window, with a steaming cup of coffee nearby.",
    "size": "2048x2048",
    "num_inference_steps": 40,
    "true_cfg_scale": 1.0,
    "seed": 42
  }'

Parameters to keep in mind:

num_inference_steps: Setting this to 40 is recommended. More steps usually increase generation time; results also depend on the prompt and sampling settings.
seed: Fix the seed when you need reproducible editing results.
true_cfg_scale: This guide sets it to 1.0. Do not mix it up with guidance_scale. If you also pass a negative prompt, consider how the guidance settings may affect latency and output quality.

2) Image editing and multi-image composition

One of the important capabilities of 2.1 is that it can take reference images and reorganize a scene while attempting to preserve subjects' identities and product appearances. Typical scenarios showcased by the Qwen team include:

Combining six portraits into a group photo: Provide individual portraits and combine them into one group image.
Combining five fashion items into an outfit: Provide a model, clothing, shoes, a bag, and a hat to generate a virtual try-on image.
Arranging a room from 10 furniture images: Provide a furniture set and compose it into a complete interior.

Reference-image limits depend on the inference backend. The model specification supports up to 10 images, while the current vLLM-Omni recipe allows up to 4 per request. Check whether your API and deployment version support local-editing controls such as selection, brush painting, and separate masks before relying on them.

Here is an edit of the image generated above. Pass the reference image as an image_url item to /v1/chat/completions and ask the model to change only the specified content while preserving the rest of the composition:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <API_KEY>" \
  -d '{
    "model": "qwen-image-2.1",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Replace the cup of coffee on the table in the reference image with a stack of books."},
          {"type": "image_url", "image_url": {"url": "data:image/png;base64,<IMG_B64>"}}
        ]
      }
    ],
    "modalities": ["image"],
    "extra_body": {
      "size": "1024x1024",
      "num_inference_steps": 40,
      "true_cfg_scale": 1.0,
      "seed": 42
    }
  }'

The returned image is at choices[0].message.content[0].image_url.url. If the field contains a data URI, extract its Base64 content and decode it as edited.png. For larger reference images, use the /v1/images/edits file-upload endpoint to avoid the data-URI size limit.

3) Native transparent images

The alpha channel is generated directly in latent space rather than added through post-processing. As a result, a PNG can be used in infographics, storyboards, stickers, and logos without first running a background-removal or segmentation model. The capability also works in reverse: you can extract a subject from a regular photo as a transparent layer, or edit text on a transparent layer while preserving the background.

Pass an existing image as an image_url item to /v1/chat/completions and ask the model to extract the subject as a transparent layer. First confirm that the current service configuration outputs RGBA. A model's ability to generate transparency does not mean every service configuration preserves the alpha channel by default.

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <API_KEY>" \
  -d '{
    "model": "qwen-image-2.1",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Extract the main subject from the reference image as a layer with a transparent background."},
          {"type": "image_url", "image_url": {"url": "data:image/png;base64,<IMG_B64>"}}
        ]
      }
    ],
    "modalities": ["image"],
    "extra_body": {
      "size": "1024x1024",
      "num_inference_steps": 40,
      "true_cfg_scale": 1.0,
      "seed": 42
    }
  }'

The returned image is at choices[0].message.content[0].image_url.url. Decoding it should give you a PNG with an alpha channel; verify that the output file actually retains transparency.

3. Important considerations

The license is not Apache-2.0. Qwen-Image-2.1 uses the Qwen Research License, which permits non-commercial research or evaluation. Commercial use requires a separate license. Consult the license text for the exact terms.
Benchmark scores depend on the evaluation method. When citing Qwen-Image-Bench, include the benchmark version, data date, and source. Note that the benchmark is published by the Qwen team; its results should not be presented as an independent third-party evaluation.
Community reports are not universal conclusions. Editing, multi-image composition, and transparent-image generation are capabilities highlighted by the project. Photorealism, artifacts, and consistency across multiple people can vary with the prompt, resolution, and inference configuration. When citing community feedback, include the source and test conditions.

Notes for international readers

Model hosting is region- and policy-dependent. This guide uses a ModelScope repository ID. The Qwen project also publishes model resources through Hugging Face; choose an accessible source that complies with your organization's network and data policies. A China-based hosting platform should not be described as universally accessible or as the only option.
The hardware and system details are specific to the author's test environment. The NVIDIA A800-SXM4-80GB, driver 570.172.08, and CUDA 12.8 are reproducibility details, not a universal hardware recommendation. Readers should check availability, export controls, and local deployment requirements in their region.
The license applies regardless of the deployment region. Do not imply that using a hosted model or deploying it outside China changes the Qwen Research License's restrictions. Commercial use requires separate authorization.

References

Summary

Qwen-Image-2.1 has a clear design direction: reduce the previous generation's 20.4B-parameter DiT to 7B, bring text-to-image generation, image editing, and transparent-image generation into one checkpoint, and use Qwen3-VL to unify text and conditioning-image encoding. Block-Causal attention enables prefix KV reuse, reducing inference overhead in multi-image editing scenarios.

With GPUStack's pluggable vLLM backend, you can configure a vLLM-Omni image that contains the Qwen-Image-2.1 integration, select the Qwen/Qwen-Image-2.1 repository, and verify that the startup command includes --omni. The image must match the vLLM-Omni code version in use. Check the official recipe for the supported parameters, input formats, and reference-image limits for your version.

Test environment: NVIDIA A800-SXM4-80GB | Driver: 570.172.08 | CUDA: 12.8 | GPUStack v2.2.2

GPUStack Day 0 Support for Kimi-K3: vLLM vs. SGLang Inference Benchmark on 8 B300 GPUs

GPUStack — Thu, 30 Jul 2026 10:44:17 +0000

This article documents the deployment and benchmarking of Kimi-K3 on a single server equipped with 8×NVIDIA B300 GPUs. It compares vLLM and SGLang under 64K and 200K long-context workloads. The key finding is straightforward: vLLM is faster at 64K, while SGLang with DCP overtakes it at 200K. The difference comes primarily from KV access during Decode—not from Prefill.

Kimi-K3 is a Mixture-of-Experts model designed for complex reasoning and long-context workloads. In this test, GPUStack was used to deploy the model on a single 8×NVIDIA B300 server with both vLLM and SGLang. Kimi-K3-DSpark was enabled as the draft model for speculative decoding.

The most important results are:

64K input, concurrency 10: vLLM completed the workload in 100.5 seconds versus 150.8 seconds for SGLang, making vLLM approximately 1.5× faster.
200K input, concurrency 10: SGLang completed the workload in 225.3 seconds versus 295.2 seconds for vLLM, making SGLang approximately 1.31× faster.
From 64K to 200K, stable Decode throughput degraded by 3.29× on vLLM but only 1.25× on SGLang.
The primary reason for the crossover is that SGLang ran with --dcp-size=8, while vLLM used decode_context_parallel_size=1.
Prefill performance was broadly comparable; most of the gap appeared during Decode.
The Random dataset produced extremely low DSPARK acceptance rates, so the speculative-decoding results should not be extrapolated directly to production traffic.

Important: This was not a strictly controlled A/B test. The two engines differed in DCP, memory allocation, kernel backends, and acceptance criteria. The results are most useful for understanding bottlenecks and deployment trade-offs—not for declaring an absolute winner.

Test Environment

Item	Configuration
Server	Single node
GPU	8×NVIDIA B300
Visible memory per GPU	267.69 GiB
Model	Kimi-K3 (MXFP4)
Draft model	Kimi-K3-DSpark
Inference engines	vLLM / SGLang
Input lengths	64K, 200K, and additional lengths
Output length	3K
Concurrency	10
Dataset	Random

Preparing the Model Weights

Download the Kimi-K3 main model and DSpark draft models from ModelScope.

Kimi-K3 Main Model (MXFP4)

modelscope download \
  --model moonshotai/Kimi-K3 \
  --local_dir ./Kimi-K3

DSpark Draft Model for vLLM

modelscope download \
  --model skyai/Kimi-K3-DSpark \
  --local_dir ./Kimi-K3-DSpark

DSpark Draft Model for SGLang

modelscope download \
  --model skyai/sglang-Kimi-K3-DSpark \
  --local_dir ./Kimi-K3-DSpark

Pulling the Inference Images

vLLM

docker pull vllm/vllm-openai:kimi-k3

SGLang

docker pull lmsysorg/sglang:kimi-k3

After pulling the images, add custom vLLM and SGLang versions under Inference Backends in GPUStack.

Deploying vLLM with GPUStack

Create a custom vLLM backend in GPUStack, select the Kimi-K3 model and all eight B300 GPUs, and configure the required environment variables.

The vLLM custom backend and model settings are shown below.

vLLM Environment Variables

VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1
VLLM_ALLREDUCE_USE_FLASHINFER=1
VLLM_ENGINE_READY_TIMEOUT_S=3600
VLLM_USE_V2_MODEL_RUNNER=1
VLLM_USE_RUST_FRONTEND=1

vLLM Startup Parameters

--trust-remote-code
--load-format fastsafetensors
--moe-backend auto
--gpu-memory-utilization 0.95
--tensor-parallel-size 8
--max-model-len 1000000
--kv-cache-dtype fp8
--attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}'
--max-num-batched-tokens 32768
--enable-prefix-caching
--enable-auto-tool-choice
--tool-call-parser kimi_k3
--reasoning-parser kimi_k3
--max-num-seqs 32
--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","num_speculative_tokens":7,"method":"dspark","attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

In this test, --gpu-memory-utilization 0.95 overcommitted the KV pool and caused runtime OOM retries. A safer configuration is discussed later.

Deploying SGLang with GPUStack

Create an SGLang custom backend, select all eight B300 GPUs, and configure the model and startup parameters.

The SGLang custom backend and model settings are shown below.

SGLang Startup Parameters

--trust-remote-code
--model-path <Kimi-K3 model path>
--tp-size 8
--dcp-size 8
--disable-custom-all-reduce
--enable-symm-mem
--mem-fraction-static 0.85
--reasoning-parser kimi_k3
--tool-call-parser kimi_k3
--mamba-full-memory-ratio 0.86
--host 0.0.0.0
--port 30000
--speculative-algorithm DSPARK
--speculative-draft-model-path <Kimi-K3-DSpark model path>
--speculative-dspark-block-size 7
--enable-linear-replayssm-spec
--chunked-prefill-size 32768
--max-prefill-tokens 32768
--context-length 1000000
--kv-cache-dtype fp8_e4m3

GPUStack exposes the live service logs, throughput, and KV Cache state after the service starts.

64K / 3K at Concurrency 10

The first comparison used 64K input, 3K output, and ten concurrent requests.

vLLM Result

SGLang Result

The corresponding inference logs are shown below.

Results Across Different Input Lengths

Additional input lengths were tested to observe how context growth affects TTFT, TPOT, ITL, and end-to-end latency.

vLLM

SGLang

Core Result: Performance Crosses Over Between 64K and 200K

Scenario	vLLM	SGLang	Result
64K end-to-end time	100.5 s	150.8 s	vLLM leads by ~1.5×
200K end-to-end time	295.2 s	225.3 s	SGLang leads by ~1.31×
64K stable Decode throughput	~501 tok/s	~335 tok/s	vLLM leads by ~1.5×
200K stable Decode throughput	~163 tok/s	~268 tok/s	SGLang leads by ~1.64×
Decode degradation, 64K→200K	3.29×	1.25×	SGLang scales better

In one sentence: vLLM is faster at shorter contexts because it avoids DCP communication overhead. As context grows, KV bandwidth becomes dominant, and SGLang’s DCP=8 distributes the KV reads across eight GPUs.

Source inspection shows that this is not simply a missing vLLM flag. vLLM explicitly prohibits DCP and DSPARK from being enabled together. SGLang allows them to coexist, but speculative decoding falls back to a non-adaptive verification mode. This is an architectural trade-off rather than a configuration oversight.

Why the Crossover Happens

1. Decode Scalability Is the Decisive Factor

Stable throughput was measured after all ten requests entered Decode.

Aggregate Decode throughput during the stable ten-request phase, in tokens/s, based on engine-reported logs.

vLLM (DCP=1)

At 64K, stable throughput was approximately 439–537 tok/s, peaking at 537 tok/s.
Per request, this is approximately 53.7 tok/s, or 18.6 ms/token.
At 200K, stable throughput was approximately 157.8–166.6 tok/s, averaging about 163 tok/s.
Per request, this is approximately 16.3 tok/s, or 61.3 ms/token.
Context length increased by about 3.1× while Decode performance fell by 3.29×, indicating near-linear degradation.

SGLang (DCP=8)

At 64K, stable throughput was approximately 313–418 tok/s, averaging about 335 tok/s.
Per request, this is approximately 33.5 tok/s, or 29.9 ms/token.
At 200K, stable throughput was approximately 252–288 tok/s, averaging about 268 tok/s.
Per request, this is approximately 26.8 tok/s, or 37.3 ms/token.
Context length increased by about 3.1× while Decode performance fell by only 1.25×.

At 64K, DCP communication overhead is greater than the reduction in KV reads, so vLLM is faster. At 200K, KV bandwidth becomes the primary bottleneck and SGLang’s DCP advantage becomes visible.

2. Prefill Performance Is Broadly Comparable

Both engines used a 32,768-token Prefill budget. Stable aggregate Prefill throughput was approximately 19K–23K tok/s, indicating comparable MoE + MLA Prefill capability on 8×B300.

The vLLM 64K run was affected by 32 soft OOM retries across eight GPUs and runtime JIT compilation of four kernels. These events occurred early in the test and made vLLM’s first-batch TTFT look worse. They were not reproduced in the 200K run, and SGLang did not report matching OOM or JIT events.

3. SGLang Logs Expose Long-Context Prefill Degradation

For a single 200K request, SGLang reported the following throughput over six chunks:

22265 → 21064 → 19628 → 18475 → 17243 → 16408 tok/s

Throughput returned to approximately 22K tok/s when the next request started. This directly shows the rising attention cost as cached context grows. vLLM reports a coarser ten-second aggregate prompt throughput, so the same pattern is less visible in its logs.

DSPARK Speculative Decoding Added Little Value

The Random dataset is effectively unpredictable, making it difficult for the draft model to produce consecutive accepted tokens. Both engines used Kimi-K3-DSpark with seven speculative tokens, but average accepted length at 200K remained close to one:

vLLM: approximately 1.05–1.17
SGLang: approximately 1.05–1.17

In the vLLM 200K run, drafted throughput was about 990 tok/s while accepted throughput was only about 21 tok/s. Approximately 98% of draft-model computation was discarded.

Scenario	Engine	Accepted Length	Acceptance / Draft Hit Rate	Interpretation
64K	vLLM	2.27–2.51	18%–21%	The only range with material benefit
64K	SGLang	1.21–1.55	3%–8%	Much weaker than vLLM on the same data
200K	vLLM	1.05–1.17	1.1%–2.5%	Draft compute is almost entirely wasted
200K	SGLang	1.05–1.17	1%–2%	The same collapse occurs

This run measured a “speculation as pure overhead” scenario. Production validation should use ShareGPT, real business prompts, or another natural-language dataset.

Different Acceptance Criteria

vLLM used probabilistic draft sampling with block rejection, while the SGLang DSPARK path used strict greedy matching. In SGLang, a mismatch on the first token invalidates the remaining tokens in the block. Strict comparison therefore requires the same dataset, sampling settings, gamma, and acceptance criteria.

Client Metrics: Excluding Unreliable Throughput Columns

Metric	vLLM 64K	SGLang 64K	vLLM 200K	SGLang 200K
Total time (s)	100.5	150.82	295.21	225.34
Mean request latency (s)	84.42	141.53	284.58	217.83
Mean TTFT (ms)	34,838.8	41,670.6	125,210.8	59,002.0
TTFT P50 (ms)	34,181.1	40,391.9	125,189.2	54,104.0
TTFT P90 (ms)	36,987.9	53,525.0	126,667.0	103,336.0
Mean TPOT (ms)	12.28	47.18	45.21	72.6
Mean ITL (ms)	9.94	33.3	3.47	52.96
Failed / incomplete	0 / 0	0 / 0	0 / 0	0 / 0

Some client-reported throughput fields cannot be reconciled with token counts, concurrency, and elapsed time. For example, the vLLM 200K run reported 1111.61 output tok/s, while 10 × 3000 ÷ 295.21 gives a physical upper bound of about 101.6 tok/s. The discrepancy is consistently close to 11×, so this article uses latency metrics from the client and throughput from engine logs.

TTFT Semantics Must Also Be Aligned

With --reasoning-parser kimi_k3, reasoning output is emitted in delta.reasoning, not delta.content. A benchmark that starts TTFT only at the first content token systematically inflates TTFT for both engines. For reasoning models, a better definition is:

time-to-first-any-delta

The timer should stop on the first reasoning or content delta.

Memory Allocation and KV Cache Capacity

Each B300 exposed 267.69 GiB. Model weight memory was similar, but the remaining memory was allocated differently.

vLLM

Item	Value
Model weights and non-torch memory	197.72 GiB
Peak activation memory	12.82 GiB
CUDA Graph	5.13 GiB
KV Cache	43.76 GiB

The KV pool held 2,687,776 tokens, giving a theoretical maximum concurrency of 2.69 at a 1M context. However, the pool was overcommitted. vLLM logs indicated that 38.48 GiB was the safe KV allocation under the configured budget.

Use either:

--gpu-memory-utilization 0.92

or:

--kv-cache-memory 41317885440

SGLang

Item	Value
Model weights	195.86 GiB
Full-attention KV Cache	11.80 GiB
Draft-model KV (K–V)	8.74 GiB
Mamba / SSM state	10.93 GiB
Remaining space	17.08 GiB

The full-attention pool held 916,672 tokens, below the configured one-million-token context. A full 1M request therefore cannot fit without adjusting the pool. max_running_requests also fell from 48 to 39 because of the Mamba state cache.

Cold-Start Cost

Stage	vLLM	SGLang	Notes
Container created	17:06:21	18:01:45	GPUStack timestamp
Weight loading	163.8 s	141.0 s	12 vs. 96 shards
Weight size per GPU	194.54 GiB	195.86 GB	MXFP4 MoE
Kernel warmup	CuTeDSL + KDA Triton	KDA / FA4 / RoPE	K3-specific JIT on both
CUDA Graph capture	85–88 s, 51 sizes	~5.5 min, 67 sizes	SGLang captured more shapes
Engine initialization	409.61 s	~370 s	SGLang inferred from timestamps
Container-to-ready	~10.6 min	~10.2 min	Same order of magnitude

Both engines required roughly ten minutes. Frequent autoscaling should therefore use warm instances, staged expansion, or prewarming.

The Configurations Were Not Fully Equivalent

Item	vLLM	SGLang	Potential Impact
Decode context parallelism	DCP=1	DCP=8, `comm=a2a`	Dominant at long context
All-Reduce	FlashInfer + Custom + SymmMem	Custom AR disabled; NCCL fallback	High for small-batch Decode
MLA Prefill backend	TRTLLM_RAGGED	`trtllm_mla`	Medium
MLA Decode backend	FLASHINFER_MLA	`cutedsl_mla`	Medium
Linear attention	FlashKDA Prefill	Triton Decode/verify/extend	Medium
MoE backend	FLASHINFER_TRTLLM_MXFP4_MXFP8	`flashinfer_mxfp4`	Medium
KV page size	32, aligned to 1536	64	Medium
Compile path	Disabled by breakable CUDA Graph	Not enabled	Medium
Memory fraction	0.95	0.92	Low–medium
Concurrency limit	`max_num_seqs=64`	48→39	Not reached in this test

Warnings Worth Addressing

FP8 KV Cache Scaling Factors

Both engines ran FP8 KV Cache with a fallback scale of 1.0. Performance conclusions should be paired with a fixed quality evaluation to confirm that FP8 KV Cache does not introduce unacceptable accuracy loss.

Slow Tokenizer

Both engines reported a slow tokenizer. At 64K and 200K, tokenization time contributes directly to TTFT. A fast tokenizer should be enabled where possible.

SGLang Device Identification

One log entry identified the device as NVIDIA L20D, while other entries correctly reported B300/GB300 and SM100/SM103. Because DCP communication backend selection can depend on the device name, the selected a2a path should be verified.

vLLM Memory Utilization

The 0.95 setting overcommitted KV memory. Reducing it to 0.92 or explicitly setting the log-recommended KV cache size removes the repeated runtime allocation retries.

Potentially Inactive vLLM Fusions

Some reported fusions are Inductor passes. Kimi-K3 enables breakable CUDA Graph mode, which can force compilation mode to NONE; the fusions may therefore not have been applied. FlashInfer autotuning was also disabled, meaning this vLLM result may be below its best achievable performance.

Source-Level Root Cause: DCP vs. Speculative Decoding

This is the strongest conclusion in the analysis, and it cannot be bypassed with a simple flag change.

vLLM structurally disallows DCP and DSPARK from running together. When decode_context_parallel_size > 1 is combined with K3DSparkModel, startup fails. To gain long-context KV sharding in vLLM, speculative decoding must be disabled.

SGLang permits DCP and DSPARK together, but dcp_size > 1 forces static ragged verification. Adaptive SPS cost tables, STS calibration, confidence planning, and block acceptance estimation are no longer active.

Capability	vLLM	SGLang	Deployment Implication
DCP + speculative decoding	Not allowed	Allowed, but non-adaptive	Both require a trade-off
Long-context KV sharding	Requires disabling speculation	Can coexist with speculation	Structural source of the 200K advantage
Adaptive speculation	No equivalent planner	Disabled when DCP is enabled	Neither offers both benefits together
Adjustable acceptance	Two explicit parameters	Threshold hard-coded in kernel	SGLang has no ready tuning knob

Additional Findings from Source Inspection

SGLang’s Fused KDA Verify Kernel Probably Did Not Run

The dispatcher assigns the verify kernel to Triton unless strict runtime shape constraints allow the CuTe path. Logs ended with verify=TritonKDAKernel, leaving additional optimization potential.

Identity of SGLang’s “Second KV Pool”

7,333,376 = 916,672 × 8, exactly matching dcp_size. The pool is the draft worker’s MHA KV pool addressed in DCP’s virtual token space, not a second architectural KV type.

Why vLLM Uses a 1536-Token Attention Block

The hybrid model requires an attention page to be at least as large as a Mamba page. The block grows from 32 to 1536, with approximately 0.70% Mamba-page padding. This coarser boundary also influences chunk alignment and can contribute to Decode starvation.

Source of vLLM’s 2.69× Concurrency Limit

The scheduler reserves KV for the full maximum sequence length at admission time. Reducing --max-model-len from 1M to the actual production requirement immediately increases concurrency.

Deployment Recommendations

Inputs Up to 64K, Throughput First

Prefer vLLM. It delivered approximately 1.5× higher Decode throughput and approximately 1.5× lower end-to-end time in this test.

200K and Longer Document Workloads

Prefer SGLang with DCP. It delivered approximately 1.64× higher Decode throughput at 200K and degraded much more slowly as context increased. Increase the KV pool to at least the target context length before production.

Long Context Plus High Concurrency

The current configuration favors vLLM’s capacity, but this conclusion requires retesting. SGLang should first rebalance KV and Mamba memory, then be compared with a vLLM configuration that disables speculation and enables DCP.

Next-Round Retest Plan

#	Action	Owner	Rationale
1	Set `--long-prefill-token-threshold 4096`	vLLM	Prevent one long Prefill from consuming the entire Decode budget
2	Replace Random with ShareGPT or real long documents	Benchmark platform	Current acceptance is only 1%–8%
3	Measure time-to-first-any-delta	Benchmark platform	Include reasoning deltas in TTFT
4	Fix throughput accounting	Benchmark platform	Current vLLM output throughput is inflated by ~11×
5	Reduce memory utilization to 0.92 or pin KV memory	vLLM	Remove runtime OOM retries
6	Add an SGLang DCP-off group	SGLang	Isolate DCP and static-verification effects
7	Set max model length to the business requirement	vLLM	Avoid unnecessary full-length KV reservation
8	Expand warmup to cover runtime JIT shapes	vLLM	Remove first-batch compilation noise
9	Re-enable FlashInfer autotuning	vLLM	Allow better MoE tactics
10	Re-enable custom All-Reduce and diagnose multicast	SGLang	Reduce small-batch communication overhead
11	Diagnose the KDA CuTe verify fallback	SGLang	Recover unused optimization potential
12	Provide FP8 KV scaling factors and run quality regression	Both	Add an accuracy basis to performance conclusions
13	Increase Mamba memory ratio or use bf16 SSM	SGLang	Remove the 39-request limit and raise the KV pool
14	Add P99 and concurrency ≥64	Benchmark platform	Exercise scheduler and concurrency limits

Conclusion

The most valuable result of this 8×B300 test is not a simple ranking. It shows how the bottleneck changes with context length:

At 64K, communication overhead matters more, favoring vLLM.
At 200K, KV read bandwidth dominates, allowing SGLang with DCP to overtake.
Prefill is not the main differentiator; Decode scalability is.
Speculative decoding is ineffective on Random data.
Memory allocation, TTFT semantics, and client-side throughput accounting can materially change the apparent result.

Production selection should therefore evaluate context length, concurrency, KV capacity, DCP, real request distributions, and output quality together—not rely on a single TPS number.

Day 0 Deployment of GLM-5.2-FP8-DSpark on GPUStack

GPUStack — Wed, 22 Jul 2026 10:28:37 +0000

This article is curated from a real-world deployment experience shared by a GPUStack community member. GLM-5.2-FP8-DSpark is an enhanced version of GLM-5.2-FP8 that applies Speculative Decoding by loading an external draft model (Speculator) provided by Red Hat AI. It keeps the original model weights unchanged while adding an additional draft model to improve decoding throughput. On the day the model was released (Day 0), a community member deployed it on an 8× H20-141G environment through GPUStack and conducted a performance comparison against the original GLM-5.2-FP8 under the same hardware conditions.

Let's start with the conclusion. The results from this test are mixed and should be interpreted objectively:

Single-concurrency scenario: In the GPUStack Playground, DSpark achieved approximately 171 TPS, while the original GLM-5.2-FP8 achieved approximately 72 TPS, delivering around a 2.2× improvement.
High-concurrency scenario (64K input / 3K output): DSpark achieved 59.8 TPS output throughput, which was actually lower than the 99.2 TPS achieved by the original MTP approach. Due to the lower acceptance rate of the external draft model on the Random dataset, DSpark did not improve performance under high concurrency and even introduced regression.
Key takeaway: With the specific external draft model + GLM-5.2-FP8 combination tested in this article, DSpark is currently better suited for single-concurrency / low-concurrency interactive scenarios where first-token latency and response throughput are critical, or for research and evaluation purposes. For high-concurrency production workloads, waiting for draft models with higher acceptance rates is recommended. (The conclusions above only apply to the draft model and dataset used in this test, and do not represent the upper limit of the DSpark speculative decoding approach.)

The following sections provide the complete deployment process and benchmark results.

1. Deploying GLM-5.2-FP8-DSpark on GPUStack

DSpark requires a newer vLLM image (v0.25.0) to support speculative decoding with an external draft model. With GPUStack's pluggable vLLM backend architecture, we only need to add a new vLLM backend version. The entire process can be completed through the GPUStack Web UI with just a few clicks.

① Add a New vLLM Backend Version

From the left navigation menu, go to Inference Backend, find the vLLM card, and click Edit.

Under Version Configuration, click Add Version and create a new version named 25. Enter the following image:

vllm/vllm-openai:v0.25.0、

Select CUDA as the framework and save the configuration. A new backend version named 25-custom will then become available.

② Create a Deployment and Select Model & Backend

Go back to the Deployments page and click Deploy Model in the upper-right corner. Configure the basic information as follows:

Source: Select ModelScope
Repository ID: ZhipuAI/GLM-5.2-FP8
Backend: Select vLLM
Backend Version: Select the newly added 25-custom

③ Configure Backend Parameters

In the Advanced section, configure the backend parameters as follows (using an 8× H20-141G environment as an example):

bash --max-model-len 131072 --tensor-parallel-size 8 --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 --enable-expert-parallel --trust-remote-code --gpu-memory-utilization=0.9 --max-num-batched-tokens 32768 --max-num-seqs 16 --speculative-config '{"model":"/var/lib/gpustack/cache/model_scope/RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens":7,"method":"dspark"}'

The key parameter is --speculative-config:

Set method to dspark
Set model to the downloaded external draft model path

If the engine takes longer to load, add the environment variable:

bash VLLM_ENGINE_READY_TIMEOUT_S=3600

to extend the ready timeout.
https://

Draft model download:
https://modelscope.cn/models/RedHatAI/GLM-5.2-speculator.dspark

Submit the deployment. Once the instance status changes to Running, the deployment is complete.

2. Performance Benchmark: DSpark vs Original GLM-5.2-FP8

Under the same hardware environment (8× H20-141G), we compared the two models from two perspectives: single-concurrency interactive inference and high-concurrency batch inference.

Scenario 1: Single-Concurrency Interactive Inference

Run a conversation directly in the GPUStack Playground and monitor the real-time output throughput.

GLM-5.2-FP8-DSpark:
Output throughput reached 171.6 Tokens/s, remaining stable in the 170+ range.

Original GLM-5.2-FP8:
Output throughput reached 71.81 Tokens/s.

Under single-concurrency workloads, DSpark achieved approximately 171 TPS, around 2.2× faster than the original version (~72 TPS). This demonstrates the typical benefit of speculative decoding in low-concurrency interactive scenarios.

Scenario 2: High-Concurrency Batch Inference (64K Input / 3K Output, 10 Requests)

Using GPUStack's built-in benchmark tool, we tested the model with a Random dataset containing 64K input tokens / 3K output tokens and 10 requests (with an actual average concurrency of approximately 6).

The baseline for comparison is the original MTP approach with speculative decoding enabled.

Original MTP (method: mtp, num_speculative_tokens: 5):

Total throughput: 2216.39 Tokens/s
Output throughput: 99.22 Tokens/s
TPOT: 63.92 ms

DSpark (method: dspark, num_speculative_tokens: 7):

Total throughput: 1336.12 Tokens/s
Output throughput: 59.82 Tokens/s
TPOT: 103.08 ms

Under high-concurrency workloads, DSpark's output throughput (59.8 TPS) did not improve and instead regressed, performing significantly below the original MTP approach (99.2 TPS). TPOT was also higher.

The main reason is that the external draft model provided by Red Hat AI achieved a relatively low acceptance rate on the Random dataset. When speculative decoding has insufficient acceptance, the additional draft computation overhead can negatively impact overall throughput.

Performance Summary

Scenario	Metric	Original GLM-5.2-FP8	GLM-5.2-FP8-DSpark	Comparison
Single-concurrency (Playground)	Output TPS	≈ 71.8	≈ 171.6	≈ 2.2×
High-concurrency 64K/3K	Total Throughput (TPS)	2216.39 (MTP)	1336.12	≈ 0.60×
High-concurrency 64K/3K	Output Throughput (TPS)	99.22 (MTP)	59.82	≈ 0.60×
High-concurrency 64K/3K	TPOT (ms)	63.92 (MTP)	103.08	Higher

Summary

The performance characteristics of GLM-5.2-FP8-DSpark show a clear trade-off:

Single-concurrency / low-concurrency scenarios:

Speculative decoding provides significant benefits. In this test, DSpark achieved 1.2–2.2× the throughput of the original GLM-5.2-FP8, making it suitable for interactive workloads that are sensitive to single-request throughput and response latency.
High-concurrency scenarios:

With the external draft model used in this test, the acceptance rate on the Random dataset was relatively low. As a result, throughput did not improve significantly and was even lower than the original MTP approach. It is currently not recommended for similar high-concurrency production workloads.
Recommendation:

Based on the model and workload tested in this article, DSpark is currently more suitable for research validation and low-concurrency scenarios. For high-concurrency production deployment, it is recommended to wait for draft model versions with higher acceptance rates on these workloads.

(The conclusions above only apply to the draft model and dataset used in this test, and do not represent the upper limit of the DSpark speculative decoding approach.)

Thanks to GPUStack's pluggable vLLM backend architecture, adding a new image version, integrating an external draft model, or switching between different inference configurations can all be completed through the Web UI with just a few clicks. This enables teams to evaluate new models from Day 0 with minimal adaptation effort.

Test Environment: 8× H20-141G ｜ GPUStack v2 ｜ vLLM v0.25.0 (25-custom image)

Thanks to the GPUStack community member for sharing this real-world deployment experience.

Where Did Your GPU Resources Go? GPUStack Usage Tracking Gives You the Answer at a Glance

GPUStack — Wed, 08 Jul 2026 02:17:30 +0000

You bought a stack of GPUs and deployed a bunch of models. Then, at the end of the month, your boss asks: “Are these GPUs actually worth the cost?” Can you answer?

If not, this article is for you.

GPUStack’s new Usage feature is now available. It provides clear visibility into three key types of resource consumption: token usage, GPU/CPU instance runtime, and storage usage.

Who is using the resources, how much they are using, and which models the usage is associated with — you no longer have to guess.

1. Why Do You Need It?

In a shared GPU cluster with multiple users and multiple models, the biggest pain point is not always “not enough compute.” More often, it is: “Where did the compute go?”

You want to calculate costs, but cannot get usage details broken down by user.
Platform admins want to know which model consumes the most resources, but have to dig through logs.
Business teams ask for more GPU capacity, but no one can clearly explain the actual utilization of existing resources.

GPUStack Usage is built to answer exactly these questions. It connects the full workflow from collection → aggregation → visualization, making every unit of compute consumption traceable.

2. Five Tabs Covering All Usage Scenarios

Click the bar chart icon in the left navigation bar to enter the Usage page. You will see five tabs:

Tab	What You Can See
Summary	A complete overview of token, compute, and storage usage
Tokens	LLM token usage, broken down by model / user / API key
GPU Instances	GPU/CPU instance runtime, broken down by instance type / instance / user
Storage	Storage usage, broken down by storage volume / user
Resource Events	Resource lifecycle audit logs that explain where each number comes from

What exactly is being measured?

Resource	When It Is Measured	Granularity
Token	When an inference request is served through the gateway	Daily
GPU/CPU Instance	When an instance is in the “running (billable)” state; usage stops accumulating after the instance is stopped or deleted	Hourly
Storage	Measured from creation to deletion, regardless of whether it is mounted or used	Hourly

3. Scenario 1: View Usage by User, Clearly See Who Used What

This is one of the most important capabilities in a shared cluster. GPUStack has two built-in roles: Admin and User.

Admin: Can view usage across all users and drill down into any member through the “filter by user” control.
User: Can only view their own usage details and cannot see other users’ data.

In other words, platform admins can easily answer questions such as: “How many tokens did this user consume this month?” or “Who used the most GPU hours?” For cost allocation and quota planning, the data is already there.

💡 The detailed tables in each tab support grouping by user. This is only visible to admins, making each user’s usage bill clear and easy to review.

For regular users, the Usage page shows a dedicated usage bill of their own: how many tokens they used this month, how many GPU hours they ran, and how much storage they occupied. Everything is clear at a glance. This helps users stay aware of their usage, compare it against their quota, and avoid overuse.

They cannot see anyone else’s data, so the permission boundary remains clear and the experience is more secure.

4. Scenario 2: Break Down Usage by API Key to See Which Application Is Driving Costs

Viewing usage by user gives you a people-oriented perspective. Viewing usage by API Key gives you an application-oriented perspective — and in many cases, applications are the real source of consumption.

In practice, each connected application or business pipeline usually uses a dedicated API Key. The Tokens tab supports grouping by API Key, so you can directly see:

Which application / integration partner is continuously consuming tokens
Whether the cost of a business line has suddenly increased
Which keys are barely used and can be reclaimed

By combining the dimensions of users and applications, cost attribution becomes much more complete.

5. Scenario 3: Clearly Track Usage Across Different Types of Models

Running large language models, embedding models, and open-source models of different sizes in the same cluster? No problem.

At the top of the Tokens tab, GPUStack shows the key metrics directly: input / output / total tokens, API requests, and number of models used. The detailed table below can be grouped by model, showing:

Input tokens, including cached input tokens, which are marked separately
Output tokens
Total tokens
API requests
Last active time

This lets you quickly identify which models are receiving the most traffic, which models were requested but are barely used, and whether a model should be taken offline or scaled up. The data helps you make the decision.

📌 Easily verify the effect of cache optimization: The Input Tokens Cached field separately marks the portion of input tokens that hit the prompt cache. After optimizing prompt caching, you can check the proportion of this number to directly quantify how many input tokens were saved. Whether the optimization worked is clear at a glance.

Also: Even if a model has been deleted, its historical usage is still retained in the table and marked as Deleted. It is still included in the total usage, so historical records are not lost.

6. Scenario 4: GPU Compute Usage Is No Longer a Black Box

The GPU Instances tab breaks down instance runtime clearly. The key is that it distinguishes between two metrics that are often confused:

Metric	Meaning
Instance Hours	The wall-clock runtime of an instance, regardless of how many GPUs it uses
GPU Hours	Accelerator runtime = runtime × number of GPUs

For example, a 2-GPU instance running for 1 hour equals 2 GPU hours, but only 1 instance hour.

This distinction lets you calculate actual GPU usage instead of being misled by the number of instances. The trend chart can also be grouped by instance type / instance / user, helping you understand the current situation before scaling capacity.

⚙️ CPU instances do not have accelerator cards, so they are counted only in Instance Hours, not GPU Hours.

7. Scenario 5: Storage Usage, Including Idle Volumes

The Storage tab focuses on storage capacity usage over time. The key metrics are GB-Days and GB-Hours: capacity × duration. Usage can be broken down by storage volume / user.

The important detail is this: storage is measured from creation to deletion, regardless of whether it is mounted. This means there are no blind spots in the usage records. Even if a volume is not mounted or accessed, as long as it still exists, its usage is counted in GB-Days. It will not disappear from the bill simply because it is “not being used.”

This makes forgotten idle volumes easy to find. Sort by GB-Days, identify the volumes occupying the most capacity, check whether they are still used by any instance, and clean up what should be removed.

8. Scenario 6: Find Zombie Resources and Stop Paying for Waste

After a cluster has been running for a while, it often accumulates resources that no one remembers: models that are no longer called, or instances that were never stopped. They no longer create value, but they still occupy resources.

For resources that are measured only when used, such as models and GPU/CPU instances, each detailed table includes a Last Active column. It records the last time the resource generated usage. Once the resource is no longer used, usage stops accumulating and Last Active stays at that moment. Long-inactive models and forgotten stopped instances immediately become visible, making cleanup much more targeted.

9. Scenario 7: View Trends and Plan Capacity Without Guesswork

Both the Summary page and the individual tabs include trend charts. With adjustable granularity by hour / day / week / month, you can see how usage changes over time instead of only looking at a static total.

Is usage stable, or is it growing faster?
When did the growth inflection point occur, and does it match a business launch?
Based on the current trend, how long can existing capacity last?

Once the trend is clear, capacity planning moves from guesswork to data-driven decisions. Trend charts can also be split by group, making structural changes easy to spot.

10. Scenario 8: Every Number Has a Source, and Resource Events Show the Full Lifecycle

Each tab above shows the result: how many GPU hours were used, how many GB-Days were consumed, and so on. The Resource Events tab shows where those numbers come from. The lifecycle of every instance and storage volume is recorded as events.

Sorted in reverse chronological order, it becomes a resource timeline. The lifecycle of an instance is easy to follow, with colored labels showing each state clearly:

🟢 Created → 🔵 Started and measured → 🟠 Stopped and no longer measured → 🔴 Deleted

This means:

Trace usage for reconciliation: Why does an instance have this number of Instance Hours? Check the period from Started to Stopped, and you can see exactly how long it ran.
Filter and investigate: Filter by date / resource type / event type / name to quickly answer questions such as “which volumes were deleted last week?” or “which instance was repeatedly started and stopped?”
Explain inactivity: If Last Active in Scenario 6 shows that an instance has stopped generating usage, come here to check when it was Stopped or Deleted.

11. Put the Data to Work: Export Usage for Reconciliation

Seeing the data is not enough. You also need to take it out and use it.

The Tokens, GPU Instances, and Storage tabs all support export through the download icon. You can preview the currently filtered details and then download them. Monthly billing, financial reconciliation, internal reports — once you have the data, you can work with it directly, without asking engineers to write custom scripts.

12. Details That Make Usage Tracking Reliable

Unified Time Zone

All timestamps across tabs, including trend buckets, Last Active, and event time, are displayed using the same rollup time zone. This ensures that calendar boundaries are aligned across all tabs. It can be configured through the GPUSTACK_USAGE_ROLLUP_TIMEZONE environment variable and follows the server’s local time zone by default.

Flexible Common Controls

Date range, defaulting to the last 30 days, filtering by user / model / API Key, refresh, metric switching, grouping, and granularity adjustment are all available out of the box.

Data Retention and Archiving

Aggregated token, compute, and storage usage data is retained for about 13 months by default, and then archived by background tasks. The retention window and archive schedule can be configured through the GPUSTACK_*_RETENTION_MONTHS and GPUSTACK_*_ARCHIVE_CRON environment variables.

Final Thoughts

Compute is expensive, but compute you cannot see clearly is even more expensive.

GPUStack Usage makes every token, every GPU hour, and every GB of storage measurable, traceable, and attributable. Whether you are in finance handling cost allocation, an administrator planning capacity, or a business owner trying to prove ROI, it gives you answers backed by data.

Upgrade to the latest version of GPUStack, open the Usage page, and see where your compute resources are really going.

Day 0 Benchmark: Deploying DeepSeek-V4-Flash-DSpark on GPUStack Doubles Throughput

GPUStack — Wed, 01 Jul 2026 14:05:03 +0000

This article is based on a community benchmark contributed by a GPUStack user. DeepSeek-V4-Flash-DSpark enhances DeepSeek-V4-Flash by adding a Speculative Decoding module. Using the same model weights with an additional speculative decoder, it significantly improves both inference throughput and Time to First Token (TTFT).

On Day 0 of the model release, a GPUStack community member deployed and benchmarked DeepSeek-V4-Flash-DSpark on an 8× H20-141G setup, comparing it against the original DeepSeek-V4-Flash (DSV4F) under identical deployment settings. Here are the key results:

Single-stream throughput: In the 1K input / 1K output workload, DSpark achieved 195 tokens/s, approximately 2× the throughput of the original model (96 tokens/s)；
Overall throughput: In the 64K input / 3K output workload with 10 concurrent requests, DSpark reached 338 tokens/s, approximately 1.7× higher than the original (198 tokens/s)；
Time to First Token (TTFT): Reduced to roughly half that of the original model.

The following sections walk through the complete deployment process and benchmark results.

1. Deploying DSpark on GPUStack

GPUStack comes with the SGLang inference backend built in. To deploy DSpark, simply add a container image that supports the model. The entire process can be completed in just a few clicks through the web UI.

① Go to Inference Backends and edit SGLang.

From the left navigation menu, go to Inference Backends, locate the SGLang card, then click the menu in the upper-right corner and select Edit.

② Add a DSpark-compatible image

Under Version Configuration, click Add Version to create a new version named dspark, then specify the following container image:

swr.cn-north-4.myhuaweicloud.com/desaysv/gpustack/sglang-dspark:v1.0

Select CUDA as the framework. Set the image entrypoint override to sglang serve, and set the command to: --model-path {{model_path}} --host {{worker_ip}} --port {{port}}。

③ Create a Deployment

Return to the Deployments page and click Deploy Model in the upper-right corner. For Source, select ModelScope.

④ Select the Model and Inference Backend

Search for and select deepseek-ai/DeepSeek-V4-Flash-DSpark.
Set the Inference Backend to SGLang.
Select dspark-custom as the backend version.

⑤ Configure Backend Parameters

Under Advanced, configure the backend parameters as follows (using an 8× H20-141G setup as an example):

--context-length 1000000
--trust-remote-code
--tp-size 8
--ep-size 8
--moe-runner-backend flashinfer_mxfp4
--speculative-moe-runner-backend flashinfer_mxfp4
--speculative-algorithm DSPARK
--speculative-eagle-topk 1
--speculative-num-steps 1
--mem-fraction-static 0.85
--cuda-graph-max-bs 32
--max-running-requests 32
--disable-overlap-schedule

⑥ Configure Environment Variables

Add the following environment variable to ensure the required dependencies are installed correctly:

Key	Value
`PYPI_PACKAGES_INSTALL`	`-U distro -i https://mirrors.aliyun.com/pypi/simple/`

⑦ Start the Deployment and Monitor the Logs

Once submitted, GPUStack will start the deployment. In the logs, you should see CUDA Graph capture, Application startup complete, and Uvicorn listening on the inference port, indicating that the model has started successfully:

⑧ Verify the Deployment Status

Once the instance reaches the Running state, the deployment is complete.

⑨ Verify the Deployment

Open the Playground and send a few prompts to the model. The real-time throughput indicator in the lower-right corner should show an output rate of 185.94 tokens/s, with single-stream throughput remaining stable at around 200 tokens/s.

⑩ View the Inference Service Port

If you want to run benchmarks directly against the service, open the instance details to find the inference service IP address and port (in this example, 10.91.3.213:40048).

2. Benchmark Results: DSpark vs. DSV4F

Under identical hardware and deployment settings, we benchmarked the original DeepSeek-V4-Flash (with MTP enabled) against DSpark across two workloads. All benchmarks were conducted using SGLang's built-in bench_serving tool.

Scenario 1: 1K Input / 1K Output (Single Request)

HF_ENDPOINT=https://hf-mirror.com python3 -m sglang.bench_serving \
    --backend sglang --port 40048 \
    --dataset-name random --random-input-len 1024 --random-output-len 1024 \
    --random-range-ratio 1.0 --num-prompts 1 \
    --max-concurrency 1 --request-rate inf --host <Inference Server IP Address>

Original DSV4F: Output throughput: 96.20 tokens/s, TTFT: 300.45 ms, Accept length: 2.71

DSpark（DSV4FD）： Output throughput 195.18 tok/s，TTFT 129.34 ms，Accept length 4.42

Single-stream throughput reached 195 tokens/s, approximately 2× that of the original DSV4F (96 tokens/s), while Time to First Token (TTFT) was reduced to about half.

Scenario 2: 64K Input / 3K Output (10 Concurrent Requests)

HF_ENDPOINT=https://hf-mirror.com python3 -m sglang.bench_serving \
    --backend sglang --port 40048 \
    --dataset-name random --random-input-len 64000 --random-output-len 3000 \
    --random-range-ratio 1.0 --num-prompts 10 \
    --max-concurrency 1 --request-rate inf --host <Inference Server IP Address>

Original DSV4F (MTP enabled): Output throughput: 198.60 tokens/s, Speculative acceptance rate: 20.91%, Acceptance length: 1.21

DSpark（DSV4FD）： Output throughput 338.17 tok/s，Accept length 4.90

In the long-context workload, DSpark achieved 338 tokens/s, approximately 1.7× the throughput of the original DSV4F (198 tokens/s), nearly doubling overall throughput.

Summary

Workload	Metric	Original DSV4F	DSpark (DSV4FD)	Improvement
1K Input / 1K Output	Output Throughput (tokens/s)	96.20	195.18	≈ 2.0×
1K Input / 1K Output	TTFT (ms)	300.45	129.34	≈ 0.43×
1K Input / 1K Output	Acceptance Length	2.71	4.42	—
64K Input / 3K Output	Output Throughput (tokens/s)	198.60	338.17	≈ 1.7×
64K Input / 3K Output	Acceptance Length	1.21	4.90	—

Conclusion

Across both single-request and long-context workloads, DeepSeek-V4-Flash-DSpark consistently delivered around 2× the throughput of the original DSV4F, while reducing Time to First Token (TTFT) to roughly half. Achieving these gains requires nothing more than switching to the DSpark model weights and container image with the integrated speculative decoding module.

With GPUStack's built-in SGLang inference backend, the entire deployment can be completed in just a few clicks through the web UI, making the model production-ready on Day 0.

Benchmark environment: 8× H20-141G | GPUStack v2 | SGLang 0.5.14 (using the dspark-custom image based on a patched SGLang 0.5.14)

Acknowledgments: Thanks to the GPUStack community member for sharing the benchmark results.

GPUStack v2.2: From Model Serving to Token Operations, from Compute Pooling to GPU-as-a-Service

GPUStack — Tue, 30 Jun 2026 02:34:35 +0000

Deploying a model and bringing it online is only the starting point of AI service delivery.

As large language model applications move into scaled production, AI infrastructure is entering an inevitable phase of maturity — from simply being able to run to becoming operations-ready.

This shift is not just about adding more features. It reflects an evolution in platform positioning: from reliably serving inference workloads to becoming the infrastructure foundation that can truly support enterprise AI service delivery.

At this stage, the core challenge lies in advancing two areas in parallel: model serving must deliver operations-grade reliability and visibility, while compute management must expand from “serving inference workloads” to “unified allocation of the diverse resources required by AI.”

GPUStack v2.2 continues to move deeper in both directions: model serving is evolving from available to operations-ready, while compute management is extending from unified scheduling to on-demand services.

Deepening Support for Model Inference Scenarios and Lifecycle Management

The stability of model serving is not only a deployment-stage concern. After an instance starts running, issues such as OOM errors, hanging inference requests, and silent process crashes are often the more common problems in production environments.

Previously, GPUStack’s health checks mainly covered the startup phase. Once an instance started successfully, the platform had no way to detect issues that occurred later. Faulty instances could remain in the service pool and continue receiving traffic, resulting in silent failures that would only be addressed after someone noticed and handled them manually.

In v2.2, health probing is extended across the entire runtime lifecycle. The platform continuously checks the actual inference capability of each instance. When an abnormal instance is detected, it is immediately removed from the service pool and automatically restarted. Once recovered, it is automatically added back. Service availability is now proactively maintained by the platform, rather than relying on manual inspections or user reports.

Troubleshooting capabilities have also been systematically enhanced. In production environments, what teams often need most is a complete record of what happened at the time of failure. Previously, this meant logging into the terminal and manually checking logs. v2.2 introduces three types of log access:

Historical logs before restart, allowing you to view the complete output before an instance crashed, instead of losing access to past failure logs after a restart;

Distributed sub-instance logs, allowing you to inspect the output of each node separately in multi-node deployments and quickly identify where the issue occurred;

Ray container logs, allowing you to view Ray container logs directly in the UI without troubleshooting through terminal commands.

Most production troubleshooting workflows can now be completed end to end within the GPUStack UI.

For distributed inference, v2.2 introduces vLLM MP auto-distributed mode.

Previously, GPUStack only supported Ray-based auto-distributed vLLM deployments. MP-based distributed deployments had to be configured manually, and GPUStack could not automatically spin up all distributed instances.

With the rapid evolution of vLLM, the new MP-based distributed mode offers clear advantages over Ray-based auto-distributed deployments in terms of operational overhead and inference performance.

Users can now choose the vLLM auto-distributed deployment strategy that best fits their needs.

Another notable update is support for Multi-LoRA.

In enterprise environments, fine-tuning models for different business scenarios is a common requirement. Previously, each LoRA Adapter had to run as a separate model instance, causing GPU memory overhead to grow linearly with the number of tasks and leading to significant resource waste.

With v2.2, multiple LoRA Adapters can be mounted to the same base model instance and switched dynamically. This allows the same hardware to support more fine-tuned tasks while significantly improving GPU memory utilization.

Token Usage Governance for Model Services

Models may be running, but where exactly are tokens being consumed? This question may not be obvious at an early stage with limited usage. But once multiple teams and applications start sharing the same platform, it quickly becomes an operational pain point.

GPUStack previously supported usage statistics by model and by user, helping teams understand overall consumption trends.

However, these two dimensions were not sufficient for precise attribution.

When different applications and business lines share multiple keys under the same user account, usage cannot be clearly separated, making cost accounting difficult to perform.

v2.2 introduces usage statistics at the API Key level. Token consumption for each key is metered independently, allowing administrators to clearly see which caller is consuming what, and how much. This provides a direct basis for cross-team cost attribution and quota management.

Another important change is giving visibility back to users. Previously, users who wanted to understand their own consumption had to ask administrators to pull the data for them.

v2.2 introduces self-service personal usage queries on the user side. Consumption history by model and by time range can now be viewed directly in the UI, without going through a request process.

Once metering capabilities are in place, token consumption is no longer a black box. It becomes operational data that can support quota allocation, internal chargeback, and cost analysis.

Enhanced Production Deployment Capabilities

Platform capabilities can only be fully realized when supported by a solid deployment experience. In v2.2, GPUStack addresses key gaps in enterprise production deployment across three areas.

Kubernetes has become a mainstream choice for enterprise infrastructure. However, deploying GPUStack in K8s environments previously lacked a standardized cloud-native path.

v2.2 provides an official Helm Chart, enabling installation and configuration through Helm in a single streamlined process. This allows GPUStack to fit directly into existing GitOps workflows and CI/CD systems, significantly reducing the operational cost of deployment and upgrades.

v2.2 also expands database compatibility with support for OceanBase and openGauss, giving teams more flexibility in enterprise deployment environments.

On the network topology side, v2.2 supports a Worker-to-Server one-way access mode.

In cross-region or cross-network-boundary deployment scenarios, many environments make it difficult to establish bidirectional connectivity between Server and Worker.

With one-way networking, Worker nodes only need to access the Server, while the Server does not need to initiate reverse connections. This removes a key networking barrier for unified management of multi-region clusters.

From Compute Pooling to GPU Services

Unified scheduling of heterogeneous compute resources has always been one of GPUStack’s core capabilities — bringing GPUs from different vendors and with different specifications into a single compute pool for unified scheduling and monitoring. Previously, however, this compute pool primarily served one scenario: inference.

Data scientists often need an interactive development environment, while algorithm engineers may need dedicated GPUs for experimentation and debugging. In many teams, these needs were handled outside GPUStack through separate systems, resulting in fragmented resource allocation and metering.

v2.2 introduces the GPU Instance Service, bringing the process of “allocating an isolated GPU environment” under unified platform management.

Users can request isolated GPU instances on demand, specify the GPU vendor, model, and quantity, and select runtime templates that include storage mounts and port configurations. Once the instance is ready, they can access it through SSH or the web.

Usage is metered centrally by the platform and shares the same scheduling and usage system as inference services.

This marks an expansion of GPUStack’s compute service model: the same compute resource pool can serve as the foundation for inference services, while also being exposed as on-demand GPU instances.

With unified scheduling and unified metering, management no longer becomes fragmented simply because resources are used in different scenarios. Future capabilities such as resource scheduling for training and fine-tuning and more fine-grained virtualized compute partitioning can naturally build on this foundation.

The Next Stage of AI Infrastructure Platforms

The evolution of AI infrastructure follows a clear path: from being able to run models, to operating model services reliably, and then to unified management and allocation of the compute resources required by AI.

v2.2 takes a step forward in both dimensions. On the model serving side, deeper inference scenario support, improved instance lifecycle management, a token usage governance system, and enhanced production deployment capabilities provide the foundation for scaled service delivery.

On the compute side, the introduction of the GPU Instance Service expands resource allocation and metering beyond inference scenarios.

Both directions point toward the same goal: making GPUStack an infrastructure foundation that enterprises can truly rely on for AI service delivery.

Get started:

GitHub: https://github.com/gpustack/gpustack

Documentation: https://docs.gpustack.ai

GPUStack v2.2 Enterprise Edition: Coming Soon

The v2.2 open-source release lays the foundation for platform-level operational capabilities.

In more complex enterprise environments, there is another layer of capabilities that the open-source edition is not designed to fully cover. When a platform needs to serve multiple isolated tenants, manage compute consumption down to each API Key and model route, ensure end-to-end high availability, and support Token/GPU billing management, these are the challenges that GPUStack Enterprise Edition is built to address.

GPUStack v2.2 Enterprise Edition is designed for enterprise scenarios that require organization-level governance and commercial operations. It will support multi-tenant isolation, fine-grained quota, rate limiting, and access control, production-grade high availability, resource topology visualization, and billing management. Stay tuned.

If you are interested in the Enterprise Edition, feel free to contact us to learn more and explore early access opportunities.

Contact us: https://gpustack.ai/contact/

GPUStack MaxKB: Build a Powerful and Easy-to-Use Open-Source Enterprise AI Agent Platform

GPUStack — Tue, 10 Mar 2026 02:54:50 +0000

GPUStack × MaxKB: Build a Powerful and Easy-to-Use Open-Source Enterprise AI Agent Platform

Summary: By leveraging GPUStack for efficient model deployment and management, and connecting those models to MaxKB, you can easily build an AI assistant with knowledge base retrieval + intelligent Q&A capabilities.

As AI applications become increasingly common within organizations, more teams are beginning to focus on two core challenges:

How to efficiently manage and deploy local large models
How to quickly build enterprise knowledge bases and AI Agents

If you are looking for solutions to both problems, the combination of GPUStack + MaxKB is well worth exploring.

GPUStack: Focuses on GPU resource management and model deployment, supporting multi-node clusters and multi-model services.
MaxKB: An open-source enterprise knowledge base and AI application platform that enables rapid development of knowledge-based Q&A systems and AI Agents.

By connecting GPUStack-provided model services to MaxKB, you can easily build a practical enterprise AI knowledge assistant.

This article will walk through the entire process from scratch.

📌 What You'll Learn

Deploy the latest GPUStack v2.1.0
Deploy required models in GPUStack
Obtain GPUStack model connection information
Deploy MaxKB
Connect GPUStack models in MaxKB
Practical example: Build a GPUStack documentation knowledge base

Install GPUStack v2.1.0

1. Install GPUStack Server

sudo docker run -d --name gpustack-server \
  --restart unless-stopped \
  -p 80:80 \
  -v gpustack-data:/var/lib/gpustack \
  -v /data/gpustack_cache:/var/lib/gpustack/cache \
  gpustack/gpustack:v2.1.0 \
  --bootstrap-password "123" \
  --debug

After running the command above, open your browser and visit:

http://your_host_ip

You will enter the GPUStack UI.

Default login credentials:

admin / 123

2. Create a Cluster

GPUStack manages worker nodes in units called Clusters.

When deploying GPUStack Server for the first time, you will be prompted to create your first cluster. Click:

Create Your First Cluster

Follow the UI instructions to complete the setup.

You can also go to the Clusters page from the sidebar and click Add Cluster to create one manually.

3. Add a Worker

After creating a cluster, the system will prompt you to Add Worker.

Follow the instructions in the UI.

You can also add one manually via the Workers page in the sidebar.

Run the diagnostic command provided in the guide interface.

If the drivers and container runtime are correctly installed, you will see two OK messages.

If not configured appears, follow the provided links to check dependency documentation and install the missing components according to your environment.

Model Cache Volume Mount: Mount this directory to the model cache directory /var/lib/gpustack/cache.

GPUStack Data Volume: Mount this directory to the data directory /var/lib/gpustack.

Then run the Worker startup command:

sudo docker run -d --name gpustack-worker \
   -e "GPUSTACK_RUNTIME_DEPLOY_MIRRORED_NAME=gpustack-worker" \
   -e "GPUSTACK_TOKEN=gpustack_7b42996d3f5571d5_8181f986537c100369eaa2dfcf6d6359" \
   --restart=unless-stopped \
   --privileged \
   --network=host \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   --volume gpustack-worker-data:/var/lib/gpustack \
   --volume /data/gpustack_cache:/var/lib/gpustack/cache \
   --runtime nvidia \
   gpustack/gpustack:v2.1.0 \
   --server-url http://192.168.50.14 \
   --worker-ip 192.168.50.14

Deploy Models in GPUStack

Click Deployments in the sidebar to open the model deployment page.

If no models are currently deployed, you will see a Deploy Now button in the center of the page.

Click it to enter the Model Catalog, select the desired model, and follow the prompts to deploy it.

Additional deployment methods are available under the Deploy Model menu in the top-right corner.

For this tutorial, we deploy the following three models:

Qwen3-Reranker-4B
Qwen3-Embedding-4B
Qwen3.5-35B-A3B

GPU memory allocation can be adjusted according to your environment.

Deploy Qwen3-Reranker-4B

After deployment, you can test it in the Playground.

Deploy Qwen3-Embedding-4B

After deployment, test it in the Playground.

Deploy Qwen3.5-35B-A3B

Here we additionally set the PYPI_PACKAGES_INSTALL environment variable to upgrade the transformers library.

After deployment, test it in the Playground.

Obtain GPUStack Model Access Information

Open the Routes page from the sidebar.

Click the three-dot menu next to the Route, then select:

API Access Info

Record the following information:

Base URL
Model Name
API Key

Example:

Base URL: http://192.168.50.14/v1

Model Name:
qwen3.5-35b-a3b
qwen3-reranker-4b
qwen3-embedding-4b

API Key:
gpustack_xxxxxxxxxxxxxxxxx

You can create an API Key following the instructions in the UI.

Deploy MaxKB

MaxKB supports one-command Docker deployment:

docker run -d --name=maxkb --restart=always -p 8080:8080 -v ~/.maxkb:/opt/maxkb 1panel/maxkb

Default credentials:

admin / MaxKB@123..

Upon first login, you will be prompted to change the password.

Follow the instructions to update it.

Connect GPUStack Models in MaxKB

In the top navigation bar of MaxKB, select Model.

Click Add Model in the upper-right corner.

Note
API URL and API Key will only appear after entering the Base Model and pressing Enter.

Add the following models in the same way:

qwen3-reranker-4b
qwen3-embedding-4b

For qwen3-reranker-4b, you must enable Generic Proxy:

This is because MaxKB uses the endpoint:

/v2/rerank

After configuration, it should look like this:

Practical Example: Build a GPUStack Documentation Knowledge Base

Open the Knowledge page at the top and click Create to create a knowledge base.

Select Web Knowledge.

Enter the GPUStack documentation URL.

MaxKB will automatically crawl and parse the page content.

After crawling is complete:

Create an AI Agent

Go to the Agent page.

Click Create to create a new Agent.

After completing the configuration, click Publish.

Once published successfully, you can start chatting with the agent.

Chat Demo

Open the chat interface:

Example result:

🙌 Join the GPUStack Community

If you have already started using GPUStack,
or are exploring local large models / GPU resource management / AI infrastructure,
you are welcome to join our community group to exchange practical experience, pitfalls, and best practices.

https://discord.gg/QAzGncGs

No More Token Anxiety: Build an “Unlimited-Use” Local AI Assistant with GPUStack + OpenClaw

GPUStack — Fri, 06 Mar 2026 02:32:29 +0000

Over the past two years, more and more teams have integrated AI into their daily workflows.
But soon, a practical issue emerged:

The more the model is used, the faster Tokens are consumed, and both costs and psychological pressure rise accordingly.

Many people rely on AI to improve efficiency, while at the same time having to “use it sparingly” and “let it think less.”
In the end, AI instead becomes a carefully budgeted consumable.

If AI can run on your own GPU,
without being billed by Token, available for conversation at any time, and running long-term inside collaboration tools,
then it truly feels like a real “work assistant.”

Based on the local model capabilities provided by GPUStack, combined with OpenClaw (supporting multiple collaboration platforms such as WhatsApp, Telegram, Discord, Slack, Lark, etc.) and Telegram,
this article will walk through step by step how to build a truly usable, sustainably running, and almost Token-worry-free local AI assistant.

📌 What This Article Covers

Deploying a model with GPUStack
Creating a Telegram bot application and configuring permissions
Installing, configuring, and key considerations for OpenClaw
First-time authorization and connectivity testing on the Telegram side
Practical example: Let the assistant star the GPUStack project
Built-in assistant commands
Useful OpenClaw commands and resource links

I. Deploy a Model with GPUStack and Prepare Access Information

Before connecting OpenClaw, we need to complete model deployment in GPUStack and obtain the model service access information.

This section will use Qwen3.5-35B-A3B as an example to demonstrate the complete process from
Custom inference backend → Deploy model → Obtain access information.

1. Environment Preparation and Version Information

GPUStack version: v2.0.3
Custom inference backend image: vllm/vllm-openai:qwen3_5
Model weights: Qwen/Qwen3.5-35B-A3B

⚠️ OpenClaw has requirements for the model context window:
Minimum 16K, recommended 128K or above.

2. Configure Custom Inference Backend (vLLM)

In the GPUStack console, go to:

“Inference Backends” → “Edit vLLM” → “Add Version”

3. Deploy the Qwen3.5-35B-A3B Model

Example parameters:

--tensor-parallel-size=2
--mm-encoder-tp-mode data
--mm-processor-cache-type shm
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

If you encounter:

text Error 803: system has unsupported display driver / cuda driver combination

You can try adding the environment variable:

bash LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/lib/x86_64-linux-gnu

4. Obtain GPUStack Model Access Information

Record the following three items:

API Base URL
Model ID
API Key (create it in GPUStack)

II. Create a Telegram Bot

Open Telegram and search for BotFather
Open the BotFather APP

Create a new Bot and fill in the basic information

Copy the Bot Token

For details, please refer to: https://docs.openclaw.ai/channels/telegram

III. Install and Configure OpenClaw

Demo environment: Ubuntu 24.04

1. One-Click Installation

bash curl -fsSL https://openclaw.ai/install.sh | bash

The script will automatically install dependencies such as Node and Git.

2. Interactive Configuration Wizard

Model/Auth Provider Select Custom Provider (Any OpenAI or Anthropic compatible endpoint)

Enter the GPUStack API Base URL / API Key

Select Telegram for Channel

Paste the Bot Token

IV. First-Time Authorization and Testing

Send a message to the bot in Telegram
On first use, it will prompt for Pairing authorization

On the server, run:

bash openclaw pairing approve telegram <Pairing-Code>

V. Practical Example: Let the Bot Star the GPUStack Project

1. Prepare a GitHub PAT

Use Tokens (classic)
Check the repo permission

2. Write to Environment Variables

bash vim ~/.openclaw/.env

Restart:

bash openclaw gateway restart

3. Send a Command to the Bot

Result:

VI. Common Commands

/new: Start a new session
/status: Check bot status
/reset: Reset context
/model: View / switch model

VII. Useful OpenClaw Commands and Resources

Common CLI Commands

bash openclaw logs --follow openclaw doctor openclaw gateway --help openclaw dashboard openclaw tui

Documentation and Ecosystem

📘 https://docs.openclaw.ai
🌐 https://clawhub.ai

Conclusion: When AI Becomes Infrastructure, Not a Consumable

Looking back, the essence of Token anxiety is not that models are expensive, but that AI is treated as an “external consumable resource.”

When models run in the cloud and capabilities are controlled by others,
we become accustomed to careful budgeting, limiting usage, and controlling call frequency.

But when the model truly runs on your own GPU,
when inference capability, context, and tool calls all become part of your infrastructure,
the role of AI changes accordingly—

It is no longer a paid API call each time,
but a readily available, long-term online, continuously evolving work assistant.

This is exactly the significance of combining GPUStack and OpenClaw:
Let AI return from a “cost item” to “productivity.”

If you already have GPU resources,
you might as well try it yourself and truly integrate AI into your daily workflow.

When you no longer worry about Tokens,
you will truly begin to make good use of AI.

🙌 Join the GPUStack Community

If you have already started using GPUStack,
or are exploring local large models / GPU resource management / AI Infra,
you are welcome to join our community group to exchange practical experience, pitfalls, and best practices together.

https://discord.gg/QAzGncGs