As GPU prices continue to rise, hardware that once seemed difficult to justify deserves another look. NVIDIA DGX Spark is one such system. Its price-to-performance positioning initially made us hesitant, but under today's procurement constraints, a small inference cluster built from several Spark systems is worth evaluating seriously.
We connected four DGX Spark systems, managed them as one cluster with GPUStack, and served GLM-5.3-Flash-NVFP4 through vLLM. After two days of network tuning, MTP speculative-decoding work, and multi-node troubleshooting, we obtained the following results.
Under our test conditions, single-stream decode throughput increased from 24.3 to 51.3 tok/s, or 2.11× the baseline. Aggregate output throughput at four concurrent requests increased from 72.0 to 104.1 tok/s.
This article focuses on three questions: what network tuning actually improved, why MTP initially failed to start, and which GPUStack multi-node deployment issues are worth avoiding in advance.
All numbers in this article come from this specific four-node cluster. The performance results apply to the model, configuration, and test cases described here. Interface names, paths, and parameters must be adapted to your own environment.
1. Test Environment: Four Spark Nodes with Cross-Node TP=4
Hardware and Network
| Item | Configuration |
|---|---|
| Compute nodes | 4 × NVIDIA DGX Spark with the GB10 Grace Blackwell Superchip |
| Unified memory | 128 GB per node |
| Network adapter | ConnectX-7; the logical interfaces used in this deployment are listed in the appendix |
| Switch | MikroTik CRS804-4DDQ, four 200G access ports, RouterOS 7.19.6 |
| Rail 1 | 192.168.100.0/24 |
| Rail 2 | 192.168.101.0/24 |
One detail about the “dual-rail” wording matters: according to our deployment records, the two logical interfaces on each node shared one 200G physical link; they were not two independent 200G links. Each of the four nodes connected to one physical switch port.
The topology also had an important limitation: the switch did not expose usable DCB/PFC/ECN configuration for this setup, so our tuning was concentrated on the NIC side. We therefore describe the configuration and results for this topology without claiming that it formed an end-to-end lossless network.
Software and Model
| Item | Configuration |
|---|---|
| Orchestration | GPUStack; Server on node01, with four Workers registered in one cluster |
| Inference engine | vLLM with the B12X attention backend |
| Container image | eugr/spark-vllm-b12x:latest |
| Model | GLM-5.3-Flash-NVFP4 |
| Quantization and cache | NVFP4 weight quantization and FP8 KV cache |
| Parallelism | TP=4 across four nodes; PP=1 |
GPUStack handled model configuration, Worker scheduling, and instance management. The lower-level inference and communication work centered on vLLM, MTP, and NCCL.
2. Fixing the Network First: Making Better Use of RoCE with NCCL
Cross-node tensor parallelism makes inter-node communication unavoidable. We started with MTU consistency, NIC QoS, and NCCL interface selection.
2.1 Use a Consistent MTU
We set the host MTU to 9000 and persisted it through netplan. On the switch, we used the platform-specific setting mtu=9216; the recorded effective value was 9000.
The important part is to verify the effective configuration and connectivity across the entire path. Changing only the host configuration is not enough.
2.2 Configure NIC-Side QoS
The following commands show one interface from our deployment:
mlnx_qos -i enp1s0f0np0 --trust dscp
mlnx_qos -i enp1s0f0np0 -f 0,0,0,1,0,0,0,0
This configuration used DSCP trust mode and enabled PFC for priority 3.
One easy-to-miss detail caused confusion during early testing: mlnx_qos -p configures priority mapping, while -f controls PFC. Always read the configuration back after applying it and confirm that the effective state matches the intended one.
2.3 Let NCCL Use Both Logical RDMA Interfaces
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_IB_TC=104
NCCL_IB_QPS_PER_CONNECTION=4
NCCL_IB_SPLIT_DATA_ON_QPS=1
In this setup, NCCL_IB_TC=104 corresponded to DSCP 26. Verify that it maps to the intended priority and queue under your NIC QoS configuration.
What Did the Network Tuning Improve?
In our recorded four-node, 256 MB AllReduce test, bandwidth increased from 4.0 GB/s under TCP fallback to 22.9 GB/s, or approximately 5.7×. The single-rail comparison measured 11.4 GB/s, while the dual-rail configuration reached approximately 2.0× that result.
These are bandwidth values from our NCCL test records. They should not be interpreted as physical port line rate.
Time to first token (TTFT) also improved substantially in the long-input workload, as shown in Section 6. Network tuning did not double every metric, however. In our records, its direct effect on decode throughput was only about 2%–5%.
Network tuning removed the communication bottleneck. The much larger improvement in single-stream decoding came after MTP was working.
3. Getting MTP Working: The Key to Doubling Single-Stream Decode Throughput
GLM-5.3-Flash includes an MTP, or Multi-Token Prediction, module. We attempted to use it for speculative decoding, but this environment exposed two separate problems: weight loading and KV-cache layout compatibility.
3.1 MTP Weights Were Loaded with the Wrong Quantization Format
In this checkpoint, the MTP layer was layer 45, and its name did not explicitly contain mtp. Its expert weights were actually BF16, but the quantization configuration's ignore list did not exclude the layer. The draft side therefore allocated it as NVFP4, resulting in a 256/512 dimension mismatch.
We fixed the issue by adding quantization-exemption rules to the model configuration on all four nodes.
First, back up the configuration file:
cd /models/GLM-5.3-Flash-NVFP4
cp -p config.json config.json.bak-ignore45
Then append the following four strings to both of these arrays:
quantization_config.ignore-
text_config.quantization_config.ignore
[
"model.language_model.layers.45",
"model.language_model.layers.45*",
"model.layers.45",
"model.layers.45*"
]
These are entries to append. Do not replace the existing ignore arrays with this snippet. Layer 45 is specific to the checkpoint used in our test; verify the correct layer for other weight versions.
After updating all four nodes, we deleted the corresponding model instance in GPUStack to trigger instance recreation and weight reloading. This interrupts the instance, so schedule it during an acceptable maintenance window.
After the change, the MTP draft path loaded the layer as BF16 and the dimension mismatch disappeared. To roll back, restore the backup on every node and recreate the instance.
3.2 The KV-Cache Layouts Were Incompatible
Once the weights loaded, a second problem appeared between the FlashInfer backend used by the main model and the KV-cache layout used by the MTP draft path.
The layout constraints conflicted. Some layouts could not split a 2,560-token management block into 256-token kernel blocks, while layouts that could be split were rejected by the parser. Initialization could not proceed.
We initially built a locally patched image to relax the layout declaration. The configuration that ultimately worked was simpler: use B12X as the attention backend for both the main model and the draft path. With this configuration, the management and kernel blocks no longer required the problematic split.
--attention-backend=B12X
--speculative-config={"method":"mtp","num_speculative_tokens":3,"moe_backend":"humming","attention_backend":"B12X"}
We then repeated the test with a base image that did not contain our local layout patch. The same configuration ran successfully and showed no meaningful performance difference in our records, so we removed the patch.
The lesson was straightforward: when a backend compatibility issue appears, first check the support boundaries of the available backends before committing to a local patch.
3.3 Should the Speculative Depth Be 3 or 5?
Once MTP was running, we still had to decide how many tokens to predict per step.
The reference recipe used num_speculative_tokens=5. In our four-node TP=4 environment, we compared depths 3 and 5. Each configuration ran three times, and the table reports the median:
| Test case | Depth 3 | Depth 5 | Relative change at depth 5 |
|---|---|---|---|
| Single-stream, long-output decoding | 51.9 tok/s | 48.7 tok/s | −6.2% |
| Four-request aggregate output | 110.0 tok/s | 96.7 tok/s | −12.1% |
The result was clear: depth 3 was faster in this test.
Our working hypothesis is that the additional computation from deeper speculation was not offset by an increase in accepted tokens. Confirming that explanation would require per-layer acceptance-rate and timing data; throughput alone is not enough to prove the cause.
We therefore selected depth 3. A reference recipe provides a starting point, but workload-specific A/B testing should determine the final setting.
Performance with MTP Enabled
The following results come from the same benchmark suite. They compare the no-MTP baseline with the combined MTP × 3 and dual-rail configuration:
| Test case | No-MTP baseline | MTP × 3 + dual rail | Final/baseline |
|---|---|---|---|
| Single-stream, short-output decoding | 24.4 tok/s | 50.0 tok/s | 2.05× |
| Single-stream, long-output decoding | 24.3 tok/s | 51.3 tok/s | 2.11× |
| Four-request aggregate output | 72.0 tok/s | 104.1 tok/s | 1.45× |
| Long-input decode throughput | 24.4 tok/s | 51.9 tok/s | 2.13× |
These numbers represent the benefit of the combined configuration. The entire gain should not be attributed to one parameter. The speculative-depth A/B test above came from a different run, which explains the small differences in absolute values.
For functional regression, all 17 test prompts passed, including character-counting cases. This provides a basic functional check, but it is not equivalent to a full output-distribution equivalence study.
4. A Silent Startup Hang: Gloo Selected the Loopback Address
One failure was particularly difficult to diagnose from the dashboard. All four vLLM rank processes stopped during initialization, GPU utilization remained at 0%, and the logs contained no explicit error.
We captured the call stacks with py-spy and found the processes waiting in:
gloo::transport::tcp::Pair::waitUntilConnected
The relevant Worker processes were listening only on 127.0.0.1.
Further investigation showed that /etc/hosts inside the instance container mapped the hostname to a loopback address. When Gloo selected an address from the hostname, it attempted to use that loopback address for cross-node communication, preventing initialization from completing.
The fix was to explicitly select the network interface for Gloo:
GLOO_SOCKET_IFNAME=enp1s0f0np0
Setting only NCCL_SOCKET_IFNAME did not resolve this failure. GLOO_SOCKET_IFNAME had to be configured separately.
We retained this variable in the final cluster configuration. For similar initialization hangs, check the call stacks, listening addresses, and hostname resolution inside the container together.
5. Six Operational Lessons from a Multi-Node GPUStack Deployment
GPUStack placed all four nodes under one configuration and instance-management workflow, which made it easier to repeat tests across different combinations. The following operational lessons were worth keeping.
This section describes observations from our deployment. API paths, log locations, and container behavior may vary by GPUStack version.
5.1 Read the Configuration Back After an Update
We used three main API operations:
GET /v2/models/<model_id>
PUT /v2/models/<model_id>
DELETE /v2/model-instances/<instance_id>
They read model configuration, update it, and delete an instance to trigger recreation. The paths are illustrative; actual calls also require the appropriate authentication and request body.
A useful habit is to run GET after PUT and verify that every critical field was saved as intended.
5.2 Verify the Effective Custom Image Separately
In our deployment, we repeatedly observed image_name becoming None after a PUT, after which follower nodes used an unexpected image.
We handled this by reading the configuration back, restoring the field when necessary, resubmitting it, and verifying the image actually running on every node. This is a troubleshooting record from our environment, not a claim that every GPUStack version behaves the same way.
5.3 Use Local Logs to Track Follower Progress
In this deployment, the Web dashboard logs mainly reflected the leader instance. When a follower remained in “Preparing model files,” we inspected the log inside its Worker container:
docker exec gpustack-worker cat /var/lib/gpustack/log/serve/INSTANCE_ID.0.log
Confirm the actual path and filename suffix for your instance. Restarting the affected Worker recovered one failure in our tests, but doing so affects services on that Worker. Preserve the logs and assess the impact first.
5.4 Scope Residual-Container Cleanup Precisely
After deleting an instance, old containers occasionally remained on follower nodes and affected the next follower health check. A new instance could also begin creating containers almost immediately, making broad cleanup filters dangerous.
We separated discovery from deletion: list candidate containers first, verify that they belong to the old instance, and only then delete the exact container ID.
# List candidates first. The name filter only helps locate them;
# verify the instance identity separately.
docker ps -a --filter 'name=<OLD_INSTANCE_UNIQUE_SUFFIX>' \
--format 'table {{.ID}}\t{{.Names}}\t{{.Status}}'
# After verification, replace CONTAINER_ID with the exact old container ID.
docker rm -f CONTAINER_ID
5.5 Verify the Container Runtime Requirements for RDMA
The instance containers we inspected ran in privileged mode, used host networking, and had access to /dev/infiniband, providing the runtime conditions needed for RoCE inside the container.
Those conditions alone do not prove that RDMA communication is working. We still used the container configuration, NCCL logs, and communication benchmarks together to verify the effective transport.
5.6 Use a Custom Image When the Backend Requires It
Under GPUStack's inference-backend configuration, we specified:
Image: eugr/spark-vllm-b12x:latest
Entry command: vllm serve
This provided the required B12X-enabled vLLM environment. We then managed the test combinations through model parameters and environment variables. As with every other configuration change, verify the image and launch arguments on the running instances.
6. Final Configuration and Results
We settled on the following configuration:
- Four nodes with TP=4 and PP=1.
- B12X as the attention backend for both the main model and MTP draft path.
- MTP speculative depth 3.
- Two logical RDMA interfaces for NCCL and an explicitly selected Gloo interface.
-
gpu-memory-utilization=0.85andmax-model-len=500000.
We did not retain kv_cache_memory_bytes=8G from the reference configuration, nor did we raise gpu-memory-utilization to 0.87. We preferred additional operating headroom over maximizing allocated memory.
Performance Results
Application-level results are the median of three bench.py runs. NCCL bandwidth is listed separately so that communication bandwidth is not confused with model output throughput.
| Application metric | Baseline | Final result |
|---|---|---|
| Single-stream, long-output decoding | 24.3 tok/s | 51.3 tok/s, 2.11× |
| Four-request aggregate output | 72.0 tok/s | 104.1 tok/s, 1.45× |
| TTFT in the long-input case | 1.73 s | 0.12 s, approximately 93% lower |
The TTFT result applies to this particular test case. Because the article does not fully specify input length, cache-hit state, and warm-up conditions, it should not be extrapolated to arbitrary long-input requests or attributed to one network parameter alone.
| Communication metric | Baseline | After tuning |
|---|---|---|
| Recorded NCCL AllReduce bandwidth | 4.0 GB/s | Approximately 19–23 GB/s |
We also raised max-model-len from the previously recorded 32K tokens to 500,000 tokens. This is a configuration limit, not evidence that we completed end-to-end validation of one 500K-token request or four concurrent 500K-token requests.
Restart Stability
Three scripted restart tests all succeeded on the first attempt. Time to readiness was approximately 12.5–13.5 minutes, including about five minutes for weight loading, seven minutes for B12X kernel tuning and related initialization, and the remaining time for other startup steps.
These three runs were sufficient for our configuration sign-off, but longer-duration stability still requires continued observation.
Final Takeaways: Give Every Important Choice a Comparison
Two lessons stood out after the two-day tuning effort.
First, select a backend combination that is actually compatible. B12X resolved the MTP layout issue in our environment and allowed us to remove the local patch used during troubleshooting.
Second, use controlled comparisons to choose parameters. Single rail versus dual rail, speculative depth 3 versus 5, and no MTP versus the final combined configuration answered different questions. Testing them separately made it easier to identify where each gain came from.
GPUStack gave us one multi-node management plane for the experiment, while the final performance came from the network, inference backend, and model configuration working together. If you are experimenting with your own Spark cluster, we hope this record provides a useful troubleshooting path.
Share your hardware configuration, benchmark method, or GPUStack multi-node deployment experience in the comments.
Appendix: Deployment Configuration
The values below reproduce our environment. Verify your GPUStack and model versions, checkpoint layout, interface names, and available memory before using them.
Basic Model Configuration
| Setting | Value used in this test |
|---|---|
| Model source | local_path |
| Local path | /models/GLM-5.3-Flash-NVFP4 |
| Distributed inference | Cross-Worker distributed inference enabled |
| Backend version label | vLLM (0.31.0-spark-custom, preserved from the deployment record) |
| Image | eugr/spark-vllm-b12x:latest |
| Replicas | 1 |
backend_parameters
Each line below is one backend parameter. Pass the JSON value for --speculative-config as one complete argument.
--distributed-executor-backend=mp
--tensor-parallel-size=4
--trust-remote-code
--max-model-len=500000
--gpu-memory-utilization=0.85
--max-num-seqs=4
--max-num-batched-tokens=8192
--kv-cache-dtype=fp8
--block-size=256
--mamba-cache-mode=align
--no-enable-flashinfer-autotune
--enable-auto-tool-choice
--tool-call-parser=glm47
--reasoning-parser=glm47
--attention-backend=B12X
--speculative-config={"method":"mtp","num_speculative_tokens":3,"moe_backend":"humming","attention_backend":"B12X"}
Environment Variables
Enter these as separate key-value pairs in GPUStack:
VLLM_USE_AOT_COMPILE=1
VLLM_USE_MEGA_AOT_ARTIFACT=1
VLLM_USE_V2_MODEL_RUNNER=1
CUTE_DSL_ARCH=sm_121a
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
VLLM_SSM_CONV_STATE_LAYOUT=DS
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_IB_TC=104
NCCL_IB_QPS_PER_CONNECTION=4
NCCL_IB_SPLIT_DATA_ON_QPS=1
NCCL_SOCKET_IFNAME=enp1s0f0np0
GLOO_SOCKET_IFNAME=enp1s0f0np0
We additionally enabled these variables during troubleshooting:
NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS=INIT,NET
Pre-Launch Checklist
- Apply the layer 45 quantization exemptions required by this checkpoint on all four nodes, and retain a backup of the original configuration.
- Read back and verify the effective MTU, DSCP trust, priority mapping, and NIC-side PFC settings. Confirm that persistent configuration matches runtime state.
- Confirm that the RDMA and Gloo interface names match the actual devices on every node.
- This configuration does not pin
NCCL_IB_GID_INDEX; NCCL selects it automatically. Check the selected value in the startup logs. - After recreating the instance, verify the effective image and parameters on every node, then run communication, health, and real-inference checks.
Top comments (0)