DEV Community

ram.mehta.1899@gmail.com
ram.mehta.1899@gmail.com

Posted on Originally published at rammehta1899.github.io

Serving 753B MoE Across Nodes: vLLM & RDMA Limits

Serving massive frontier models like GLM-5.3 NVFP4 across multiple hosts requires splitting model weights using tensor parallelism (TP=2) over RDMA interconnects. Benchmarking two NVIDIA DGX Stations running vLLM over ConnectX-8 RDMA demonstrates strong baseline performance, reaching a peak output of 607.8 tokens per second at concurrency 16 on coding tasks.

However, scaling to concurrency 32 triggers a severe performance cliff: output throughput drops to 385.0 tokens per second while mean request latency skyrockets to 453.5 seconds. Inter-node layer synchronization overhead saturates network links under high load, proving that platform teams must enforce strict concurrency limits on multi-node serving endpoints.


Read the full article: Serving 753B MoE Across Nodes: vLLM & RDMA Limits

Top comments (0)