Current MoE serving pipelines treat every decode worker as interchangeable, assuming that equalizing request counts suffices to minimize latency. In practice, the weight‑loading cost of each request varies wildly because different inputs activate disjoint expert subsets, and ignoring this fact inflates inference time.
Prefill‑decode (PD) disaggregated serving has historically relied on routers that balance only the number of pending tokens per worker, without considering which experts will be needed later. This load‑centric view leaves a hidden dimension—expert locality—that standard schedulers never exploit.
ELDR cuts median time‑per‑output token by 5.9–13.9 % across three MoE models and two workloads while leaving generated images unchanged. “Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9–13.9% over the strongest of four load‑balancing baselines across three MoE models and two workloads, with model outputs unchanged.” [1]
The router’s premise holds because prefill‑time expert activations predict decode‑time usage. “The correlation is strong for Qwen3-30B-A3B and Gemma-4‑26B‑A4B, and substantial for GPT‑OSS‑120B.” [1] This predictive signal lets the system assign each request to a worker whose cached experts already match its signature.
The improvement is confined to PD‑disaggregated scenarios; ELDR depends on offline K‑means partitions of the signature space and assumes a homogeneous GPU pool up to 40 devices. Outside this regime the routing gains may diminish or require re‑training of the partitioning step.
This suggests that future serving stacks should explore online clustering or hierarchical signatures to accommodate larger, heterogeneous clusters and very short prompts where cache warm‑up overhead dominates.
Practitioners can swap the default vLLM load balancer for ELDR’s locality‑band routing and may achieve up to a double‑digit latency reduction (5.9–13.9%) without retraining any MoE weights. The change is a drop‑in configuration tweak that immediately translates into faster image generation pipelines. [1]
If expert activation patterns become first‑class scheduling metadata, will the next generation of serving frameworks render brute‑force scaling obsolete?
Top comments (0)