Token‑level routing enables a dedicated serving stack to extract up to sixty‑four times more tokens per second compared to existing baselines, without retraining any models.
Historically, production systems have relied on coarse‑grained routing at the session or query level, treating each request as an atomic unit and dispatching it to a single LLM. Academic work showed that fine‑grained token decisions could improve efficiency, yet existing inference engines were built around single‑model assumptions and suffered from step desynchronization and batch admission delays.
TokenRouter delivers between 2.01× and 64.15× higher decoding throughput than any existing serving baseline across diverse routing algorithms, workloads, and model pairs. “Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01–64.15 higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing.” [1]
End‑to‑end latency drops by a comparable factor, ranging from 2.03× to 63.64× on standard serving workloads. “TokenRouter also reduces the end-to-end latency of Std. Serving by 2.03–63.64.” [1]
The system’s gains hinge on a delayed‑batching scheduler whose optimal hyperparameters are derived from a mathematical throughput model, meaning performance can vary if those knobs are mis‑tuned for a new workload. Moreover, TokenRouter launches a subserver for each LLM; it remains an open question whether the same scaling factors hold under dozens of heterogeneous experts.
If these throughput and latency improvements persist in production, the default serving stack for long‑context applications should be replaced by a token‑level router, and any benchmark that reports raw latency on vanilla batch‑first pipelines becomes obsolete.
Top comments (0)