Current MoE inference pipelines still stall on expert traffic during batched decoding, limiting latency improvements despite sparsity. SlimWise shows that pruning experts only in the decode phase and reusing prefill KV caches can boost throughput by nearly twofold without perceptible quality loss.
Earlier serving optimizations either prune the full model—including the costly prefilling step—or accept a trade‑off where speed gains come at the cost of generation fidelity. Those approaches sacrifice the rich context built in the prefill stage, leading to noticeable drops in perplexity or output length when the same pruning ratio is applied during both phases.
On Qwen3.6-35B-A3B, SlimWise improves decode throughput by up to 1.81× at 50 % expert pruning with minimal accuracy loss[1]. The system keeps the prefill stage untouched, activates only half of the experts during generation, and still delivers token‑level latency reductions that translate into a roughly 78 % end‑to‑end speedup in typical request patterns.
Without additional training, SlimWise’s KV cache handoff recovers much of the accuracy lost to pruning in many evaluated settings[1]. By feeding the decoder with the full‑model KV cache generated during prefill, the pruned model inherits a high‑quality attention context, so the model’s accuracy remains comparable to the unpruned baseline even when half the experts are disabled.
Because expert pruning preserves the attention architecture and KV cache format, this KV cache handoff requires no KV cache conversion[1]. The unchanged cache layout lets the runtime simply swap the weight mask between phases, eliminating costly data reshaping or additional memory copies that would otherwise erase any latency gains from pruning.
SlimWise’s gains are demonstrated on two MoE backbones and three pruning criteria; performance on larger models or more aggressive pruning ratios remains untested, and the low‑cost distillation stage—while helpful—adds a modest extra training step before deployment. One open question is whether dynamic token‑level expert selection could further tighten the latency–quality curve without requiring a static prune threshold.
Production stacks that serve Mixture‑of‑Experts should replace monolithic decode with a prefill‑decode split and directly feed the full‑model KV cache into a pruned decoder, eliminating the need for costly runtime cache rewrites. Re‑running latency benchmarks under this disaggregated regime will likely reveal hidden speed margins that current end‑to‑end numbers conceal.
Top comments (0)