DEV Community

Cover image for How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job
Mayur Agnani
Mayur Agnani

Posted on Edited on Originally published at theaicommit.com

How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job

A mixture-of-experts model reads part of its weights on every step, and the mix of prompts in the batch decides which part. Put similar prompts next to each other and the same job finishes sooner.

Engineer's view. This is a batch job that touches records in arrival order. Each record needs a different index page, so the cache keeps reloading pages. Sort by key and each page loads once per batch. A mixture-of-experts model loads expert weights the same way, so a mixed batch pays for reads nobody shares.

TL;DR. A mixture-of-experts model reads only the experts the batch asked for, and similar prompts ask for the same ones. Sort an offline job by topic, and each step reads less, so the job finishes sooner.

Explain Like I'm 5

A library has a hundred shelves. A helper runs to fetch books for everyone at one table. If the table holds a cook, a pilot and a poet, the helper runs to many shelves each round. If everyone at the table is a cook, they mostly want the same cooking shelf. The helper makes fewer trips, and the table gets its books sooner. Nobody reads faster. The seating just changed.

The Problem

You have shipped this bug before, in a system with no AI in it.

A nightly job processes a million records in the order they arrived. Each record looks up a row in a big table. Records from the same customer sit on the same index pages, but arrival order scatters them. So the database keeps loading a page, evicting it, and loading it again an hour later. The job is slow, and the code is fine. Someone sorts the input by customer ID, and the job runs twice as fast. Nothing about the work changed. Only the order did.

A mixture-of-experts model has the same bug inside every decode step. Each layer holds many small blocks of weights, called experts. Each token uses only a few of them. Qwen3-30B-A3B has 128 experts per layer, and each token picks 8. But a server runs many requests together as one batch. On each step it must read every expert that any request in the batch picked.

That read is the expensive part. Producing one token per request is limited by how fast the GPU reads weights from memory, not by arithmetic. A random batch of 32 Qwen requests touches about 94 of the 128 experts in a layer. Code prompts favor one set of experts, French prompts another. Mix them, and the step reads both sets.

The fix: before an offline batch job, sort the prompts so similar ones run together. Then each step reads fewer experts. Hugging Face measured this on unchanged vLLM and got 5 to 17% more throughput.

Read the full post on The AI Commit

How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.


Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.

Top comments (0)