A 256-expert MoE promises 32x. At batch 1 you collect 12.5x.

Two posts ago I computed what a sparse model reads per token. DeepSeek-V3 at FP8 reads 36 GB, a sixteenth of what it stores, and that read is the reason a 671B model is servable at all.

The bound in that piece quietly assumed the eight experts a token picks are spread evenly across the machine. They are not. On eight GPUs one of them always draws more experts than the others, and a step takes as long as the slowest GPU takes. That gap is not a rounding error at the batch sizes people actually serve, so this piece computes it.

The setup

256 experts, top-8, 8 GPUs, 32 experts resident on each. Every decode step, each token is handed to eight experts somewhere in the node. A GPU holding three of a token's experts reads three experts' worth of weights. The step ends when the last GPU finishes.

If the eight experts scattered one per GPU, every GPU would read exactly one expert per token and you would collect the full 32x discount over a dense model of the same total size. Eight balls into eight bins does not leave one in each bin. It leaves a bin with two and often three.

So the useful quantity is not the average load. It is the maximum. Divide the ideal per-GPU load by the realized maximum and you get the fraction of the sparsity discount you actually collect.

What the distribution costs

Monte Carlo over the layer, uniform routing, 300k token-assignments per row.

tokens/step	ideal experts/GPU	busiest GPU	tax	discount collected
1	1.00	2.56	2.56x	39%
4	4.00	6.99	1.75x	57%
8	8.00	12.16	1.52x	66%
32	32.00	40.09	1.25x	80%
128	128.00	144.03	1.13x	89%
512	512.00	543.36	1.06x	94%
2,048	2,048.0	2,113.8	1.03x	97%
8,192	8,192.0	8,315.7	1.02x	98%

Read the first row. A single decode step with a batch of one collects 39% of the discount the config promised. The model was sold as 32x sparser than dense and at batch 1 it behaves like 12.5x. Nobody serves at batch 1, but everybody serves at some batch, and the difference between 80% and 94% is the difference between a throughput number from a benchmark and one from your deployment.

It collapses to one line

For eight groups that whole table is

tax ≈ 1 + 1.4 / sqrt(tokens per step)

which reproduces the rows within 3% from batch 8 upward. Invert it and you get the number that decides your serving config: to hold the straggler under 5% at eight GPUs you need about 800 tokens per step. Under 2%, about 4,900.

Tokens per step is not the same as sequences in flight. A prefill of a 4k prompt contributes 4,096 token-assignments in one shot, so prefill never pays this tax; at 4k the busiest GPU is within 2% of the mean. A decode step contributes one token per sequence. So the 800-token number, in decode, means 800 concurrent sequences. Below that your MoE is running with a visible straggler, and if you are serving at 20 users you are running at 1.3x tax, permanently.

That is the shape of the thing. Sparsity is a discount on weight reads, batching pays the discount back, and the tax is charged per step rather than per token, so it hurts exactly the deployments that care about latency.

The correction to my own number

Two pieces ago I wrote that 8xH200 sustains about 1,060 tokens per second aggregate for DeepSeek-V3, so at 50 tok/s per user the ceiling is 21 concurrent sequences. The 1,060 assumed perfect balance. At 21 sequences the tax is 1.32, so the honest figure is about 800 tokens per second aggregate, and 16 users at 50 tok/s each. The bandwidth math was right. The scheduling assumption inside it was not, and it cost 25% of the answer.

Wider expert parallelism makes it worse

Expert parallelism looks like a way out: split 256 experts over more devices and each GPU has less to do. It has less to do and it also has less to average over, and the second effect wins.

Tax at 21 sequences, same 256 experts, top-8:

GPUs	experts/GPU	tax at 21 seq
8	32	1.32x
16	16	1.58x
32	8	1.98x

At EP=32 the busiest GPU does twice the mean work, and you pay for that in latency as well as in all-to-all volume. This is the reason production MoE stacks spend memory on redundant expert copies and whole subsystems on rebalancing placement. My arithmetic here does not model that fix; it only shows what the fix is buying.

The all-to-all is not the problem people think

The standard objection to expert parallelism is fabric traffic, and it is falsifiable with a multiplication.

DeepSeek-V3 dispatches hidden_size 7168 in BF16, so one token's eight experts move 8 x 7168 x 2 = 114 KB out and the same back, per MoE layer. There are 58 MoE layers. That is 13.3 MB moved per token, round trip, every token. At 1,060 tokens per second that is 14 GB/s.

An 8xH100 NVSwitch domain carries about 7,200 GB/s aggregate. 14 GB/s is 0.2% of it. Even spread over 32 GPUs on 50 GB/s InfiniBand, the same traffic is under 1% of the fabric.

All-to-all is a real engineering problem for overlapping and for tail latency, and it is not a bandwidth problem on a node. The straggler is. If you are deciding whether to widen EP, the traffic argument will tell you it is fine and the load-balance argument will tell you to stop.

What you cannot batch away

At batch 32 the union of distinct experts touched per step is 163 of 256, so each token effectively reads 5.1 experts' weights instead of 8. Batching makes MoE cheaper per token and the tax makes the step later, and both come from the same repetition. There is a floor: the memory-bound regime ends around the ridge batch of 296 on an H100 at BF16, half that at FP8, and past it you are comparing FLOPs instead of bytes, where the same imbalance reapplies to expert FLOPs on the busiest GPU.

The floor is what matters for small models. Qwen3-30B-A3B has 128 experts and reads 6.0 GB per token, which on 8xH100 is the 4,186 tok/s aggregate figure from the earlier table. Same 8 GPUs, 16 experts each, and at batch 8 the tax is 1.5, so the honest number is about 2,750 tokens per second, or 55 users at 50 tok/s instead of 83. Sparse models are aggressive about small batches because their whole advantage is a read they cannot amortize.

If you serve fewer than a few hundred sequences, the arithmetic says run tensor parallelism and a dense model, or run expert parallelism and accept a quarter of your throughput in straggler. What it does not say is that you can have the 32x and the low latency.

The script

import random

E, K, G = 256, 8, 8          # experts, top-k, expert-parallel groups
EPG = E // G

def run(T, trials=300, seed=3):
    rng = random.Random(seed)
    tot_max = tot_mean = 0.0
    for _ in range(trials):
        load = [0] * G
        for _ in range(T):
            for e in rng.sample(range(E), K):
                load[e // EPG] += 1
        tot_max += max(load)
        tot_mean += T * K / G
    return tot_mean / trials, tot_max / trials

for T in (1, 4, 8, 32, 128, 512, 2048, 8192):
    m, x = run(T, trials=max(24, 300_000 // T))
    print(f"{T:6d}  ideal {m:8.1f}  busiest {x:8.1f}  "
          f"tax {x/m:5.2f}  collected {100*m/x:5.1f}%")

Change E, K and G to your model and sharding. The shape does not change and the tax is monotone in G, which is the part I would check first if I were sizing a deployment.

The rest of this series, including the bytes-per-token piece this one corrects, is in my archive.

Numbers are computed, not quoted. If one is wrong, say which and I will re-derive it.