A 671B model reads 36GB per token and still needs 671GB of HBM

Two numbers describe DeepSeek-V3 and they look like a contradiction. Decoding one token reads about 36 GB of weights. Storing the model takes 671 GB. If you have only ever sized dense models, those two figures should be the same number, because a dense model reads all of its weights to emit one token.

Mixture-of-experts breaks that identity on purpose. It is worth knowing exactly how much it breaks it, because the break is asymmetrical and most capacity planning carries the wrong half.

What MoE buys, derived from the config

Take the published config.json for three sparse models and compute the parameters that actually participate in one token: attention on every layer, plus the top-k experts and the shared expert, plus the embedding. Nothing quoted, everything counted.

Model	Total params	Active per token	Stored (weights)	Read per token	Read / stored
Qwen3-30B-A3B (BF16)	30.5B	3.0B	61 GB	6.0 GB	0.099
Qwen3-235B-A22B (BF16)	235B	21.5B	470 GB	43 GB	0.092
DeepSeek-V3 (FP8)	671B	36.0B	671 GB	36 GB	0.054
Llama-3.1-8B (BF16), dense	8.03B	8.03B	16.1 GB	16.1 GB	1.000

My script prints 36.0B active for DeepSeek-V3 where the paper says 37B. The gap is the multi-token-prediction layer in the checkpoint, which I do not count. Every other row lands within 2% of its published active size, which is my check that the arithmetic below is not invented.

Read the last column. A sparse model reads between a tenth and a twentieth of what it stores. That is not a small effect, and it is the whole reason a 671B model is servable at all: on a bandwidth-limited decode step it behaves like a 36B model.

It does not fit like a 36B model. That is the asymmetry, and it is not adjustable. Bandwidth scales with the read, capacity scales with the store, and MoE only moves the first one.

The ceiling is set by the read

Decode reads active weights once per token and performs two multiply-adds per active parameter, so the numbers that matter are bytes per token and the machine's bandwidth.

Model	Bytes read / token	H100, 4k ctx	H100, 128k ctx	8×H100	8×H200
Qwen3-30B-A3B	6.0 GB	523 tok/s	177	4,186 tok/s	5,998
Qwen3-235B-A22B	43 GB	76	49	612	877
DeepSeek-V3 (FP8)	36 GB	92	74	739	1,058
Llama-3.1-8B	16.1 GB	202	101	1,615	2,314

The single-card columns are honest only for the 8B. For the other three, the weights live across a node, so read those columns as the tokens per second you get per unit of aggregate bandwidth. The two node columns are the useful ones: that is the total decode throughput a node can sustain at short context, and it is the number you divide by your target per-user speed to get a batch ceiling. On 8×H200, DeepSeek-V3 sustains about 1,060 tokens per second in aggregate, so if you want 50 tok/s per user you cap at roughly 21 concurrent sequences.

Three things fall out of that table and none of them are what the marketing copies say.

Qwen3-30B-A3B beats Llama-3.1-8B on throughput by 2.6x while storing 3.8x the weights and a large lead in quality. Its read per token is 6.0 GB against 16.1 GB. Sparsity bought it a smaller read than a dense model less than a third its size.

DeepSeek-V3 reads half of what Qwen3-235B-A22B reads, despite three times the stored parameters. FP8 halves the bytes, and MLA halves the cache (more on that below). Both levers act on the read.

Nothing here fixes capacity. Look at the store column again. That is the next section.

The capacity bill arrives up front

DeepSeek-V3 at FP8 needs 671 GB before the first token. There is no batch size, no paging trick, and no amount of HBM efficiency that removes weights from the read path; every expert must be resident or reachable.

On 8×H200 that is 671 GB of 1,128 GB, leaving 457 GB for KV cache and activations. It fits. It does not fit comfortably, and it is the tightest constraint in that deployment, not the cache.

The dense comparison is worse. A 671B dense model in BF16 is 1.34 TB and does not fit on 8×H200 at all. At FP8 it is 671 GB, exactly where the sparse model already sits, and it would read 671 GB per token instead of 36. Sparsity buys the multiply, not the storage.

So the honest framing for a MoE deployment is: you are buying a bandwidth discount with a capacity tax. Budget the capacity first. If the node holds the weights with room left over, the model serves; if it does not, no batching policy rescues it. This is the mirror image of the last piece in this series, where KV compression bought capacity at almost no cost in speed. MoE buys speed at the cost of capacity.

One more count that surprised me. DeepSeek-V3's cache is 69 KiB per token, against 128 KiB for Llama-3.1-8B. The 671B model carries a cache half the size of the 8B's, because MLA stores one compressed latent of 576 elements per layer instead of a key and a value per head. At 128k context, that is 8.8 GB per sequence. On a node that already paid 671 GB for weights, the cache is rounding error. This is why the long-context economics of DeepSeek-V3 look nothing like the same math applied to a GQA model of the same size.

FP8 halves the read and halves the batch you need

From the previous piece, the batch at which decode crosses from memory-bound to compute-bound is ridge × bytes_per_param / 2. At BF16 that is the ridge number itself, about 296 on an H100. At FP8 it is 148.

FP8 does not reduce FLOPs. It reduces the bytes those FLOPs wait on, so you reach compute-bound with half the batch. Applied to a sparse model the effect compounds, because the read was already a tenth of the store. DeepSeek-V3 hits the ridge at batch 148 instead of 296, and 148 sequences is roughly the ceiling of what a node can hold anyway once you add cache.

That is a rare case of the two limits landing near each other, and it is why FP8 on MoE is not just a memory optimization. It changes which resource you run out of first.

Batching an MoE does not amortize the way you expect

Here is where the clean arithmetic above starts to lie, and it is worth saying so in the same article rather than in a footnote.

With a dense model, batch 32 reads the weights once and produces 32 tokens. The read per token falls by 32. Pure win.

With a sparse model, each token in the batch routes to a different set of 8 experts out of 256. The decode step must stream the union of every expert touched, not the 8 one token would touch. If routing were independent across tokens, the expected unique experts per layer is 256 × (1 − (248/256)^B):

Batch	Unique experts hit / layer	Read multiplier vs batch 1
1	8.0	1.00×
4	30.5	3.82×
8	57.4	7.18×
32	163.3	20.4×

So at batch 32 an MoE reads about 20x more weight bytes per step than a single sequence does, while a dense model reads the same bytes it always did. The throughput gain from batching is real but it flattens far earlier, and the aggregate numbers in my table are short-context, batch-1 best cases.

Real routing is not independent. Tokens from the same domain route to overlapping experts, so the true union is smaller than this bound, and expert-parallel placement turns most of the extra reads into a network exchange between GPUs rather than a local HBM read. That is the actual shape of MoE serving: past a certain batch, the bottleneck moves from HBM to NVLink and then to the interconnect between nodes, and the number you should be watching stops being bytes per token.

I have not measured any of this on hardware. It is arithmetic from public configs, plus an independence assumption that I have flagged as a bound rather than a measurement. If you run a sparse model and have the routing histogram, you can replace my estimate with your own and I would like to see it.

Check it on your model

The script reads a config's shape and prints the three numbers that decide a deployment: stored bytes, bytes read per token, and the aggregate bandwidth you need for a target throughput.

GiB = 1024**3

# (label, params_b, bytes_per_param, layers, kv_heads, head_dim, active_params_b)
MODELS = [
    ("Qwen3-30B-A3B  BF16",  30.5,  2, 48,  4, 128,  3.0),
    ("Qwen3-235B-A22B BF16", 235.0, 2, 94,  4, 128, 21.5),
    ("DeepSeek-V3  FP8",     671.0, 1, 61, None, None, 36.0),  # MLA: 576-element latent
    ("Llama-3.1-8B  BF16",     8.03, 2, 32, 8, 128,   8.03),
]

NODE_BW = {"8xH100": 8 * 3.35e12, "8xH200": 8 * 4.80e12}

for name, total, bpp, layers, kvh, hd, active in MODELS:
    store = total * 1e9 * bpp
    read = active * 1e9 * bpp
    kv = (layers * 576 * 2) if kvh is None else (2 * layers * kvh * hd * bpp)
    print(f"\n{name}")
    print(f"  store {store/1e9:7.1f} GB   read/token {read/1e9:6.1f} GB   "
          f"kv/token {kv/1024:5.0f} KiB   read/store {read/store:.3f}")
    for node, bw in NODE_BW.items():
        print(f"  {node}: {bw/(read + 4096*kv):7.0f} tok/s aggregate at 4k context")

# Expert-union bound for a 256-expert, top-8 layer
E, k = 256, 8
for B in (1, 4, 8, 32):
    uniq = E * (1 - (1 - k / E) ** B)
    print(f"  batch {B:3d}: {uniq:6.1f} experts touched, read x{uniq/k:5.2f}")

Run it on your own config and the columns change. The shape does not: read is set by active parameters and precision, capacity by total parameters, and only one of those scales with your batch.

The rest of this series, including the bytes-per-token KV piece this one borrows its 128 KiB figure from, lives in my archive.

Numbers are computed from public configs, not quoted. If one of mine is wrong, say which and I will re-derive it.