Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff

TL;DR

SGLang 0.5.23 (released 2026-11-02) ships an opt-in feature called Paged Experts. It keeps K of the E experts of every MoE layer in GPU memory, holds all E in pinned host RAM, and copies in only the experts a step routes to that are not resident. The expert math still runs on the GPU.

• What the PR measured: on Qwen3-30B-A3B, single-stream decode of 205.6, 87.7 and 32.6 tok/s (int4 on a 16 GB card, FP8 on a 16 GB card, int4 on an 8 GB laptop), against 16.6, 2.9 and 1.2 for --cpu-offload-gb. That is 12x, 30x and 27x.
• Against llama.cpp tuned to its best --n-cpu-moe: a tie when most of the model fits or the link is slow (206 vs 194 and 33 vs 33 tok/s), and nearly 2x faster in the FP8 case (87.7 vs 45.2).
• What I computed from the PR's numbers: the measured speeds are only reachable if consecutive tokens reuse experts. Uniform random routing would cap the FP8 case near 40 tok/s on a 50 GB/s link.
• Where it breaks: with 4 concurrent requests the laptop run fell to 11 tok/s against 41 for llama.cpp. The dividing line is K // top_k, the largest decode batch that fits one wave.
• I did not run any of this. The measurements come from one PR author's two consumer machines. The arithmetic on top of them is mine.

What does Paged Experts actually do?

A mixture-of-experts layer routes each token to top_k of E experts. For Qwen3-30B-A3B that is 8 of 128, across 48 layers. I checked the model's config.json on Hugging Face: 48 layers, 128 experts, 8 per token, hidden size 2048, MoE intermediate size 768. The experts are most of the 30B parameters, so a model that is cheap to compute can still be impossible to hold on a small GPU.

There are three ways to serve it when the experts do not fit:

1. --cpu-offload-gb in SGLang. Offloaded weights are streamed to the GPU on every forward pass, whatever the router picked.
2. llama.cpp --n-cpu-moe N. The experts of N layers stay in system RAM and are computed on the CPU. No weight traffic, but CPU-speed math.
3. Paged Experts. A K-slot table per layer lives on the GPU. Each step, the experts it needs that are missing get copied from pinned host memory into slots (least recently used out), then the model's own fused-MoE kernel runs on the GPU as usual.

The third option is the HiCache idea applied to expert weights: the GPU holds the working set, host RAM holds the rest. PR #42557 (merged 2026-10-19) added bf16, block-FP8 and GPTQ experts, and PR #45787 (merged 2026-10-25) added AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8.

Turning it on takes two flags, quoted from the docs:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-30B-A3B-GPTQ-Int4 \
  --enable-paged-experts \
  --paged-experts-num-resident 92

The constraints matter as much as the flags. The docs list single GPU only (tensor, expert, pipeline and data parallelism are rejected at startup), speculative decoding rejected at startup, and a page-locked host store that has to hold every expert of every layer. Under WSL2, pinned memory is capped at about half of system RAM.

How fast is it, and against what?

The PR benchmarks on two machines: an RTX 4070 Laptop (8 GB, PCIe 4.0 x8, 54 GB RAM, WSL2) and an RTX 5070 Ti desktop (16 GB, PCIe 5.0 x16, 62 GB RAM, native Linux). The workload is ShareGPT prompts with 256 output tokens, prefix caching off in both engines, same KV capacity (10,240 tokens), byte-matched quantization (GPTQ int4 against IQ4_XS, FP8 against Q8_0). llama.cpp's --n-cpu-moe was searched in steps of 1 and only counted when its own memory projection fit physical VRAM. The PR's predecessor compared against --cpu-moe only, which in the author's own words overstated the advantage.

Setup (Qwen3-30B-A3B)	Paged Experts	llama.cpp, best	--cpu-offload-gb	vs offload	vs llama.cpp
16 GB, int4, K=92	205.6	194.4	16.6	12.4x	1.06x
16 GB, FP8, K=40	87.7	45.2	2.9	30.2x	1.94x
8 GB laptop, int4, K=26	32.6	32.8	1.2	27.2x	0.99x

Decode tok/s, single stream, from the PR tables. The two ratio columns are my division. Each headline cell was measured at least twice.

The 12x to 30x headline is against the weakest baseline. Against a tuned llama.cpp the result splits in two: when the model nearly fits, or the link is slow, paging and CPU compute land in the same place. When many experts must live in RAM and the link is fast, GPU math wins by a wide margin because llama.cpp's CPU-side layers become the bottleneck.

Why is it fast when experts keep crossing PCIe?

A token at batch 1 activates 48 x 8 = 384 experts. Per the docs an int4 expert here is 2.43 MB; FP8 is about 4.72 MB (3 x 2048 x 768 weights at one byte, scales ignored). If every activation missed, one token would need 0.93 GB (int4) or 1.81 GB (FP8) over the link.

Now compare with the measured decode time. The FP8 run does 87.7 tok/s, so 11.4 ms per token. Copying 1.81 GB in 11.4 ms needs 159 GB/s, more than a PCIe 5.0 x16 link can deliver (about 63 GB/s theoretical). So most activations cannot have been misses.

For a number I need an effective bandwidth. The PR implies one: a prefill pages in about 4 GB in about 80 ms at K=92, and (128 - 92) x 117 MB per slot is 4.2 GB, so roughly 50 GB/s. With that, the highest miss rate each measured speed allows is:

Setup	Link assumed	Miss rate if routing were uniform random (1 - K/E)	Highest miss rate the speed allows
16 GB, FP8, K=40	~50 GB/s	69%	31%
8 GB laptop, int4, K=26	15.75 GB/s (peak)	80%	52%
16 GB, int4, K=92	~50 GB/s	28%	26%

Under uniform random routing the FP8 run would be capped near 40 tok/s, and it measured 87.7. The laptop would be capped near 21 tok/s on a link running at its theoretical peak, and it measured 32.6. So the router must be reusing experts from one token to the next, the premise the docs state. The int4 16 GB row is inconclusive: at 63 GB/s instead of 50 the bound moves to 33%, above the random baseline.

Two caveats. These are upper bounds on the miss rate, not measured miss rates; only a counter inside the engine gives the real value. And the reuse was seen on natural-language prompts. The PR warns that synthetic prompts mis-measure paging, so if your traffic is unusual, test on your own.

What happens with more than one request?

The PR is candid here. At 4 concurrent requests the laptop run produced 11 to 12 tok/s against 41 for llama.cpp, and FP8 on the 16 GB card gave 84 tok/s at K=40 but 31 at K=24 (llama.cpp: 55).

The docs explain the mechanism. A decode step is served by a single wave only if tokens x top_k <= K, which is decode up to K // top_k requests, and that single wave is the case a CUDA graph can capture. Larger batches run without a graph and need several waves, each with its own page-in. Four requests with top-8 routing can touch up to 32 distinct experts per layer, and the table of K values shows the pattern:

Run	K	K // 8	4-request result vs llama.cpp
int4, 16 GB	92	11	387 vs 312 tok/s
FP8, 16 GB	40	5	84 vs 55
FP8, 16 GB	24	3	31 vs 55
int4, 8 GB	26	3	11 vs 41

Both runs where K // 8 is below 4 lose; both where it is 5 or more win. Four points do not make a law, and the PR attributes the loss to steps that route to more experts than fit, the same condition seen from the other side. It does give a usable sizing rule: choose K at least batch x top_k for the batch size you plan to serve, or accept that you are outside the captured fast path. For one user this is irrelevant. For anything shared it decides whether the feature fits.

What does a resident slot cost in context length?

K and the KV cache come out of the same static budget (--mem-fraction-static), and one slot costs per_expert_bytes x num_moe_layers. Here the bf16 KV cache is 2 x 48 layers x 4 KV heads x 128 head dim x 2 bytes = 98,304 bytes (96 KiB) per token.

• One int4 slot is 117 MB, about 1,190 tokens of bf16 KV cache (about 2,370 with an FP8 KV cache).
• One FP8 slot is about 227 MB, about 2,300 tokens of bf16 KV cache.

Dropping K by 10 on the int4 model buys roughly 11,900 tokens of context at the price of more paging. The PR's whole 10,240-token cache is only about 1.0 GB, so the experts dominate the budget. If the KV cache does not fit at startup, the docs suggest lowering K or using --kv-cache-dtype fp8.

Why do short prompts get slower first tokens?

A prefill touches nearly every expert, so it pages in everything that is not resident, once. For int4 at K=92 that is about 4.2 GB. For FP8 at K=40 it is 88 experts x 227 MB, about 19.9 GB, and on the laptop (K=26) about 11.9 GB over a link that peaks at 15.75 GB/s, at least 750 ms.

TTFT p50	Paged	llama.cpp best
int4 16 GB, short / 2,048 tokens	89 / 267 ms	68 / 344 ms
FP8 16 GB, short / 2,048 tokens	284 / 443 ms	226 / 984 ms
laptop int4, short / 2,048 tokens	928 / 1,441 ms	620 / 2,396 ms

The page-in is a fixed cost, while llama.cpp's CPU-side layers scale with prompt length, so paging loses on short prompts and wins on long ones. The FP8 2,048-token TTFT of 443 ms is within about 45 ms of the 400 ms needed to move 19.9 GB at 50 GB/s, which suggests nearly all of it is page-in (arithmetic, not a profile). For short chat turns, TTFT is the number to watch.

Can paging change the model's outputs?

By design no. Routing is untouched and the paged layer returns the same result as the unpaged one. The docs report bit-identical logprobs for unquantized, FP8 and INT8 experts under deterministic inference, and agreement within Marlin's own batch-to-batch noise for the other formats. On GSM8K (200 questions), gpt-oss-20b MXFP4 scored 0.96 to 0.97 paged and 0.975 unpaged.

The more instructive part is how it can go wrong silently, as described in PR #45787:

• An expert tensor that was not listed as pageable stays unpaged, so a paged-in expert runs with another expert's values. In an AWQ control with the zero points unpaged, the median logprob error was 1.67 nats, against 0.003 when paged correctly.
• gpt-oss-20b's MXFP4 experts are padded by the Marlin kernel (2880 to 2944 or 3072). Before the fix, paged gpt-oss-20b scored GSM8K 0.000 with every answer truncated. After it, 0.960.
• The fix for the first class is structural: the pageable tensors are now derived from the layer, and at startup an independence check re-runs one more chunk of experts and fails if the post-load repack mixed experts.

None of these crash. They produce fluent wrong text, so run an accuracy smoke test against the unpaged model on your own checkpoint format.

A preflight script before you turn it on

This is the arithmetic above in a few lines: slot cost, batch cap, and whether a measured speed implies expert reuse.

def preflight(name, layers, experts, top_k, expert_mb, K, link_gbps, tok_s, kv_bytes_per_token):
    slot_mb = expert_mb * layers                    # GPU memory per slot, all layers
    batch_cap = K // top_k                          # largest single-wave decode batch
    all_miss_gb = layers * top_k * expert_mb / 1e3  # GB per token if every activation missed
    max_miss = link_gbps / (tok_s * all_miss_gb)    # highest miss rate the speed allows
    random_miss = 1 - K / experts                   # miss rate with no reuse
    verdict = "reuse implied" if max_miss < 0.8 * random_miss else "inconclusive"
    print(f"{name}: slot={slot_mb:.0f} MB (~{slot_mb*1e6/kv_bytes_per_token:,.0f} KV tokens), "
          f"batch cap={batch_cap}, random-routing miss={random_miss:.0%}, "
          f"max miss allowed by {tok_s} tok/s={max_miss:.0%}, {verdict}")

KV = 2 * 48 * 4 * 128 * 2   # Qwen3-30B-A3B bf16 KV bytes per token
preflight("int4 16GB", 48, 128, 8, 2.43, 92, 50.0, 205.6, KV)
preflight("FP8  16GB", 48, 128, 8, 4.72, 40, 50.0,  87.7, KV)
preflight("int4  8GB", 48, 128, 8, 2.43, 26, 15.75, 32.6, KV)

Output from running it:

int4 16GB: slot=117 MB (~1,187 KV tokens), batch cap=11, random-routing miss=28%, max miss allowed by 205.6 tok/s=26%, inconclusive
FP8  16GB: slot=227 MB (~2,305 KV tokens), batch cap=5, random-routing miss=69%, max miss allowed by 87.7 tok/s=31%, reuse implied
int4  8GB: slot=117 MB (~1,187 KV tokens), batch cap=3, random-routing miss=80%, max miss allowed by 32.6 tok/s=52%, reuse implied

Use a link speed measured with a pinned-memory copy test, not the spec sheet. The 50 GB/s here is inferred from the PR.

A rollout checklist

1. Confirm your format is in the supported table and the MoE runner is Triton or Marlin. Anything else is rejected at startup, which is the good failure.
2. Check that the host can pin the whole expert store (half of RAM under WSL2).
3. Choose K from the docs' formula, then check K // top_k against your real concurrency.
4. Measure host-to-device bandwidth on your machine, and benchmark your own prompts, because the PR warns that synthetic ones mis-measure paging.
5. Run an accuracy smoke test against the unpaged model, and record TTFT for short and long prompts separately.

FAQ

Does it work across several GPUs or with speculative decoding?
No to both in 0.5.23. The docs say single GPU only, with tensor, expert, pipeline and data parallelism rejected at startup, and speculative decoding is rejected too.

Does it change accuracy compared with an unpaged server?
The docs report bit-identical logprobs for unquantized, FP8 and INT8 experts under deterministic inference, and agreement within Marlin's noise for the other formats. Past bugs in this area failed silently, so run a smoke test on your format.

How do I choose K?
Use the docs' budget formula, then keep K at least batch x top_k for your target concurrency. Each extra slot costs per_expert_bytes x layers of memory that would otherwise be KV cache.

Is it faster than llama.cpp's --n-cpu-moe?
It depends. In the PR, single-stream decode tied in two of three setups and was 1.94x faster in the third, while llama.cpp had lower TTFT on short prompts and better 4-request throughput on the laptop.

How much host RAM do I need?
Enough to pin every expert: about 14.9 GB at int4 and 29 GB at FP8 for Qwen3-30B-A3B, by my arithmetic on the docs' per-expert size.

Further reading

• KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
• What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

Sources

• SGLang v0.5.23 release notes
• PR #42557: Paged Experts, serve MoE models whose weights do not fit VRAM
• PR #45787: Paged Experts for AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8 experts
• Paged Experts documentation at v0.5.23
• Qwen3-30B-A3B config.json

Comments

[Ahmet Özel]

K >= batch times top_k is a useful worst-case sizing guarantee, but K // top_k is not necessarily the largest batch that can execute in one wave. Several requests can route to overlapping experts, so the union of selected experts can be much smaller than the sum of their top-k counts.

I would instrument that per-layer union size and the actual wave count beside concurrency. Then the four-request cliff can be related to routing diversity in the workload, and the conservative capacity rule stays distinct from a measured hard limit for every possible batch.