The build of GLM-5.3-Flash I run is 181 GiB, and my GPU has 96 GiB. It still writes up to 16.5 tokens a second on one desktop. Offloading more on its own made it slower; keeping each layer's hot experts on the card made it faster.
One of the eight bugs on the way gave no error at all. The model just emitted the same token, forever. I will come back to that one.
What you get
- A model bigger than the card at about 15 tokens a second, output unchanged.
- Three changes that made writing 1.5x faster and prompt reading 2.9x faster, and the ideas that failed.
The numbers, with their conditions
One RTX PRO 6000 Blackwell card, one request at a time.
- Writing (decode), peak: 16.50 tokens a second over 8 back-to-back 200-token answers to short prompts, with 66 hot experts per layer on the card.
- Writing, as shipped: 14.99 tokens a second with 54 hot experts per layer, leaving room for long prompts.
- Reading a prompt: 1,771 and 1,729 tokens a second on two fresh prompts of about 10,300 tokens (5.82 s and 5.96 s to the first token).
- Longest prompt: 304,800 tokens in 205 s, 93% of the 327,680 configured.
- Output: unchanged; on the prompt path, log-probabilities stay within the baseline's own run-to-run spread over 13,519 tokens.
Prompts are streamed and fresh, so no cache helps. The setups differ in more than one setting (context length, hot experts). The peak ran on 2026-09-01, the rest on 2026-09-02 (prompt runs logged in artifacts/prefill-fresh-2026-09-02.txt).
How it got faster
The wall: one PCIe transfer per token
It is an NVFP4 build of GLM-5.3-Flash, so its weights are 4-bit numbers. 90% of it is "experts": each layer has 288, and a router picks 8 per token. In the first setup, 105 GiB of experts lived in host RAM, and the GPU fetched the ones each token needed over PCIe, the link to the rest of the machine. That version wrote 10.77 tokens a second, and nearly all its time went to that one transfer.
A gather kernel that reads wide
The link was not the problem; the way it was read was. The default gather (index_select) used narrow, scattered loads. A small Triton kernel that reads wide blocks roughly doubled the rate, to 52.1 GB/s, 92% of the link. Its list of experts stays on the GPU, so the CPU never stops to read it, and CUDA graphs could come back: replaying a recorded step added 13%, for 13.53 tokens a second.
Offload more, cache the hot experts
vLLM decides what stays on the card one whole layer at a time, in module order, so the experts left in VRAM were arbitrary. Summed across all layers, my routing counts looked flat and capped any cache at about 1.15x, so I closed caching as a dead end. One groupby by layer reopened it.
In each layer, the hottest 35% of the experts take 74% of the picks. So I offloaded more, 125 GiB instead of 105, and spent the freed VRAM on a cache of each layer's 66 most-used experts: 16.50 tokens a second. The same offload without the cache gave 11.34, slower than the 13.53 before it. The two only work as a pair.
Prefill in groups of 64 experts
A long prompt (prefill) routes to nearly every expert, which overflowed the 64-expert staging buffer and fell back to slow reads at about 23 GB/s. Walking the experts in groups of 64 took prompt reading from 614 to about 1,750 tokens a second.
Grouping costs short prompts (a 14-token prompt waited 2.05 s for its first token instead of 0.85 s), so it switches on only at 48 tokens or more.
What did not work, with the numbers
The bug with no error
Nothing crashed or logged an error; the model emitted the same token forever. vLLM can run the expert math with several kernels, and on this card two that compute in 4-bit directly return wrong numbers silently. Marlin, which unpacks the weights first and warns that the GPU lacks native FP4, is the one that works, and the fastest. Take this one line:
--kernel-config '{"moe_backend":"marlin"}'
Try it (it is not turnkey)
You need an RTX PRO 6000 Blackwell card (96 GiB), vLLM with GLM-5.3-Flash support from pull request #53906, the repository's patch, and host RAM for the offloaded experts. Of the 181.3 GiB checkpoint, 163.3 GiB is experts; 125 GiB of them sit in RAM (the test machine has 182 GiB), and the rest stays on the card. The tests ran on a pre-release build, and the setup was not re-run on vLLM 0.30.0 or later. Boot takes 5 to 6 minutes. Without the card, you can still reuse the ideas.
To start reading:
git clone https://github.com/pctablet505/glm53-flash-single-gpu
cd glm53-flash-single-gpu
less RUNNING-GLM53-ON-ONE-GPU.md # the build and the eight bugs, in order
less config/serve-opt.sh # the shipped settings, each with its reason
Limits
- One machine, one checkpoint and one pre-release vLLM build with local patches.
- Most numbers are single runs, with about ±1 token a second of spread.
- Without changing the output, this hardware tops out near 16 to 17 tokens a second.
- If a model that fits in 96 GiB will do, use it: Qwen3.8-Flash-Next (78.5 GiB) runs at 173 tokens a second here.
Still open: a masked variant of grouped prefill, which skips experts the prompt does not use, reached the first token in 0.36 s on the 14-token prompt, against 0.85 s for what ships. Its correctness was never properly tested. Have you tried masked staging for prefill?
Where next
- RUNNING: start here: the build and all eight bugs.
- OPTIMIZING: the cost model behind these numbers, and the settings that do nothing.
- LAB-NOTEBOOK: the same work day by day, mistakes included.
- servedeck: the small control plane I use to start, stop and switch the models on this workstation, with an OpenAI-compatible endpoint in front.
I am pctablet505 on GitHub. I measure what actually happens and leave the failures in; the repositories above are where the rest of it lives.
This article was written with the help of AI, from my own measurements; the logs are in the repository.
Top comments (0)