DEV Community

Cover image for MoE Offloading Isn’t Always Faster: I Tested 5 Models on a 16GB GPU

MoE Offloading Isn’t Always Faster: I Tested 5 Models on a 16GB GPU

Jessica Doering on October 30, 2026

Follow-up post here: I Retested My MoE Offloading Benchmarks. Some of My Conclusions Changed. Driven by reader feedback to fix f...
Collapse
 
stefan_07d23f966225156b50 profile image
Stefan •

I'd put a link to the retest near the first results table. Your update is easy to miss down in the comments, and someone choosing settings for their own laptop could leave with conclusions you've already revised. Keeping both posts linked would also make it easier to follow what changed and why.

Collapse
 
sizzlebop profile image
Jessica Doering •

Good idea. I'll do that, thank you.

Collapse
 
sizzlebop profile image
Jessica Doering •

I wanted to share a little update. I took a lot of the feedback here into account and went back for another round of testing. This time I ran all five models at Q4_K_M, used interleaved runs to help account for thermal drift, and did some longer 512-token tests to investigate the results that seemed a little questionable.

And I'm glad I did, because some of the results changed quite a bit. I also looked further into the quantization differences with Qwen and tested speculative decoding with Gemma 4, which turned up a few more surprises.

I really appreciate everyone who took the time to offer suggestions and question my original findings. I learned quite a bit from doing these additional tests, and I think the results are much more useful because of it.

I wrote up everything in a follow-up post if anyone is interested:

Thanks again for all the feedback!

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Thanks for separating configured context from actually filling it. I’d next compare time to first token and decode speed with real 4K and 16K prompts, keeping output length fixed. That’s also the distinction I’d carry into a batched-inference design exercise, steady generation speed doesn’t tell me how long the first response takes. The best offloading setting might change with that workload.

Collapse
 
sizzlebop profile image
Jessica Doering •

That's a great suggestion. I've been focusing mostly on generation speed and memory usage so far, but you're right that actual prompt length and time to first token could change which configuration performs best.

In my follow-up experiments, I increased the generation length to 512 tokens and added repeated runs, but I haven't tested with genuinely long 4K or 16K prompts yet. That's definitely something I'd like to explore, especially since these models might behave quite differently during prompt processing versus sustained generation.

And the batched-inference angle is interesting too! I've mostly been testing single-request performance, so that would add another whole dimension to the experiments. Thanks for the ideas.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Fixing generation at 512 tokens gives you a good baseline. For the longer prompts, I’d hold the output length steady and report time to first token alongside generation speed. That would make it easier to see whether the settings change the prefill or the decode phase.

Collapse
 
ilragioniere profile image
Riccardo Vitali •

Your post already contains a noise estimate, in the context table. 4K, 8K and 16K, nothing else changed: 5.67, 5.56, 5.85 tok/s. Generation speed should not care much how much context you reserved, and those three numbers drift about 5% in both directions. That is your measurement spread, one run, short output, before any model difference enters it.

It doesn't touch your main result. 5.67 vs 4.03 on Phi, 5.18 vs 3.69 on Apex, and 9.03 vs 2.11 on LFM all sit far outside 5%, so "selective offloading is not a win across the board" survives its own noise.

It does touch the VRAM finding. Read that table as a series: 13 layers 5.29, 16 layers 4.66, 20 layers 5.67. The 20 vs 13 gap is 7%, roughly the size of the context-table spread. And 16 layers sits 18% below both of its neighbours, which is not monotonic in the layer count and has no mechanism behind it in the post. That cell is the one I would repeat before explaining: either something real happens at 16, or the single-run spread is wider than the context table suggests.

One design note that costs nothing: run the two settings interleaved, A B A B A B, instead of all A then all B. On a laptop, thermals and background load drift across a session, and interleaving keeps that drift from loading onto one arm only.

You asked what others have seen, so: what spread do you get repeating one config back to back? If it is tighter than 5%, my read of the context table is wrong and I would rather know that now.

Collapse
 
sizzlebop profile image
Jessica Doering •

Thank you so much for your comment. I really appreciate it. You're right that the 13 vs 20 layer difference is small enough that I shouldn't be drawing conclusions about VRAM headroom from those runs alone. The 16-layer result is especially interesting since it doesn't follow the trend at all.

I also hadn't considered using an interleaved test order to account for thermal and background load drift. That's a great suggestion. I think I'll repeat 13, 16, and 20 with multiple runs each and longer generations to see how much variation there actually is.

And you're right about the bigger finding. The differences between selective and traditional offloading on some of these models were large enough to make me want to investigate further, even if the smaller performance differences need more validation. Thanks again for the feedback!

Collapse
 
ilragioniere profile image
Riccardo Vitali •

One number to size the repeats against, then one design point.

Your noise floor is already printed in the post: 5.67, 5.56, 5.85 across 4K, 8K, 16K. That is 0.29 tok/s of range on a 5.69 mean, roughly 5%. Put the layer table against it.

16 vs 20: 4.66 vs 5.67, a gap of 1.01, which is 3.5x your entire observed range. Drift does not plausibly explain that one. Worth repeating.

13 vs 20: 5.29 vs 5.67, a gap of 0.38, which is 1.3x the same range. One run each cannot separate that, and it needs more runs than the 16 cell does.

So the two questions are not the same size. Three repeats settle the 16 cell. The trend needs more runs than you probably want to spend on a laptop, and "not separated from noise at one run each" is a fine sentence to write there.

Design: interleave as blocks, not rounds. One block = 13, 16, 20 back to back in randomized order. Then take the difference within each block and average the differences. If you pool all your 13-runs and average them, the drift lands inside the cell and cancels into the mean instead of out of it.

Send the numbers when they are in and I will read the spread with you.

Collapse
 
maxbuildsdev profile image
Max Kaya •

Nice that you called out the counterintuitive part — I've seen the same thing where selective expert offloading loses to plain layer offloading on certain models, and it usually comes down to how frequently the offloaded experts get hit. With 32GB of RAM you also have to watch swap: a config that "loads" fine can thrash once context grows, and the tok/s collapse looks like a model problem when it's really memory bandwidth.

Did you measure with cold page cache or did RAM stay warm across the 5 models? On that size of box the first run after loading a big model can look a lot worse than steady state, which skews the comparison.

Collapse
 
sizzlebop profile image
Jessica Doering •

Good question. I did a fresh server launch for each configuration, followed by a warm-up query before collecting the benchmark numbers. So I was aiming to measure warmed-up inference rather than cold-start performance. I didn't explicitly clear the Linux page cache between tests though, so I can't guarantee identical cache conditions across runs. That's definitely something worth controlling more carefully if I repeat these with longer generations and multiple runs. And I completely agree about watching swap. I actually ran into that when I tried offloading all of Phi's experts to the CPU. It loaded, but started swapping heavily and performance tanked.

Collapse
 
paullovelace profile image
Paul Lovelace •

The LFM split is the one that got me — 2.11 tok/s selective vs 9.03 traditional is such a big gap it stops being a tuning detail. I've seen the same shape on smaller rigs where the offloaded experts just get hit too often for the swap to ever pay off. Did the Q8_0-on-CPU setup make it worse, or was LFM already the outlier before quantization entered the picture?

Collapse
 
sizzlebop profile image
Jessica Doering •

That LFM result surprised me too. I only tested it at Q8_0 so far, so I can't say for sure how much of that gap was caused by quantization versus the architecture itself. My suspicion is that the larger CPU-resident expert weights and repeated synchronization overhead contributed, but I'd really like to test a Q4 or Q5 to see if that changes things. A 4x difference is definitely worth investigating further.

Collapse
 
deanlee profile image
Dean Lee •

The asymmetry between sequential layer offloading and selective expert offloading usually comes down to bus round-trips rather than raw parameter volume. With traditional layer offloading, you pay the PCIe transfer once per token pass as the forward pass crosses the boundary. With selective expert offloading, every layer turns into a two-way synchronization barrier between GPU attention and host memory, so latency compounds with layer count. Running VRAM right up against the 16GB ceiling also squeezes out the driver's memory allocation buffer, which explains why dropping to 20 CPU expert layers actually restored throughput.

Collapse
 
sizzlebop profile image
Jessica Doering •

That's a really good point about the synchronization overhead compounding across layers. I hadn't fully considered how much those repeated CPU/GPU round trips could add up, especially on models with lots of MoE layers. The VRAM headroom result surprised me too. I'd love to do some more profiling to figure out whether it was allocation pressure, synchronization, or something else causing that difference. Definitely gave me more to investigate.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The LFM split is the result that kills any universal rule — 2.11 tokens/sec with selective MoE offloading versus 9.03 with traditional layer offloading, more than 4x apart on the same hardware. Selective offloading clearly isn't a default win; the expert routing pattern decides. Do you think the Q6_K versus Q5_K_M quantization difference explains why Apex-35B flipped the other way, or will your planned same-model Q4/Q5/Q6 test settle that?

Collapse
 
golflover2023 profile image
golflover •

Pattern in your table: selective wins on the many-small-experts model (Phi 42B, top-2 of 16), traditional wins on the few-big-active ones (A2B/A3B/A4B). Makes sense: with few, large experts, a CPU-resident one is activated almost every token and each miss moves more MB across PCIe - so you're bus-bound, which is just layer offload with extra steps. Also explains your VRAM paradox: staging headroom beats a couple more resident experts. Worth a follow-up: frequency-aware placement - calibrate router activations, keep hot experts resident, offload the cold tail - instead of first-N-layers. Did ttft/prefill show the same split, or decode only?

Collapse
 
mudassirworks profile image
Mudassir Khan •

the LFM row is the one i keep staring at. 2.11 tok/s with selective offloading against 9.03 with layer splitting, on a model with only 2B active params, so active count clearly does not predict the speed. my guess is the CPU side reads of Q8_0 experts are bandwidth bound and the per token CPU to GPU sync adds up, so watching system memory bandwidth while it generates would be a quick check.

did you try the same checkpoint at Q4 and Q6 in both modes? that would split quantization from architecture pretty cleanly.

Collapse
 
aifrontierpost profile image
AI Frontier Post •

LFM2.5 at 2.11 tok/s with selective offloading versus 9.03 tok/s with traditional offloading is a big gap, but Q8_0 weights on the CPU make memory bandwidth a likely culprit before architecture alone.