DEV Community

Cover image for MoE Offloading Isn’t Always Faster: I Tested 5 Models on a 16GB GPU
Jessica Doering
Jessica Doering Subscriber

Posted on Edited on

MoE Offloading Isn’t Always Faster: I Tested 5 Models on a 16GB GPU

Bus bottlenecks explain the massive speed drops

Follow-up post here:

I do a lot of experimenting with local language models, and I just ended up going down a bit of a rabbit hole.

It started when I came across an article about running larger Mixture-of-Experts (MoE) models on GPUs that don't have enough VRAM to hold the entire model.

The idea was pretty simple. Instead of offloading entire model layers to system RAM, you could selectively offload just the MoE experts and leave the rest of the work on the GPU.

I thought that sounded interesting. And since I have a laptop with 16GB of VRAM and 32GB of RAM, I figured I'd give it a try.

Well, one model turned into another, which turned into another, and before I knew it, I was benchmarking five different models, comparing offloading strategies, and trying to figure out why some configurations worked so much better than others.

You know, a perfectly normal way to spend a day.

And I learned quite a bit along the way.

First, what exactly is MoE offloading?

If you've experimented with running local LLMs, you're probably familiar with the problem of trying to load a model that's too large for your GPU.

You can use CPU offloading to make it fit, but there's usually a pretty significant performance penalty.

Traditional layer offloading moves entire transformer layers between the GPU and CPU. That means parts of the model that could benefit from GPU acceleration end up running on the CPU instead.

MoE models give us another option.

Unlike dense models, where essentially all the model's parameters participate in processing each token, MoE models have multiple experts, and only a subset is activated for each token.

For example, Microsoft's Phi-3.5-MoE has around 42 billion total parameters, but only about 6.6 billion are active per token.

That doesn't mean the other parameters disappear. They still need to be stored somewhere.

But since only certain experts are needed for each token, we can potentially keep some of those expert weights in system RAM while leaving attention and other parts of the model on the GPU.

In llama.cpp, this is possible using:

--n-cpu-moe N
Enter fullscreen mode Exit fullscreen mode

Where N specifies how many initial layers will have their MoE expert weights kept on the CPU.

In theory, this should let you run larger models without sacrificing quite as much performance as traditional layer offloading.

But I wanted to see how well it actually worked.

My test setup

All of these experiments were done on my Acer Predator laptop running CachyOS (Arch Linux).

Here's the hardware:

  • GPU: NVIDIA RTX 4090 Laptop GPU, 16GB VRAM
  • CPU: Intel Core i9-14900HX, 24 cores / 32 threads
  • RAM: 32GB DDR5
  • Inference engine: llama.cpp with CUDA support
  • Flash Attention: Enabled

I used GGUF models and ran everything through llama-server.

I started with Phi-3.5-MoE because it seemed like a good candidate for testing this technique. From there, I added several other MoE models with different architectures and quantization levels.

For the initial comparisons, I used a 4,096-token context window, a warm-up request, and the same short benchmark prompt.

This wasn't intended to be some exhaustive scientific benchmark. I wanted to see what actually happened on my hardware and whether the differences were significant enough to investigate further.

The models I tested

I ended up testing five models:

Model Quantization GGUF size
Phi-3.5-MoE (42B) Q4_K_M 24GB
Apex-35B V1.0 Q5_K_M 24GB
Qwen3.5-35B-A3B Q6_K 27GB
Gemma 4 26B A4B Q8_0 26GB
LFM2.5-24B-A2B Q8_0 24GB

All of them were larger than my available GPU memory.

That was the whole point of the experiment.

I wanted to compare two approaches:

  1. Traditional layer offloading: Move complete layers to the CPU until the model fits within available VRAM.
  2. Selective MoE offloading: Keep GPU-eligible non-expert operations on the GPU while moving selected expert weights and their computation to the CPU.

I adjusted the settings for each model to fit within my memory limits.

And this is where things started getting interesting.

The results were definitely not what I expected

Here's a comparison of the generation speeds I measured:

Model Selective MoE Traditional offloading
Phi-3.5-MoE 5.67 tok/s 4.03 tok/s
Apex-35B 5.18 tok/s 3.69 tok/s
Qwen3.5-35B 4.37 tok/s 5.23 tok/s
Gemma 4 26B 4.40 tok/s 5.77 tok/s
LFM2.5-24B 2.11 tok/s 9.03 tok/s

I originally expected selective MoE offloading to be the better option across the board.

That was definitely not the case.

On Phi-3.5-MoE and Apex-35B, selective offloading improved generation speed by around 40%.

But on Qwen, Gemma, and especially LFM, traditional layer offloading was faster.

And not just by a little.

LFM was more than four times faster with traditional offloading.

That one really caught my attention.

More GPU usage doesn't always mean better performance

One of my favorite discoveries came from testing Phi-3.5-MoE.

I started experimenting with how many expert layers I should keep on the CPU.

At one point, I had nearly all 16GB of VRAM occupied.

Here's what happened:

CPU expert layers VRAM used Generation
13 15,945 MiB 5.29 tok/s
16 13,833 MiB 4.66 tok/s
20 11,015 MiB 5.67 tok/s

The fastest result came from offloading more experts to the CPU.

I actually got better generation speed while using significantly less VRAM.

Now, I don't want to claim that leaving more VRAM free automatically makes inference faster. There are plenty of things that could affect these results, including CPU/GPU scheduling, memory bandwidth, caching, and normal variations between runs.

But it was still interesting to see that the configuration using the most GPU memory wasn't necessarily the fastest.

Sometimes the best configuration isn't the one that pushes your GPU to its absolute limit.

Quantization might matter more than I realized

Another thing I noticed was how differently models behaved at different quantization levels.

For example, Qwen3.5-35B at Q6_K performed better with traditional offloading.

But Apex-35B, which uses a related Qwen3.5 MoE architecture and was quantized to Q5_K_M, performed better with selective MoE offloading.

That made me wonder whether the additional CPU memory bandwidth needed for higher-precision expert weights was affecting performance.

It would make sense. When experts are running on the CPU, the amount of data being read from system RAM can become a significant bottleneck.

But I haven't proven that's the reason yet.

These are different model checkpoints, and there may be architectural or implementation differences affecting the results too.

I'd like to test the exact same model at Q4, Q5, and Q6 using both offloading methods. That would be a much better way to isolate the effect of quantization.

For now, it's an interesting observation rather than a conclusion.

Then there was LFM...

I really wasn't expecting this result.

LFM2.5-24B-A2B has around 24 billion total parameters, with approximately 2 billion active per token.

You might think that relatively low active parameter count would make it an especially good candidate for selective expert offloading.

Nope.

With selective offloading, I got:

2.11 tokens/sec

With traditional layer splitting:

9.03 tokens/sec

That's a huge difference.

My suspicion is that the Q8_0 expert weights, CPU memory bandwidth, and repeated synchronization between CPU and GPU may be playing a role.

But again, I would need more detailed profiling to know exactly what's happening.

What this really showed me is that the number of active parameters alone doesn't tell you how efficiently a model will run on your hardware.

The architecture, quantization, and inference implementation all matter.

And sometimes the approach that sounds better on paper performs much worse in practice.

I also experimented with context length

After finding a configuration I liked for Phi-3.5-MoE, I wanted to see what would happen when I increased the context window.

Using Flash Attention, I tested 4K, 8K, and 16K context settings.

Context setting VRAM used Generation
4K 11,015 MiB 5.67 tok/s
8K 11,531 MiB 5.56 tok/s
16K 12,563 MiB 5.85 tok/s

The memory increase was much more manageable than I expected.

I also tried the same thing with Apex-35B, where the measured VRAM increase from 4K to 16K was only around 252 MiB.

It's worth mentioning that these tests changed the configured context capacity. They weren't benchmarks of processing prompts that actually contained 16,000 tokens.

That distinction matters, because processing a genuinely long prompt is a different workload.

Still, it was encouraging to see how much room I had for larger context settings without immediately running out of VRAM.

What about putting all the experts on the CPU?

I tried that too.

With Phi-3.5-MoE, I used:

--cpu-moe
Enter fullscreen mode Exit fullscreen mode

And, well, that didn't go particularly well.

The model loaded, but system memory pressure increased enough to cause heavy swapping.

Generation dropped to around 3.14 tokens/sec, and I was seeing approximately 90MB/s of swap activity.

At that point, my 32GB of RAM was clearly becoming a limitation.

This doesn't mean full CPU expert offloading is always a bad idea. With more RAM, a different model, or a different quantization, it might be perfectly reasonable.

But for this particular model and my laptop, selectively offloading only some of the experts worked much better.

So, is selective MoE offloading worth using?

I think the answer is yes, but not necessarily for every model.

That's probably my biggest takeaway from all of this.

When I started, I was mostly interested in seeing whether I could run models that were too large for my GPU.

And I can.

But what I ended up learning was that getting a model to fit in memory is only half the problem.

How you distribute its workload between the CPU and GPU can make a surprisingly big difference.

For Phi-3.5-MoE and Apex-35B, selective offloading gave me better performance.

For other models, especially LFM2.5, the traditional approach was considerably faster.

I don't think there's a single offloading configuration that's going to work best for every MoE model.

At least, that's certainly not what my experiments have shown so far.

A few things I still want to test

I'm not quite finished with this rabbit hole yet.

I'd like to repeat the benchmarks with longer outputs and multiple runs per configuration, rather than relying on one short warm benchmark.

I also want to compare different quantizations of the same model to better understand how much precision affects CPU-side expert performance.

And eventually, I'd love to repeat some of these experiments on a machine with 64GB of RAM to see how much difference the extra memory makes.

There's also the question of actual usability. Tokens per second is useful, but I'd like to compare how responsive these configurations feel during longer conversations and more complicated tasks.

For anyone interested in trying this themselves, here's a basic example using llama.cpp:

llama-server \
  -m /path/to/model.gguf \
  -ngl all \
  --n-cpu-moe 20 \
  -c 8192 \
  -fa on
Enter fullscreen mode Exit fullscreen mode

The 20 is just an example, not a magic number. You'll need to adjust it for your model and hardware, and compare it against traditional layer offloading.

Also, keep an eye on system RAM and swap usage. Just because a model starts successfully doesn't mean it's going to run comfortably.

Final thoughts

I went into this experiment thinking I'd learn a neat trick for running bigger models on my laptop.

Instead, I ended up learning quite a bit about how different MoE architectures behave, how quantization can affect performance, and why CPU/GPU memory placement isn't nearly as straightforward as I originally thought.

And I still have plenty of questions.

That's one of the things I enjoy most about experimenting with local LLMs. You can read all the explanations and documentation you want, but sometimes you don't really understand what's happening until you start testing things yourself.

Even better when the results make you question what you thought you understood in the first place.

If you've experimented with MoE offloading, I'd love to hear what kind of results you've seen, especially on GPUs with limited VRAM.

I'm curious whether anyone else has run into such dramatic differences between architectures.

Top comments (19)

Collapse
 
stefan_07d23f966225156b50 profile image
Stefan •

I'd put a link to the retest near the first results table. Your update is easy to miss down in the comments, and someone choosing settings for their own laptop could leave with conclusions you've already revised. Keeping both posts linked would also make it easier to follow what changed and why.

Collapse
 
sizzlebop profile image
Jessica Doering •

Good idea. I'll do that, thank you.

Collapse
 
sizzlebop profile image
Jessica Doering •

I wanted to share a little update. I took a lot of the feedback here into account and went back for another round of testing. This time I ran all five models at Q4_K_M, used interleaved runs to help account for thermal drift, and did some longer 512-token tests to investigate the results that seemed a little questionable.

And I'm glad I did, because some of the results changed quite a bit. I also looked further into the quantization differences with Qwen and tested speculative decoding with Gemma 4, which turned up a few more surprises.

I really appreciate everyone who took the time to offer suggestions and question my original findings. I learned quite a bit from doing these additional tests, and I think the results are much more useful because of it.

I wrote up everything in a follow-up post if anyone is interested:

Thanks again for all the feedback!

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Thanks for separating configured context from actually filling it. I’d next compare time to first token and decode speed with real 4K and 16K prompts, keeping output length fixed. That’s also the distinction I’d carry into a batched-inference design exercise, steady generation speed doesn’t tell me how long the first response takes. The best offloading setting might change with that workload.

Collapse
 
sizzlebop profile image
Jessica Doering •

That's a great suggestion. I've been focusing mostly on generation speed and memory usage so far, but you're right that actual prompt length and time to first token could change which configuration performs best.

In my follow-up experiments, I increased the generation length to 512 tokens and added repeated runs, but I haven't tested with genuinely long 4K or 16K prompts yet. That's definitely something I'd like to explore, especially since these models might behave quite differently during prompt processing versus sustained generation.

And the batched-inference angle is interesting too! I've mostly been testing single-request performance, so that would add another whole dimension to the experiments. Thanks for the ideas.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Fixing generation at 512 tokens gives you a good baseline. For the longer prompts, I’d hold the output length steady and report time to first token alongside generation speed. That would make it easier to see whether the settings change the prefill or the decode phase.

Collapse
 
ilragioniere profile image
Riccardo Vitali •

Your post already contains a noise estimate, in the context table. 4K, 8K and 16K, nothing else changed: 5.67, 5.56, 5.85 tok/s. Generation speed should not care much how much context you reserved, and those three numbers drift about 5% in both directions. That is your measurement spread, one run, short output, before any model difference enters it.

It doesn't touch your main result. 5.67 vs 4.03 on Phi, 5.18 vs 3.69 on Apex, and 9.03 vs 2.11 on LFM all sit far outside 5%, so "selective offloading is not a win across the board" survives its own noise.

It does touch the VRAM finding. Read that table as a series: 13 layers 5.29, 16 layers 4.66, 20 layers 5.67. The 20 vs 13 gap is 7%, roughly the size of the context-table spread. And 16 layers sits 18% below both of its neighbours, which is not monotonic in the layer count and has no mechanism behind it in the post. That cell is the one I would repeat before explaining: either something real happens at 16, or the single-run spread is wider than the context table suggests.

One design note that costs nothing: run the two settings interleaved, A B A B A B, instead of all A then all B. On a laptop, thermals and background load drift across a session, and interleaving keeps that drift from loading onto one arm only.

You asked what others have seen, so: what spread do you get repeating one config back to back? If it is tighter than 5%, my read of the context table is wrong and I would rather know that now.

Collapse
 
sizzlebop profile image
Jessica Doering •

Thank you so much for your comment. I really appreciate it. You're right that the 13 vs 20 layer difference is small enough that I shouldn't be drawing conclusions about VRAM headroom from those runs alone. The 16-layer result is especially interesting since it doesn't follow the trend at all.

I also hadn't considered using an interleaved test order to account for thermal and background load drift. That's a great suggestion. I think I'll repeat 13, 16, and 20 with multiple runs each and longer generations to see how much variation there actually is.

And you're right about the bigger finding. The differences between selective and traditional offloading on some of these models were large enough to make me want to investigate further, even if the smaller performance differences need more validation. Thanks again for the feedback!

Collapse
 
ilragioniere profile image
Riccardo Vitali •

One number to size the repeats against, then one design point.

Your noise floor is already printed in the post: 5.67, 5.56, 5.85 across 4K, 8K, 16K. That is 0.29 tok/s of range on a 5.69 mean, roughly 5%. Put the layer table against it.

16 vs 20: 4.66 vs 5.67, a gap of 1.01, which is 3.5x your entire observed range. Drift does not plausibly explain that one. Worth repeating.

13 vs 20: 5.29 vs 5.67, a gap of 0.38, which is 1.3x the same range. One run each cannot separate that, and it needs more runs than the 16 cell does.

So the two questions are not the same size. Three repeats settle the 16 cell. The trend needs more runs than you probably want to spend on a laptop, and "not separated from noise at one run each" is a fine sentence to write there.

Design: interleave as blocks, not rounds. One block = 13, 16, 20 back to back in randomized order. Then take the difference within each block and average the differences. If you pool all your 13-runs and average them, the drift lands inside the cell and cancels into the mean instead of out of it.

Send the numbers when they are in and I will read the spread with you.

Collapse
 
maxbuildsdev profile image
Max Kaya •

Nice that you called out the counterintuitive part — I've seen the same thing where selective expert offloading loses to plain layer offloading on certain models, and it usually comes down to how frequently the offloaded experts get hit. With 32GB of RAM you also have to watch swap: a config that "loads" fine can thrash once context grows, and the tok/s collapse looks like a model problem when it's really memory bandwidth.

Did you measure with cold page cache or did RAM stay warm across the 5 models? On that size of box the first run after loading a big model can look a lot worse than steady state, which skews the comparison.

Collapse
 
sizzlebop profile image
Jessica Doering •

Good question. I did a fresh server launch for each configuration, followed by a warm-up query before collecting the benchmark numbers. So I was aiming to measure warmed-up inference rather than cold-start performance. I didn't explicitly clear the Linux page cache between tests though, so I can't guarantee identical cache conditions across runs. That's definitely something worth controlling more carefully if I repeat these with longer generations and multiple runs. And I completely agree about watching swap. I actually ran into that when I tried offloading all of Phi's experts to the CPU. It loaded, but started swapping heavily and performance tanked.

Collapse
 
paullovelace profile image
Paul Lovelace •

The LFM split is the one that got me — 2.11 tok/s selective vs 9.03 traditional is such a big gap it stops being a tuning detail. I've seen the same shape on smaller rigs where the offloaded experts just get hit too often for the swap to ever pay off. Did the Q8_0-on-CPU setup make it worse, or was LFM already the outlier before quantization entered the picture?

Collapse
 
sizzlebop profile image
Jessica Doering •

That LFM result surprised me too. I only tested it at Q8_0 so far, so I can't say for sure how much of that gap was caused by quantization versus the architecture itself. My suspicion is that the larger CPU-resident expert weights and repeated synchronization overhead contributed, but I'd really like to test a Q4 or Q5 to see if that changes things. A 4x difference is definitely worth investigating further.

Collapse
 
deanlee profile image
Dean Lee •

The asymmetry between sequential layer offloading and selective expert offloading usually comes down to bus round-trips rather than raw parameter volume. With traditional layer offloading, you pay the PCIe transfer once per token pass as the forward pass crosses the boundary. With selective expert offloading, every layer turns into a two-way synchronization barrier between GPU attention and host memory, so latency compounds with layer count. Running VRAM right up against the 16GB ceiling also squeezes out the driver's memory allocation buffer, which explains why dropping to 20 CPU expert layers actually restored throughput.

Collapse
 
sizzlebop profile image
Jessica Doering •

That's a really good point about the synchronization overhead compounding across layers. I hadn't fully considered how much those repeated CPU/GPU round trips could add up, especially on models with lots of MoE layers. The VRAM headroom result surprised me too. I'd love to do some more profiling to figure out whether it was allocation pressure, synchronization, or something else causing that difference. Definitely gave me more to investigate.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The LFM split is the result that kills any universal rule — 2.11 tokens/sec with selective MoE offloading versus 9.03 with traditional layer offloading, more than 4x apart on the same hardware. Selective offloading clearly isn't a default win; the expert routing pattern decides. Do you think the Q6_K versus Q5_K_M quantization difference explains why Apex-35B flipped the other way, or will your planned same-model Q4/Q5/Q6 test settle that?

Collapse
 
golflover2023 profile image
golflover •

Pattern in your table: selective wins on the many-small-experts model (Phi 42B, top-2 of 16), traditional wins on the few-big-active ones (A2B/A3B/A4B). Makes sense: with few, large experts, a CPU-resident one is activated almost every token and each miss moves more MB across PCIe - so you're bus-bound, which is just layer offload with extra steps. Also explains your VRAM paradox: staging headroom beats a couple more resident experts. Worth a follow-up: frequency-aware placement - calibrate router activations, keep hot experts resident, offload the cold tail - instead of first-N-layers. Did ttft/prefill show the same split, or decode only?

Collapse
 
mudassirworks profile image
Mudassir Khan •

the LFM row is the one i keep staring at. 2.11 tok/s with selective offloading against 9.03 with layer splitting, on a model with only 2B active params, so active count clearly does not predict the speed. my guess is the CPU side reads of Q8_0 experts are bandwidth bound and the per token CPU to GPU sync adds up, so watching system memory bandwidth while it generates would be a quick check.

did you try the same checkpoint at Q4 and Q6 in both modes? that would split quantization from architecture pretty cleanly.