I do a lot of experimenting with local language models, and I just ended up going down a bit of a rabbit hole.
It started when I came across an article about running larger Mixture-of-Experts (MoE) models on GPUs that don't have enough VRAM to hold the entire model.
The idea was pretty simple. Instead of offloading entire model layers to system RAM, you could selectively offload just the MoE experts and leave the rest of the work on the GPU.
I thought that sounded interesting. And since I have a laptop with 16GB of VRAM and 32GB of RAM, I figured I'd give it a try.
Well, one model turned into another, which turned into another, and before I knew it, I was benchmarking five different models, comparing offloading strategies, and trying to figure out why some configurations worked so much better than others.
You know, a perfectly normal way to spend a day.
And I learned quite a bit along the way.
First, what exactly is MoE offloading?
If you've experimented with running local LLMs, you're probably familiar with the problem of trying to load a model that's too large for your GPU.
You can use CPU offloading to make it fit, but there's usually a pretty significant performance penalty.
Traditional layer offloading moves entire transformer layers between the GPU and CPU. That means parts of the model that could benefit from GPU acceleration end up running on the CPU instead.
MoE models give us another option.
Unlike dense models, where essentially all the model's parameters participate in processing each token, MoE models have multiple experts, and only a subset is activated for each token.
For example, Microsoft's Phi-3.5-MoE has around 42 billion total parameters, but only about 6.6 billion are active per token.
That doesn't mean the other parameters disappear. They still need to be stored somewhere.
But since only certain experts are needed for each token, we can potentially keep some of those expert weights in system RAM while leaving attention and other parts of the model on the GPU.
In llama.cpp, this is possible using:
--n-cpu-moe N
Where N specifies how many initial layers will have their MoE expert weights kept on the CPU.
In theory, this should let you run larger models without sacrificing quite as much performance as traditional layer offloading.
But I wanted to see how well it actually worked.
My test setup
All of these experiments were done on my Acer Predator laptop running CachyOS (Arch Linux).
Here's the hardware:
- GPU: NVIDIA RTX 4090 Laptop GPU, 16GB VRAM
- CPU: Intel Core i9-14900HX, 24 cores / 32 threads
- RAM: 32GB DDR5
- Inference engine: llama.cpp with CUDA support
- Flash Attention: Enabled
I used GGUF models and ran everything through llama-server.
I started with Phi-3.5-MoE because it seemed like a good candidate for testing this technique. From there, I added several other MoE models with different architectures and quantization levels.
For the initial comparisons, I used a 4,096-token context window, a warm-up request, and the same short benchmark prompt.
This wasn't intended to be some exhaustive scientific benchmark. I wanted to see what actually happened on my hardware and whether the differences were significant enough to investigate further.
The models I tested
I ended up testing five models:
| Model | Quantization | GGUF size |
|---|---|---|
| Phi-3.5-MoE (42B) | Q4_K_M | 24GB |
| Apex-35B V1.0 | Q5_K_M | 24GB |
| Qwen3.5-35B-A3B | Q6_K | 27GB |
| Gemma 4 26B A4B | Q8_0 | 26GB |
| LFM2.5-24B-A2B | Q8_0 | 24GB |
All of them were larger than my available GPU memory.
That was the whole point of the experiment.
I wanted to compare two approaches:
- Traditional layer offloading: Move complete layers to the CPU until the model fits within available VRAM.
- Selective MoE offloading: Keep GPU-eligible non-expert operations on the GPU while moving selected expert weights and their computation to the CPU.
I adjusted the settings for each model to fit within my memory limits.
And this is where things started getting interesting.
The results were definitely not what I expected
Here's a comparison of the generation speeds I measured:
| Model | Selective MoE | Traditional offloading |
|---|---|---|
| Phi-3.5-MoE | 5.67 tok/s | 4.03 tok/s |
| Apex-35B | 5.18 tok/s | 3.69 tok/s |
| Qwen3.5-35B | 4.37 tok/s | 5.23 tok/s |
| Gemma 4 26B | 4.40 tok/s | 5.77 tok/s |
| LFM2.5-24B | 2.11 tok/s | 9.03 tok/s |
I originally expected selective MoE offloading to be the better option across the board.
That was definitely not the case.
On Phi-3.5-MoE and Apex-35B, selective offloading improved generation speed by around 40%.
But on Qwen, Gemma, and especially LFM, traditional layer offloading was faster.
And not just by a little.
LFM was more than four times faster with traditional offloading.
That one really caught my attention.
More GPU usage doesn't always mean better performance
One of my favorite discoveries came from testing Phi-3.5-MoE.
I started experimenting with how many expert layers I should keep on the CPU.
At one point, I had nearly all 16GB of VRAM occupied.
Here's what happened:
| CPU expert layers | VRAM used | Generation |
|---|---|---|
| 13 | 15,945 MiB | 5.29 tok/s |
| 16 | 13,833 MiB | 4.66 tok/s |
| 20 | 11,015 MiB | 5.67 tok/s |
The fastest result came from offloading more experts to the CPU.
I actually got better generation speed while using significantly less VRAM.
Now, I don't want to claim that leaving more VRAM free automatically makes inference faster. There are plenty of things that could affect these results, including CPU/GPU scheduling, memory bandwidth, caching, and normal variations between runs.
But it was still interesting to see that the configuration using the most GPU memory wasn't necessarily the fastest.
Sometimes the best configuration isn't the one that pushes your GPU to its absolute limit.
Quantization might matter more than I realized
Another thing I noticed was how differently models behaved at different quantization levels.
For example, Qwen3.5-35B at Q6_K performed better with traditional offloading.
But Apex-35B, which uses a related Qwen3.5 MoE architecture and was quantized to Q5_K_M, performed better with selective MoE offloading.
That made me wonder whether the additional CPU memory bandwidth needed for higher-precision expert weights was affecting performance.
It would make sense. When experts are running on the CPU, the amount of data being read from system RAM can become a significant bottleneck.
But I haven't proven that's the reason yet.
These are different model checkpoints, and there may be architectural or implementation differences affecting the results too.
I'd like to test the exact same model at Q4, Q5, and Q6 using both offloading methods. That would be a much better way to isolate the effect of quantization.
For now, it's an interesting observation rather than a conclusion.
Then there was LFM...
I really wasn't expecting this result.
LFM2.5-24B-A2B has around 24 billion total parameters, with approximately 2 billion active per token.
You might think that relatively low active parameter count would make it an especially good candidate for selective expert offloading.
Nope.
With selective offloading, I got:
2.11 tokens/sec
With traditional layer splitting:
9.03 tokens/sec
That's a huge difference.
My suspicion is that the Q8_0 expert weights, CPU memory bandwidth, and repeated synchronization between CPU and GPU may be playing a role.
But again, I would need more detailed profiling to know exactly what's happening.
What this really showed me is that the number of active parameters alone doesn't tell you how efficiently a model will run on your hardware.
The architecture, quantization, and inference implementation all matter.
And sometimes the approach that sounds better on paper performs much worse in practice.
I also experimented with context length
After finding a configuration I liked for Phi-3.5-MoE, I wanted to see what would happen when I increased the context window.
Using Flash Attention, I tested 4K, 8K, and 16K context settings.
| Context setting | VRAM used | Generation |
|---|---|---|
| 4K | 11,015 MiB | 5.67 tok/s |
| 8K | 11,531 MiB | 5.56 tok/s |
| 16K | 12,563 MiB | 5.85 tok/s |
The memory increase was much more manageable than I expected.
I also tried the same thing with Apex-35B, where the measured VRAM increase from 4K to 16K was only around 252 MiB.
It's worth mentioning that these tests changed the configured context capacity. They weren't benchmarks of processing prompts that actually contained 16,000 tokens.
That distinction matters, because processing a genuinely long prompt is a different workload.
Still, it was encouraging to see how much room I had for larger context settings without immediately running out of VRAM.
What about putting all the experts on the CPU?
I tried that too.
With Phi-3.5-MoE, I used:
--cpu-moe
And, well, that didn't go particularly well.
The model loaded, but system memory pressure increased enough to cause heavy swapping.
Generation dropped to around 3.14 tokens/sec, and I was seeing approximately 90MB/s of swap activity.
At that point, my 32GB of RAM was clearly becoming a limitation.
This doesn't mean full CPU expert offloading is always a bad idea. With more RAM, a different model, or a different quantization, it might be perfectly reasonable.
But for this particular model and my laptop, selectively offloading only some of the experts worked much better.
So, is selective MoE offloading worth using?
I think the answer is yes, but not necessarily for every model.
That's probably my biggest takeaway from all of this.
When I started, I was mostly interested in seeing whether I could run models that were too large for my GPU.
And I can.
But what I ended up learning was that getting a model to fit in memory is only half the problem.
How you distribute its workload between the CPU and GPU can make a surprisingly big difference.
For Phi-3.5-MoE and Apex-35B, selective offloading gave me better performance.
For other models, especially LFM2.5, the traditional approach was considerably faster.
I don't think there's a single offloading configuration that's going to work best for every MoE model.
At least, that's certainly not what my experiments have shown so far.
A few things I still want to test
I'm not quite finished with this rabbit hole yet.
I'd like to repeat the benchmarks with longer outputs and multiple runs per configuration, rather than relying on one short warm benchmark.
I also want to compare different quantizations of the same model to better understand how much precision affects CPU-side expert performance.
And eventually, I'd love to repeat some of these experiments on a machine with 64GB of RAM to see how much difference the extra memory makes.
There's also the question of actual usability. Tokens per second is useful, but I'd like to compare how responsive these configurations feel during longer conversations and more complicated tasks.
For anyone interested in trying this themselves, here's a basic example using llama.cpp:
llama-server \
-m /path/to/model.gguf \
-ngl all \
--n-cpu-moe 20 \
-c 8192 \
-fa on
The 20 is just an example, not a magic number. You'll need to adjust it for your model and hardware, and compare it against traditional layer offloading.
Also, keep an eye on system RAM and swap usage. Just because a model starts successfully doesn't mean it's going to run comfortably.
Final thoughts
I went into this experiment thinking I'd learn a neat trick for running bigger models on my laptop.
Instead, I ended up learning quite a bit about how different MoE architectures behave, how quantization can affect performance, and why CPU/GPU memory placement isn't nearly as straightforward as I originally thought.
And I still have plenty of questions.
That's one of the things I enjoy most about experimenting with local LLMs. You can read all the explanations and documentation you want, but sometimes you don't really understand what's happening until you start testing things yourself.
Even better when the results make you question what you thought you understood in the first place.
If you've experimented with MoE offloading, I'd love to hear what kind of results you've seen, especially on GPUs with limited VRAM.
I'm curious whether anyone else has run into such dramatic differences between architectures.
Top comments (1)
LFM2.5 at 2.11 tok/s with selective offloading versus 9.03 tok/s with traditional offloading is a big gap, but Q8_0 weights on the CPU make memory bandwidth a likely culprit before architecture alone.