anemll-flash-llama.cpp is useful for running large MoE models on Macs with limited RAM. Instead of keeping all experts in unified memory, it stores them on SSD and loads only the experts needed for the current tokens into a small cache (slot-bank). This makes models that normally would not fit in memory usable on much smaller machines, trading some speed for dramatically lower RAM requirements.
Benchmark of Qwen3.5-35B-A3B using the anemll-flash-llama.cpp Flash-MoE / slot-bank backend on a MacBook Pro with Apple M1 Pro and 16 GB unified memory.
Test Environment
| Item | Configuration |
|---|---|
| Hardware | MacBook Pro |
| SoC | Apple M1 Pro |
| Unified Memory | 16 GB |
| Metal Working Set Limit | ~12.7 GB |
| Backend | Metal + Flash-MoE |
| Flash-MoE Mode | slot-bank |
| GPU Layers | 999 |
| Top-K Override | 4 |
| Prefill Batch | 8192 |
| Cache I/O Split | 4 |
| Async Upload | Off |
| Parallel Reads | Off |
Models
| Model | Quantization | GGUF Size |
|---|---|---|
| Qwen3.5-35B-A3B-Q3_K_M | Q3_K_M | ~16 GB |
| Qwen3.5-35B-A3B-UD-Q4_K_XL | UD-Q4_K_XL | ~22 GB |
Both models are larger than the practical Metal working-set limit and therefore rely heavily on Flash-MoE expert streaming from storage.
Benchmark Results
| Model | Size | Slot Bank | Prompt | Generation | Cache Hit | Expert I/O | Routed Time / Token |
|---|---|---|---|---|---|---|---|
| Q3_K_M | 16 GB | 4 | 3.6 t/s | 5.1 t/s | 31.0% | 74.61 GiB | 119.09 ms |
| Q3_K_M | 16 GB | 8 | 4.5 t/s | 3.4 t/s | 42.2% | 62.53 GiB | 192.23 ms |
| UD-Q4_K_XL | 22 GB | 4 | 4.3 t/s | 4.5 t/s | 30.3% | 105.93 GiB | 346.22 ms |
| UD-Q4_K_XL | 22 GB | 8 | 5.6 t/s | 5.3 t/s | 44.1% | 84.97 GiB | 307.75 ms |
Detailed Flash-MoE Comparison
| Metric | Q3 / Bank 4 | Q3 / Bank 8 | Q4 / Bank 4 | Q4 / Bank 8 |
|---|---|---|---|---|
| Slot-bank cache hit | 31.0% | 42.2% | 30.3% | 44.1% |
| Cache misses / call | 2.76 | 2.31 | 2.79 | 2.24 |
| Routed expert data | 74.61 GiB | 62.53 GiB | 105.93 GiB | 84.97 GiB |
| Expert source I/O time | 60.33 s | 98.77 s | 181.54 s | 161.74 s |
| Expert upload time | 3.10 s | 3.61 s | 3.08 s | 2.37 s |
| Routed MoE time | 42.78 s | 64.83 s | 66.61 s | 57.22 s |
| Routed time / token | 119.09 ms | 192.23 ms | 346.22 ms | 307.75 ms |
| Routed bytes / token | 0.14 GiB | 0.12 GiB | 0.20 GiB | 0.16 GiB |
| Prefetch hit rate | 93.6% | 97.9% | 93.5% | 98.0% |
| Expert evictions | ~59k* | 49,057 | 59,369 | 47,419 |
*Approximately, based on the per-run Flash-MoE statistics.
Key Findings
Q3_K_M is clearly the better fit for a 16 GB M1 Pro. It requires substantially less expert I/O than UD-Q4_K_XL and produces much lower routed-MoE latency.
The best result in this test is Q3_K_M with slot-bank=4, reaching only 119 ms/token of accumulated routed-MoE overhead. Despite its lower 31% expert-cache hit rate, it was considerably faster than the Q3 slot-bank=8 run.
Increasing the slot bank from 4 to 8 consistently improves caching. Cache hit rates increased from roughly 30–31% to 42–44%, while prefetch hit rates reached approximately 98%. It also reduced the amount of expert data read from storage.
For UD-Q4_K_XL, slot-bank=8 is clearly preferable to slot-bank=4: routed overhead dropped from 346 ms/token to 308 ms/token, expert traffic fell from 105.9 GiB to 85.0 GiB, and the cache hit rate improved from 30.3% to 44.1%.
However, the benchmark is overwhelmingly storage-I/O bound. Expert source I/O accounts for approximately 96–98.5% of Flash-MoE routed processing time. Metal upload overhead is comparatively small.
Conclusion
| Category | Best Result |
|---|---|
| Best model for M1 Pro 16 GB | Q3_K_M |
| Lowest routed MoE latency | Q3_K_M / slot-bank 4 |
| Highest expert cache hit | Q4_K_XL / slot-bank 8 — 44.1% |
| Highest prefetch hit | Q4_K_XL / slot-bank 8 — 98.0% |
| Lowest expert traffic | Q3_K_M / slot-bank 8 — 62.53 GiB |
| Best Q4 configuration | UD-Q4_K_XL / slot-bank 8 |
Overall, Qwen3.5-35B-A3B is technically usable on a 16 GB M1 Pro through Flash-MoE expert streaming, even when the GGUF itself is 16–22 GB. The Q3_K_M quantization is considerably more practical on this hardware.
The main performance bottleneck is not GPU compute or Metal upload but expert data being streamed from storage. As a result, SSD I/O behavior and Flash-MoE cache effectiveness have a very large impact on generation performance.
Top comments (0)