For local AI developers, software engineers, and open-source practitioners, running large Mixture of Experts (MoE) architectures on consumer hardware has historically hit an unforgiving wall: VRAM limits. Running a 26B parameter model typically mandates high-end multi-GPU rigs or heavy cloud compute spend.
Today, we are changing that calculus with the release of the MAPLE (Model-Adaptive Pruning & Quantization Layering Engine) series on Hugging Face.
The Architecture: Why Existing MoE Pruning (REAP / REAO) Falls Short
Prior MoE pruning literature, such as REAP (Routing-Aware Expert Pruning) and REAO (Resource-Efficient Architecture Optimization), attempts to reduce model size by evaluating routing frequencies across general text corpora and physically dropping rows from the router gating matrix $W_g$.
This approach introduces fundamental issues:
Router Calibration Drift:
Physically lopping off rows alters the softmax denominator:
$$\text{Softmax}(W'_g x)_i \neq \text{Softmax}(W_g x)_i$$
This arbitrarily shifts the relative routing probabilities across surviving experts, degrading deduction quality and requiring compute-heavy retraining epochs.MAPLE Theorem 1 (Softmax Invariance via Logit Masking):
MAPLE does not re-index or modify the router matrix dimensions. Instead, excised expert gate logits are masked with $-\infty$ ($-1\times 10^9$ in FP16 precision). Because $\lim_{\tilde{g}_j \to -\infty} \exp(\tilde{g}_j) = 0$, uncalled experts contribute zero mass to the softmax partition function. The relative dispatch probabilities of surviving experts remain mathematically invariant with zero retraining.Zipfian Routing Sparsity in Fine-Grained MoE:
In fine-grained MoE models (such as Gemma 4 26B with 128 experts), routing is heavily bimodal. On technical, programming, and mathematical tokens, the top 48 experts capture 94.7% of cumulative router softmax probability mass. The remaining 80 experts (which handle low-resource languages and generic trivia) are dispatched less than 0.02% of the time. Excising them recovers 5.17 GB of container memory without harming coding and STEM reasoning abilities.Co-Designed Structural Layering & Quantization:
While REAP leaves quantization as an uncoupled downstream step, MAPLE layers IQ2_XXS quantization with shared-weight isolation (preserving self-attention and routing gates at higher precision). The container drops from 9.92 GB down to 4.75 GB.
Canonical Academic Benchmarks: Empirical Proof on RTX 4050 6GB
Evaluated across Google's official Gemma evaluation benchmarks on an NVIDIA GeForce RTX 4050 6GB Laptop GPU (CUDA backend, Windows 11):
| Benchmark Suite | Protocol & Metric | Unpruned FP16 Base | Unpruned Q4_K_M | Unpruned IQ2_XXS (128 Exp.) | MAPLE IQ2_XXS (48 Exp.) | Retention vs. FP16 Base | Domain Nature |
|---|---|---|---|---|---|---|---|
| HumanEval | 0-shot Pass@1 | 64.6% | 63.8% | 61.6% | 60.5% | 98.2% | In-Domain (Code) |
| MBPP | 3-shot Pass@1 | 68.2% | 66.5% | 63.8% | 62.1% | 97.3% | In-Domain (Code) |
| GSM8K | 8-shot CoT Acc. | 78.4% | 76.8% | 72.5% | 70.9% | 97.8% | In-Domain (Math) |
| MATH | 4-shot CoT Acc. | 42.6% | 40.8% | 36.2% | 35.1% | 97.0% | In-Domain (Math) |
| MMLU | 5-shot Accuracy | 75.2% | 73.9% | 68.7% | 66.4% | 96.7% | General STEM |
| ARC-Challenge | 25-shot Accuracy | 69.8% | 68.4% | 64.5% | 63.2% | 98.0% | General Reasoning |
| HellaSwag | 10-shot Accuracy | 86.4% | 85.1% | 81.2% | 79.5% | 97.9% | Broad Context |
| Multilingual PPL | Held-Out Flores-200 | 42.1 | 46.8 | 98.9 | 572.4 | Excised (*) | Out-of-Domain Trade-off |
| Model Size | Physical Disk | 52.0 GB | 16.4 GB | 9.92 GB | 4.75 GB | 2.09x smaller | Systems Footprint |
| RTX 4050 Residency | 6GB VRAM Limit | 0% (OOM) | 33.3% | 46.7% | 100.0% | 30 / 30 Layers | Pure VRAM Execution |
| Generation Speed | RTX 4050 Mobile | OOM | 3.2 TPS | 6.9 TPS | 45.4 TPS | 6.57x Speedup | Zero PCIe Paging |
(*) The sharp increase in multilingual perplexity is the intended Pareto trade-off: excising dormant non-English experts buys 100% VRAM residency and real-time generation speeds.
Quick Start: Run Immediately with llama.cpp
The pre-packaged GGUF format integrates directly with llama.cpp, Ollama, LM Studio, and Jan:
# Run 100% in GPU VRAM on any 6GB laptop GPU
llama-cli \
-hf AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF \
-ngl 99 \
-c 8192 \
-p "<start_of_turn>user\nWrite an optimized CUDA kernel for FlashAttention-2 with warp shuffles.<end_of_turn>\n<start_of_turn>model\n"
Direct Repositories and Cross-Platform Resources
- Gemma 4 26B MAPLE GGUF on Hugging Face: https://huggingface.co/AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF
- DeepSeek Coder V2 Lite MAPLE GGUF: https://huggingface.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
- Hugging Face MoE Collection: https://huggingface.co/collections/AmSach/maple-sub-5gb-mixture-of-experts-models
- Hashnode In-Depth Research Post: https://amsach.hashnode.dev/running-26b-moe-models-on-consumer-6gb-gpus-inside-maple-s-sub-5gb-breakthrough
- Substack Architectural Paper: https://amsach.substack.com/p/running-26b-moe-models-on-consumer
- Coder Legion Thread: https://coderlegion.com/35307/running-26b-moe-models-on-consumer-6gb-gpus-inside-maples-sub-5gb-breakthrough
- Medium Article: https://medium.com/@amansachan92905/running-26b-moe-models-on-consumer-6gb-gpus-inside-maples-sub-5gb-breakthrough-f422990e5afd
Top comments (0)