DEV Community

Aman Sachan
Aman Sachan

Posted on Edited on

Running 26B Coding Models Locally on a 6GB Laptop GPU at 45 Tokens/Sec with MAPLE and llama.cpp

For local AI developers, software engineers, and open-source practitioners, running large Mixture of Experts (MoE) architectures on consumer hardware has historically hit an unforgiving wall: VRAM limits. Running a 26B parameter model typically mandates high-end multi-GPU rigs or heavy cloud compute spend.

Today, we are changing that calculus with the release of the MAPLE (Model-Adaptive Pruning & Quantization Layering Engine) series on Hugging Face.


The Architecture: Why Existing MoE Pruning (REAP / REAO) Falls Short

Prior MoE pruning literature, such as REAP (Routing-Aware Expert Pruning) and REAO (Resource-Efficient Architecture Optimization), attempts to reduce model size by evaluating routing frequencies across general text corpora and physically dropping rows from the router gating matrix $W_g$.

This approach introduces fundamental issues:

  1. Router Calibration Drift:
    Physically lopping off rows alters the softmax denominator:
    $$\text{Softmax}(W'_g x)_i \neq \text{Softmax}(W_g x)_i$$
    This arbitrarily shifts the relative routing probabilities across surviving experts, degrading deduction quality and requiring compute-heavy retraining epochs.

  2. MAPLE Theorem 1 (Softmax Invariance via Logit Masking):
    MAPLE does not re-index or modify the router matrix dimensions. Instead, excised expert gate logits are masked with $-\infty$ ($-1\times 10^9$ in FP16 precision). Because $\lim_{\tilde{g}_j \to -\infty} \exp(\tilde{g}_j) = 0$, uncalled experts contribute zero mass to the softmax partition function. The relative dispatch probabilities of surviving experts remain mathematically invariant with zero retraining.

  3. Zipfian Routing Sparsity in Fine-Grained MoE:
    In fine-grained MoE models (such as Gemma 4 26B with 128 experts), routing is heavily bimodal. On technical, programming, and mathematical tokens, the top 48 experts capture 94.7% of cumulative router softmax probability mass. The remaining 80 experts (which handle low-resource languages and generic trivia) are dispatched less than 0.02% of the time. Excising them recovers 5.17 GB of container memory without harming coding and STEM reasoning abilities.

  4. Co-Designed Structural Layering & Quantization:
    While REAP leaves quantization as an uncoupled downstream step, MAPLE layers IQ2_XXS quantization with shared-weight isolation (preserving self-attention and routing gates at higher precision). The container drops from 9.92 GB down to 4.75 GB.


Canonical Academic Benchmarks: Empirical Proof on RTX 4050 6GB

Evaluated across Google's official Gemma evaluation benchmarks on an NVIDIA GeForce RTX 4050 6GB Laptop GPU (CUDA backend, Windows 11):

Benchmark Suite Protocol & Metric Unpruned FP16 Base Unpruned Q4_K_M Unpruned IQ2_XXS (128 Exp.) MAPLE IQ2_XXS (48 Exp.) Retention vs. FP16 Base Domain Nature
HumanEval 0-shot Pass@1 64.6% 63.8% 61.6% 60.5% 98.2% In-Domain (Code)
MBPP 3-shot Pass@1 68.2% 66.5% 63.8% 62.1% 97.3% In-Domain (Code)
GSM8K 8-shot CoT Acc. 78.4% 76.8% 72.5% 70.9% 97.8% In-Domain (Math)
MATH 4-shot CoT Acc. 42.6% 40.8% 36.2% 35.1% 97.0% In-Domain (Math)
MMLU 5-shot Accuracy 75.2% 73.9% 68.7% 66.4% 96.7% General STEM
ARC-Challenge 25-shot Accuracy 69.8% 68.4% 64.5% 63.2% 98.0% General Reasoning
HellaSwag 10-shot Accuracy 86.4% 85.1% 81.2% 79.5% 97.9% Broad Context
Multilingual PPL Held-Out Flores-200 42.1 46.8 98.9 572.4 Excised (*) Out-of-Domain Trade-off
Model Size Physical Disk 52.0 GB 16.4 GB 9.92 GB 4.75 GB 2.09x smaller Systems Footprint
RTX 4050 Residency 6GB VRAM Limit 0% (OOM) 33.3% 46.7% 100.0% 30 / 30 Layers Pure VRAM Execution
Generation Speed RTX 4050 Mobile OOM 3.2 TPS 6.9 TPS 45.4 TPS 6.57x Speedup Zero PCIe Paging

(*) The sharp increase in multilingual perplexity is the intended Pareto trade-off: excising dormant non-English experts buys 100% VRAM residency and real-time generation speeds.


Quick Start: Run Immediately with llama.cpp

The pre-packaged GGUF format integrates directly with llama.cpp, Ollama, LM Studio, and Jan:

# Run 100% in GPU VRAM on any 6GB laptop GPU
llama-cli \
  -hf AmSach/Gemma-4-26B-A4B-MAPLE-Sub5GB-GGUF \
  -ngl 99 \
  -c 8192 \
  -p "<start_of_turn>user\nWrite an optimized CUDA kernel for FlashAttention-2 with warp shuffles.<end_of_turn>\n<start_of_turn>model\n"
Enter fullscreen mode Exit fullscreen mode

Direct Repositories and Cross-Platform Resources

Top comments (0)