Edge0: How a Prerouter and SSD Offloading Let a 35B MoE Model Run in 3 GB of RAM

Running a 35-billion-parameter Mixture-of-Experts (MoE) model on a consumer laptop sounds like a contradiction in terms. At 4-bit precision, those weights alone occupy roughly 19.5 GB — far beyond what most machines can hold in active memory. Yet Edge0, an open-source inference engine released this week, does exactly that: it runs a 35B MoE at 15–18 tokens per second while keeping peak active memory under 3 GB.

The trick is not compression alone. Edge0 combines SSD-based expert offloading with a trained routing predictor — the "prerouter" — that hides disk latency behind compute. The result is a practical system for running large sparse models on hardware that was never designed for them.

The Memory Wall in MoE Inference

Standard dense models scale memory linearly with parameter count. MoE models are different: only a small fraction of experts activate per token, so the compute cost stays manageable even as the total parameter count grows. The problem is that all those dormant experts still need to live somewhere, and "somewhere" is usually RAM.

For a 35B MoE like Qwen3.6-35B-A3B, the full weight set at int4 precision is around 19.5 GB. A Mac mini M4 Pro with 24 GB of unified memory can technically hold it, but that leaves almost nothing for the OS, the KV cache, or any other process. On a machine with 16 GB, it simply does not fit.

Edge0 reframes the problem: instead of treating RAM as the primary weight store, it treats the SSD as the primary tier and RAM as a bounded streaming pool. Expert weights are stored as int4 stacked safetensors on disk and mmap-streamed into a fixed-size active set on demand. Peak active memory tracks the working set — the experts currently in use — not the total parameter count.

The Prerouter: Hiding Disk Latency

SSD offloading introduces a new bottleneck. In standard MoE execution, the routing decision for layer N+1 depends on the output of layer N. If expert weights must be fetched from disk, the system stalls: it cannot start loading the next layer's experts until the current layer finishes computing.

Edge0 solves this with the prerouter, a small per-layer head trained to predict the routing for the next layer one token ahead of time. Because the prediction is available before it is needed, the system can initiate SSD reads for the upcoming experts while the current layer's forward pass is still running. Disk latency is hidden behind compute rather than added to it.

The prerouter's prediction also replaces the standard router's output at decode time — it is not just a prefetch hint but the actual routing decision. This means the system never waits for the standard router to finish before starting the next load. According to the paper, this overlap increases decode throughput by up to 59% compared to on-demand loading.

To avoid the quality loss that comes from routing approximation and int4 quantization, Edge0 adds Unmerged Recovery LoRA adapters. These are trained via distillation from an FP16 teacher while the quantized base weights remain frozen. Crucially, the adapters are kept as unmerged parallel delta branches rather than merged back into the quantized weights — merging would require re-quantization, which degrades quality. The inference path is int4 base + prerouter routing + LoRA delta, all applied in a single forward pass.

Fixed-Slot Double Buffering

A subtler engineering detail is how Edge0 manages the streaming pool itself. Naive implementations rebuild the layer's tensor stack at every decode step, which adds overhead. Edge0 uses fixed-slot double buffering: routing indices are mapped through a slot table, and the system performs in-place slot updates (incr_stack) rather than rebuilding from scratch. This eliminates the per-step stack-rebuild cost and keeps the memory layout stable across tokens.

Performance on Consumer Hardware

The Edge0 GitHub repository reports the following benchmarks on a Mac mini M4 Pro (24 GB unified memory), using a ~3,300-token prompt and 200 timed decode tokens:

Model tier	Decode speed	Peak active memory
edge0-35b	14.9–17.7 tok/s	~2.9 GiB
edge0-8b	23.9–25.3 tok/s	~1.0 GiB

The 35B model runs at roughly 15–18 tokens per second — usable for interactive inference — while consuming less than 3 GB of active memory. The 8B tier, based on the Ling 3.0 bailing hybrid architecture, reaches 24–25 tokens per second in just 1 GB.

Quality benchmarks using OpenCompass show that the int4 pipeline retains high fidelity relative to FP16 baselines, with an average quality drop of approximately 3.9 points for the 35B model across tasks like MMLU-Pro and HumanEval. That is a meaningful but not catastrophic gap, and one that the recovery LoRA is specifically designed to close.

What the Prerouter Gets Right Architecturally

The prerouter is worth examining as a design pattern beyond Edge0. The core insight is that routing in MoE models is highly predictable: the same input regions tend to activate the same experts across tokens. A small trained head can exploit this regularity to predict the next routing decision with enough accuracy to use the prediction as the actual decision — not just a hint.

This is different from speculative decoding, which predicts tokens and verifies them. The prerouter predicts routing indices and commits to them. There is no verification step, which means any prediction error propagates directly into the output. The paper reports that routing accuracy is high enough in practice that the quality impact is small, and the recovery LoRA absorbs most of the residual error.

The approach also generalizes naturally to other offloading scenarios. Any system that needs to prefetch data based on a future decision — whether that data lives on SSD, in a remote cache, or across a network — can benefit from a lightweight predictor that makes the decision one step early.

Practical Implications

Edge0 is currently implemented on the MLX backend for Apple Silicon, with the architecture designed to be backend-agnostic. The project is open-source under Apache 2.0, and both the 35B and 8B model tiers are available as end-to-end pipelines including the base checkpoint, LoRA adapters, and prerouter heads.

For practitioners, the immediate takeaway is that the hardware threshold for running large sparse models has dropped significantly. A machine with 24 GB of unified memory and a fast SSD can now run a 35B MoE at interactive speeds — not as a research demo but as a deployable inference setup. The SSD read bandwidth becomes the primary constraint, which means NVMe drives matter more than RAM capacity for this use case.

The broader implication is that the "memory wall" for MoE inference is not a fixed barrier. It is an engineering problem, and Edge0 demonstrates one concrete solution: treat storage as a tiered memory system, predict routing decisions early enough to hide latency, and use distillation-trained adapters to recover quality. Each of those components is independently useful, and the combination makes large sparse models accessible on hardware that most developers already own.

---

Primary source: Edge0 paper (arXiv 2609.18063) | GitHub repository | Model page | HuggingFace model