Learned routing slashes the active memory of a 35‑billion‑parameter Mixture‑of‑Experts model to under 3 GiB, turning laptop‑scale inference from fantasy into practice. By predicting which experts will be needed one token ahead, the engine streams only the relevant weights from SSD and never holds the full parameter matrix in RAM.
Until now MoE inference on consumer hardware has been bottlenecked by weight storage rather than compute, with naive offloading offering no speed gain because layer N+1’s routing cannot start until layer N’s output is produced. A fully‑resident engine therefore demands more than 18 GiB of memory just to hold the int4‑quantized weights.
Edge0 reduces peak active memory from 18.2 GiB to 2.9 GiB while boosting decode throughput to 20.4 tok/s across five public benchmarks, staying within a few points of its fp16 teacher on average [1]. The quoted comparison—“the Mac mini M4 Pro has 24 GB…while Edge0’s profile decodes at 20.4 tok/s occupying 2.9 GiB”—captures the five‑fold speedup and the order‑of‑magnitude memory saving.
The quality gap introduced by int4 quantization is largely recovered through an unmerged LoRA trained on the student path, so the routed set exactly matches the staged expert set without dropping any weights [1]. This “prerouter” trick preserves most of the fp16 model’s accuracy while keeping memory tiny.
The technique hinges on a per‑layer prerouter that must be pre‑trained and on fast SSD streaming, which limits portability to machines lacking high‑throughput storage; moreover, 20 tok/s may still be too slow for real‑time latency‑critical workloads. These constraints leave open whether the same memory savings can be achieved on lower‑end disks or with larger MoE configurations.
If the memory wall no longer excludes extreme‑scale models from commodity inference, benchmark suites such as MMLU should add Edge0’s 35 B MoE as a baseline, reshaping how we evaluate accessibility versus performance.
Top comments (0)