The real constraint on running large models isn't "not smart enough" — it's "doesn't fit."
A 70B or 100B MoE model has weights in the hundreds of GB. A single GPU can't hold it, so you quantize, go cloud, go cluster — every step is cost and friction. For enterprises doing on-prem deployment, "doesn't fit" is the biggest wall.
Edge0 (2056 stars, Apache-2.0, Python, with an arXiv paper) attacks it from a property of MoE. Its one-liner: an open-source streaming MoE inference framework — put the experts on SSD, load them on demand.
How it shrinks the model
MoE models have a key property: each forward pass activates only a small fraction of experts — the rest are asleep. So why keep all of them in VRAM?
Edge0 leans into that with three moves:
- SSD expert offload — keep rarely-used expert weights on disk, load them only when needed, leaving only active experts in VRAM;
- Recover-LoRA — parallel LoRA adapters claw back the accuracy lost to offloading;
- Prerouter routing prediction — predict which experts fire next, hiding load time inside the inference pipeline.
The result: a big MoE model that "doesn't fit" runs on much smaller hardware. They ship two preview models — Edge0-35B-A3B (35B params, 3B active) and Edge0-8B-A1B — on both Hugging Face and ModelScope.
Three signals it's worth watching
- It hits the sharpest pain in on-prem deployment. Enterprises doing private LLMs fear expensive GPUs more than expensive models. SSD offload trades cheap disk for scarce VRAM — a very real cost lever for budget-sensitive, must-run-locally scenarios.
- It's engineered with a paper behind it. An arXiv paper (2609.18063), a full paper directory, benchmarks. "Theory first, then implementation" beats README-only projects by an order of magnitude.
- Its backend isolation is disciplined. It ships MLX first (Mac-first) and isolates the backend so more platforms can follow. "Make one platform solid" beats claiming "all platforms" from day one.
The honest caveat
It's early — Alpha — and currently Apple-first. 2056 stars, 184 forks, and the paper says preview. Real constraints: the current backend is MLX (Mac / Apple Silicon first, Linux/NVIDIA not fully there); 35B/8B are preview weights, not final; SSD offload has an accuracy/speed trade-off — not a free lunch.
It's for teams researching MoE inference optimization or doing "big model on small memory / on-device" pre-study, not a turnkey production system.
I've localized the README, paper, and core docs to Chinese: https://github.com/yangshun2005/Edge0-cn
If you find this project useful, a star on the original repo supports the author's ongoing maintenance.
Top comments (0)