Running big models is the root of "compute anxiety": you want a 70B or 100B model, you need the GPUs to match, and a single machine can't cut it — so you go cloud, go cluster.
SwarmLLM (416 stars, MIT, JavaScript) proposes something different: if one machine can't run it, let every device in the room help.
Its one-liner: peer-to-peer LLM inference — every device brings a slice, and together they run the whole model.
How it works
SwarmLLM splits a large model across multiple devices on the same network, each computing a slice, then reassembles the result. Two hard pieces of engineering:
- A from-scratch WebGPU inference engine — model inference inside the browser, nothing to install;
- A WebRTC peer-to-peer runtime — devices talk directly to each other, splitting a 27B model across them.
The kicker: no app install, no environment setup — just open a browser tab. The official demo is concrete: a MacBook and an iPhone, each with a browser tab open on the same Wi‑Fi, running Qwen 3.8 27B at 10.7 tok/s for 400 tokens.
Three signals it's worth watching
- It reimagines distributed inference. Traditional distributed = pile up GPUs and spend. SwarmLLM = pool the idle devices you already own. The former burns money; the latter reuses spare compute at near-zero cost. That's naturally appealing to individual devs, small teams, and "many devices but none of them strong" scenarios.
- The engineering bar is high. Writing a WebGPU inference engine and a WebRTC P2P runtime from scratch are both nontrivial. An open-source project that lands both has real substance.
- It carries a clean decentralization story. No central server, no cloud sign-up, the model flows between your own devices. That rhymes with the bigger "data sovereignty" and "on-device AI" direction — even though it's still early.
The honest caveat
It's early and experimental — not a production inference system. 416 stars, 58 forks, and the demo's 27B split is mostly a feasibility proof; 10.7 tok/s and stability are a long way from production. Browser-tab distribution also has hard constraints: devices must share a network, stay online, and volunteer compute. It's for studying and understanding the "P2P inference" direction, not for replacing your current inference deployment.
I've localized the README and architecture docs to Chinese: https://github.com/yangshun2005/swarmllm-cn
If you find this project useful, a star on the original repo supports the author's ongoing maintenance.
Top comments (0)