Lumabri: Running Mixture-of-Experts Models on a P2P Swarm of Consumer GPUs

A Show HN post for Lumabri — a project that lets you run Mixture-of-Experts (MoE) models on a P2P swarm of consumer GPUs — just appeared on Hacker News. While it hasn't hit the front page yet (44 points, 18 comments), the concept is worth examining because it represents a growing trend: distributed inference as an alternative to centralized AI.

What Lumabri Does

Lumabri, built on top of the Colibri framework, enables distributed inference of MoE models across multiple machines connected via peer-to-peer networking. Instead of needing one massive GPU (or a cluster of them), you can pool together multiple smaller GPUs:

• Machine A (RTX 3060, 12GB) handles experts 1-8
• Machine B (RTX 4070, 16GB) handles experts 9-16
• Machine C (MacBook Pro, 32GB unified) handles experts 17-24
• They coordinate via P2P to serve the full model

Why This Matters

The bottleneck for running large AI models has always been GPU memory. A 70B parameter model needs ~140GB in FP16 — that's three A100 80GB GPUs at ~$10,000 each. Most developers and small companies simply can't afford that.

MoE (Mixture-of-Experts) architectures offer a partial solution. Instead of activating all parameters for every token, MoE models activate only a subset of "experts" for each input. This means:

• DeepSeek V3 (671B total params, ~37B active): You need to load all 671B into memory, but only ~37B are active per token
• Mixtral 8x22B: 141B total, ~39B active per token

The challenge is that you still need to load all parameters into memory, even if only a fraction are active. That's where distributed inference comes in.

How P2P Swarm Inference Works

The concept is elegant:

1. Model partitioning: Split the model's experts across available GPUs
2. P2P networking: Machines communicate directly without a central coordinator
3. Dynamic routing: When a token needs expert #5, the request is routed to whichever machine holds that expert
4. Load balancing: Experts are distributed based on each machine's available memory and compute

The result: a 70B+ model can run on 3-4 consumer GPUs that individually could never handle it.

The Colibri Framework

Lumabri is built on Colibri, which provides the underlying P2P infrastructure:

• Peer discovery: Find other machines on the network (or internet) willing to contribute GPU time
• Communication layer: Efficient tensor serialization and transfer between nodes
• Fault tolerance: If one machine drops out, experts can be re-distributed
• Scheduling: Intelligent routing of computation to minimize network latency

Practical Implications

For the open-source AI community, this is a big deal:

Research labs:

• Run large models on existing hardware instead of buying new GPUs
• Experiment with 100B+ parameter models using lab workstations
• Avoid cloud GPU costs during development

Small companies:

• Pool employee workstations for after-hours inference
• Build private inference clusters without datacenter investment
• Scale incrementally by adding machines

Individual developers:

• Run large models across a desktop GPU + laptop GPU + cloud GPU
• Participate in distributed inference networks (similar to SETI@home but for AI)
• Access frontier-scale models without frontier-scale budgets

Challenges and Limitations

The Hacker News discussion surfaces several concerns:

Network latency:

• P2P communication adds latency vs. NVLink/PCIe
• Home internet (50-100Mbps) is much slower than inter-GPU links (600GB/s)
• This makes Lumabri better for batch processing than real-time chat

Reliability:

• Consumer machines go to sleep, lose network, crash
• P2P networks need robust fault tolerance
• No guarantees on uptime or performance

Security:

• Sending model weights and activations over the network raises privacy concerns
• Need encryption and authentication
• Can you trust other peers in the swarm?

Complexity:

• Setting up P2P networking is harder than ollama run
• Debugging distributed systems is notoriously difficult
• Still very much a research project, not production-ready

The Bigger Trend: Distributed AI

Lumabri is part of a broader movement toward distributed AI inference:

• Petals: Distributed inference for large models (been around since 2023)
• Exo: Run models across Apple devices using Thunderbolt/Thunderbolt networking
• llama.cpp RPC: Distributed inference across multiple machines
• HuggingFace TGI: Supports multi-GPU inference (though not P2P)

The vision is compelling: instead of a few companies owning all the AI compute, distribute it across thousands of consumer machines. It's the same philosophy that powered SETI@home and Folding@home, but applied to neural networks.

Should You Try It?

If you have:

• Multiple machines with GPUs (even modest ones)
• A local network (Ethernet preferred over WiFi)
• Comfort with command-line tools and Python

Then yes, give it a try. The project is on GitHub at JustVugg/lumabri.

For everyone else, watch this space. P2P inference is one of those ideas that seems niche until suddenly it isn't.

---

Have you tried distributed inference? What was your experience? The comments are open — I'd love to hear from anyone running models across multiple machines.