Most "run a big model at home" projects are really compression projects. Quantize harder, prune, distill, and eventually a smaller model wearing a big model's name fits in your VRAM.
Colibri does something else, and the idea is worth understanding even if you never run it. It is an inference engine in pure C. One file. No BLAS, no Python at runtime, no GPU required. It runs Mixture-of-Experts models from 744 billion to 2.8 trillion parameters on hardware you already own.
Don't fit the model, place it
A 744B MoE model does not use 744B parameters per token. It activates roughly 40B, about 5.4%, and of those only around 11 GB actually change from one token to the next.
So the model never needed to fit in fast memory. It needed to be placed:
- The dense part, attention and shared experts and embeddings, about 17B parameters, stays resident in RAM at int4. Roughly 9.9 GB.
- The 19,456 routed experts, about 19 MB each, live on disk at around 372 GB and get streamed on demand, with a per-layer LRU cache and a learned hot-store of pinned experts.
The project's own analogy is the one that makes it click: a JIT, but for weights.
A compiler JIT never compiles the whole program. It watches what actually runs and compiles hot paths just in time. Colibri makes the same bet about a 744B parameter space. Parameters are not resident state to be held, they are data to be staged, exactly when needed. The router runs a layer ahead so prefetch hides the latency, and routing heat decides which experts earn which tier.
It works because expert routing has measurable structure, and structure is cacheable.
The numbers, which are the honest part
| Hardware | Decode speed |
|---|---|
| 6x RTX 5090, full residency | 5.8 to 6.8 tok/s |
| 128 GB CPU-only desktop | ~1.8 tok/s warm |
| Single RTX 5070 Ti, laptop class | 1.07 tok/s |
| 25 GB dev box | 0.05 to 0.1 tok/s cold |
That is the real answer to "can I run a 744B model at home?" Yes, at roughly one to two tokens per second.
Not a gotcha, just the trade. Interactive chat, unusable. Batch analysis, overnight jobs, or privacy-critical work where the alternative is shipping data to an API, entirely fine.
The barrier most people hit is not the GPU, which is optional. It is 372 GB of fast NVMe, because expert streaming is read-bound.
Full breakdown with the placement mechanics, the comparison against llama.cpp and Ollama, and the caveats worth knowing before you clear that much disk: Colibri: self-host a 744B LLM on hardware you already own.
If you are cleaning up model output afterwards, a Markdown converter saves hand-fixing every code fence.
Top comments (0)