A 744-billion-parameter AI model is supposed to need a data center. GLM-5.2, released by Z.ai, is a Mixture-of-Experts (MoE) model with 744B total parameters. Running it typically requires multiple H100 GPUs or hundreds of gigabytes of VRAM.
Colibri, an open-source project by JustVugg, runs this same model on a laptop with 25 GB of RAM and no GPU. Not fast. But correct.
How MoE Models Work (And Why Colibri Is Possible)
A traditional (dense) model activates all its parameters for every token. A 744B dense model would need all 744B parameters in memory for every single word it generates.
GLM-5.2 is different. It is a Mixture-of-Experts model with 19,456 expert sub-networks. For each token, the router picks only 8 experts per layer across 75 layers. The model activates roughly 40B parameters per token, only about 5.4% of the total.
This means only about 11 GB of expert weights actually change from token to token. The rest sit idle.
Colibri's insight is simple: if 94.6% of experts are not needed for any given token, why keep them all in memory?
The Three-Tier Memory Strategy
Colibri treats VRAM, RAM, and disk as one unified memory hierarchy:
- Dense part (attention, shared experts, embeddings, about 17B parameters): stays resident in RAM at int4 precision, consuming about 9.9 GB
- 19,456 routed experts (about 19 MB each at int4, 370 GB total on disk): streamed on demand with a per-layer LRU cache
- Optional VRAM tier: if you have a GPU, hot experts can live in VRAM for faster access
The engine keeps a learning cache. It records which experts your specific workload routes to (stored in a .coli_usage file) and pins the hottest ones in available RAM. The more you use it, the faster it gets.
Real Benchmark Numbers
All numbers below are measured by the community, not estimated. The full table with hardware configs is in the project benchmarks page.
| Machine | Speed |
|---|---|
| 6x RTX 5090, 251 GB RAM (full residency) | 5.8 to 6.8 tok/s |
| Apple M5 Max, 128 GB unified, Metal backend | 1.83 tok/s warm |
| Dell DGX Spark (Grace ARM), 121 GB | 2.4 tok/s |
| 128 GB CPU-only desktop | 1.8 tok/s warm |
| Single RTX 5070 Ti laptop | 1.07 tok/s |
| Ryzen 9 9950X, 123 GB, PCIe 5.0 NVMe | 1.23 tok/s |
| Intel Core Ultra 7, 24 GB RAM, WSL2 | 0.07 to 0.11 tok/s |
| Dev box baseline: 12 cores, 25 GB RAM | 0.05 to 0.1 tok/s cold |
The baseline (25 GB RAM, no GPU) is 0.05 to 0.1 tokens per second. That is roughly one word every 10 to 20 seconds. Slow, but it produces correct output from a 744B frontier model on hardware that costs less than a single H100 GPU fan.
What Makes Colibri Different
Pure C, Zero Dependencies
The runtime engine is a single C file (c/glm.c) plus small headers. No BLAS, no Python at runtime, no GPU required. Python is only used by the one-time model converter and the optional API gateway.
Never Silently Reduces Quality
The README states this explicitly: "Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics." The router makes the same decisions whether an expert answered from VRAM, RAM, or disk. The weights have the same precision. The output is identical.
Speculative Decoding Done Honestly
GLM-5.2 has a native Multi-Token Prediction (MTP) head that can draft tokens for the main model to verify. Colibri ships this with two rules learned through hard experience:
- The MTP head must be int8 (int4 heads collapse to near-zero draft acceptance)
- Draft and verify must compute the same function
The result is 2.2 to 2.8 tokens per forward pass when speculation pays off.
Token-Exact Validation
The forward pass is validated token-exact against a transformers oracle. 32 out of 32 test tokens match exactly. MLA attention stores a compressed KV state (576 floats per token instead of 32,768, which is 57x smaller) and persists it across restarts. Conversations reopen warm with zero re-prefill.
OpenAI-Compatible API
Colibri includes an OpenAI-compatible API server (coli serve). You can point any existing tool that speaks the OpenAI API at your local 744B model.
How to Run It
Step 1: Get the Engine
cd c && ./setup.sh
This checks for gcc/OpenMP, builds the engine, and runs self-tests.
Step 2: Get the Model
A pre-converted GLM-5.2 int4 container is available on HuggingFace:
https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
Make sure you get the version with int8 MTP heads. The original mirror ships int4 MTP heads which have near-zero draft acceptance.
Step 3: Run
# Interactive chat
COLI_MODEL=/path/to/glm52_i4 ./coli chat
# Check placement plan before running
COLI_MODEL=/path/to/glm52_i4 ./coli plan
# OpenAI-compatible API server
COLI_MODEL=/path/to/glm52_i4 ./coli serve
# Web dashboard with live metrics
COLI_MODEL=/path/to/glm52_i4 ./coli web
What the Web Dashboard Shows
The coli web command opens a dashboard with:
- Live token metrics: tokens per second, time to first token, per-turn breakdown
- VRAM/RAM/disk tier bar: how many experts are in each tier
- Brain page: all 19,456 experts as a visual cortex. Color shows the storage tier, brightness shows routing heat. Every expert that gets routed flashes white.
- Atlas page: measured expert affinity as a 3D galaxy. Experts cluster by topic (poetry, law, SQL, Chinese, etc.) based on actual routing data, not learned embeddings.
When Is Colibri Practical?
Practical for:
- Experimenting with a frontier model on consumer hardware
- Privacy-sensitive use cases (100% local, no API calls)
- Research and education on MoE architectures
- Use cases where correctness matters more than speed
Not practical for:
- Production serving (sub-1 tok/s is too slow for most real-time use)
- Your CI pipeline (370 GB model download takes hours)
- Anything requiring fast response times
Quality Cost of int4 Quantization
The README is honest about quality. The int4 container scored 62.5% mean accuracy on hellaswag, arc, and mmlu benchmarks. A measured fp16-vs-int4 A/B test showed the pure quantization cost at minus 8.2 percentage points, concentrated on the hardest tasks. Grouped scales recover about 63% of that loss.
This is not the same quality as running the full-precision model. It is a tradeoff: massive size reduction (370 GB on disk instead of an estimated ~1.5 TB at fp16, based on 744B parameters times 2 bytes each) at a measurable quality cost on difficult reasoning tasks.
The Bigger Picture
Colibri proves something important: the barrier to running frontier models is not as high as GPU vendors suggest. A 744B model does not need a data center if you accept slow token generation. The key insight is that MoE models are inherently sparse, and that sparsity can be exploited by treating storage as a memory tier.
As MoE models become the standard architecture (DeepSeek, GLM, Mixtral, and others all use MoE), expect more tools like Colibri that exploit this sparsity for consumer hardware.
The project has 16,800+ GitHub stars and active community benchmarking. It is Apache 2.0 licensed, and the GLM-5.2 model weights are MIT licensed.
Sources:
- Colibri GitHub: https://github.com/JustVugg/colibri (16,884 stars, Apache 2.0)
- Benchmark data: https://github.com/JustVugg/colibri/blob/main/docs/benchmarks.md
- GLM-5.2 model on HuggingFace: https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
- GLM-5.2 by Z.ai (MIT licensed weights)
Top comments (1)
The interesting part is not just fitting a huge model onto small hardware. It is what tradeoffs become acceptable when local experimentation is cheap enough that more people can test workflows before renting infrastructure.