DEV Community

Cover image for Colibri: Running a 744B AI Model on Your Laptop
Md Jamilur Rahman
Md Jamilur Rahman

Posted on

Colibri: Running a 744B AI Model on Your Laptop

#opensource #webdev #ai #programming

GLM-5.2 is a frontier AI model with 744 billion parameters. It normally requires H100 GPUs, hundreds of gigabytes of VRAM, and a cloud bill larger than a car payment. Colibri runs it on a laptop with 25 GB of RAM.

The secret? It never loads the whole model.

What Is Colibri?

Colibri is a pure C inference engine for GLM-5.2, a Mixture-of-Experts (MoE) model released by Z.ai. It streams model experts from disk instead of keeping them in memory, trading disk I/O for RAM.

"This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan."
— Colibri README

How It Works: The MoE Architecture

GLM-5.2 uses Mixture-of-Experts. For each token, only a small subset of "experts" are active. The model has:

  • 21,504 routed experts (19 MB each at int4) — stored on disk (~370 GB)
  • Dense parts (attention, shared experts, embeddings) — resident in RAM (~10 GB)

The key insight: only ~11 GB of parameters change token-to-token. Those 11 GB are the routed experts. The remaining 34 GB are dense layers that stay loaded.

Colibri keeps the dense parts in RAM and streams experts from NVMe SSD on demand. It uses per-layer LRU caching, optional pinned hot-store for frequently used experts, and relies on the OS page cache as a free L2 cache.

Technical Specifications

Metric Value
Model on disk (int4) ~370 GB
Resident RAM 9.9 GB
Minimum RAM 25 GB
Load time ~30 seconds
Peak RSS during chat ~20 GB (auto-capped)
Cold decode cost ~11 GB disk reads/token
MTP speculation (int8 head) 2.2–2.8 tok/forward

Key Features

  1. Faithful GLM-5.2 forward — token-exact validation against transformers oracle
  2. MLA attention with compressed KV-cache (576 floats/token vs. 32,768 — 57× smaller)
  3. DeepSeek-V3-style sigmoid router with shared expert
  4. Native MTP speculative decoding — GLM-5.2's multi-token-prediction head drafts tokens verified in batched forward
  5. Grammar-forced speculative drafts for constrained JSON/NDJSON output
  6. Integer-dot kernels — Q8_0-style int8 activations, AVX2 maddubs
  7. MLA weight absorption for decode
  8. Async expert readahead
  9. Quantization kernels: int8 / packed int4 / packed int2
  10. DSA sparse attention — GLM-5.2's lightning indexer
  11. KV-cache persistence — conversations reopen warm across engine restarts
  12. Router-lookahead prefetch (experimental) — 71.6% predictable routing

Platform Support

Platform Status Notes
Linux ✅ Full Primary platform
macOS ✅ Full Metal backend option
Windows 11 ✅ Native MinGW-w64 with _WIN32 compatibility layer

Requirements: gcc with OpenMP, AVX2, ≥16 GB RAM, ~370 GB model on local NVMe.

How to Use

Quick start:

cd c
./setup.sh  # Build and self-tests
COLI_MODEL=/path/to/glm52_i4 ./coli chat
Enter fullscreen mode Exit fullscreen mode

OpenAI-compatible API:

COLI_MODEL=/nvme/glm52_i4 ./coli serve \
  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
Enter fullscreen mode Exit fullscreen mode

Quality benchmarks:

./coli bench  # hellaswag, arc_challenge, mmlu
Enter fullscreen mode Exit fullscreen mode

Performance: Slow but Correct

This is the trade-off. Colibri is not fast. Performance depends heavily on disk speed, RAM capacity, and expert caching:

Hardware Tokens/second
Apple M5 Max (128 GB RAM, Metal) 1.06–1.83 tok/s
Ryzen AI 9 HX 370 (128 GB RAM) 0.37 tok/s
Ryzen 9 9950X (PCIe 5.0 NVMe) 0.28 tok/s

For context, a cloud H100 inference might run at 30–50 tok/s. Colibri is 10–100× slower. But it costs nothing after setup and runs entirely locally.

Why This Matters

Frontier AI models are becoming inaccessible. GPT-4, Claude 3.5, GLM-5.2 — these require proprietary APIs, monthly subscriptions, and internet access. Your data leaves your machine. Your conversations are logged.

Colibri proves that with clever engineering, you can run a frontier model on consumer hardware. You need:

  • A laptop with ≥25 GB RAM
  • A fast NVMe SSD with ~370 GB free space
  • Patience (0.3–2 tok/s)

The model answers correctly. The quality is there. The trade-off is speed.

Real-World Use Cases

Offline coding assistant: Run GLM-5.2 locally without API keys or quotas. Your code never leaves your machine.

Privacy-first knowledge base: Build a RAG system with local retrieval and local generation. Zero data exfiltration.

Benchmarking research: Test frontier model quality without cloud costs. Run hellaswag, arc_challenge, mmlu locally.

Edge deployment: Install on a rugged laptop for field deployment where internet is unreliable.

What Makes This Possible

Colibri is a one-person project built on a 12-core laptop with 25 GB of RAM. The author (JustVugg) exploited the Mixture-of-Experts architecture:

  • MoE models only activate ~40B parameters per token
  • Only ~11 GB change token-to-token
  • The rest is dense, shared layers that stay resident
  • Streaming experts from disk is viable with NVMe

Quantization (int4) reduces the model from ~1.5 TB to ~370 GB on disk. MLA attention compresses KV-cache by 57×. Speculative decoding with MTP head drafts 2–3 tokens per forward pass.

These techniques are not new. Colibri combines them into a single engine optimized for streaming experts.

What Is Missing

Colibri does not try to be fast. It does not support LoRA fine-tuning, RLHF alignment loading, or custom quantization recipes. It is purpose-built for one thing: running GLM-5.2 faithfully on minimal hardware.

For BD developers with cheap VPS or older laptops, this is more theoretical than practical. You need 370 GB of NVMe storage and a modern CPU with AVX2. But it shows the direction: frontier models can run locally if we stop loading them entirely into RAM.

The Future of Local AI

Colibri is a proof of concept. It proves that streaming experts from disk works. It proves that int4 quantization preserves quality. It proves that MLA attention compression is viable.

The next step is making it fast. Better caching, smarter prefetching, GPU acceleration for matmul, multi-threaded expert loading. Each improvement inches closer to "fast enough" for real work.

But even at 0.3 tok/s, Colibri is useful. It runs a 744B model on a machine that costs less than a single H100 GPU fan. That is remarkable.

How to Try It

If you have the hardware, here is the path:

  1. Clone the repo: git clone https://github.com/JustVugg/colibri
  2. Get GLM-5.2 weights: Download from Z.ai (released under MIT)
  3. Quantize to int4: Use the tools in tools/ directory
  4. Run setup: cd c && ./setup.sh
  5. Chat: COLI_MODEL=/path/to/glm52_i4 ./coli chat

You will need patience. The first token takes 30–60 seconds. Subsequent tokens are faster. But the answers are correct. The quality is there.

Conclusion

Colibri is not for everyone. It requires 370 GB of NVMe storage, a modern CPU, and patience. But it proves that frontier AI models do not require cloud infrastructure.

A one-person project, a 12-core laptop, and clever engineering can run a 744B model that would otherwise cost tens of thousands in GPU time. That is the story of local AI.

The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. Colibri keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.

Sources

  • GitHub: JustVugg/colibri
  • GLM-5.2 weights released by Z.ai under MIT license
  • Apache 2.0 license for Colibri engine

Top comments (0)