Introduction to High-Parameter Inference on Low-RAM Systems
Recent developments in machine learning have challenged the conventional assumption that running massive Mixture-of-Experts (MoE) models requires workstation-class memory kits. The Colibri project has emerged as a groundbreaking approach for running Z.ai’s GLM-5.2 model—a colossus with 744 billion parameters—on standard hardware equipped with only 25 GB of RAM. By decoupling the dense model backbone from the sparse expert layers, engineers have successfully moved the heavy lifting from memory to disk I/O.
Architecture Fundamentals: The Disk-Streaming Paradigm
Unlike traditional inference engines that attempt to map model weights entirely into VRAM or system RAM, Colibri utilizes a granular streaming strategy. The architecture splits the model into two distinct segments:
- The Dense Backbone: Contains the attention mechanisms, embeddings, and shared experts. These are quantized to int4 and kept resident in memory (~9.9 GB footprint).
- The Routed Experts: Consists of 21,504 experts. These reside on a high-speed NVMe SSD and are loaded into an LRU cache on demand based on router activation.
This approach effectively turns a memory-bound problem into an I/O-bound one. The engine, written in pure C with zero runtime dependencies, leverages OpenMP for multi-core processing, avoiding the overhead of heavy frameworks.
Implementation Strategy and Performance
Performance is highly dependent on disk bandwidth rather than raw clock speed. During cold-cache operations, expect performance in the 0.05–0.1 tokens/second range. However, as the expert-pinning heuristics populate the "hot store" in memory for frequent modules, throughput increases significantly to 2–4 tokens/second. Key technical optimizations include:
- Speculative Multi-token Prediction: Boosts output generation by drafting multiple tokens per forward pass.
- KV-cache Persistence: Ensures state recovery across process restarts.
- Router-lookahead Prefetch: Attempts to predict expert usage to mitigate latency.
Deployment via the API Server
Colibri ships with coli serve, an OpenAI-compatible API server. To set up and run the service locally, follow these steps:
git clone https://github.com/JustVugg/colibri.git
cd colibri/c
./setup.sh
# Run the server with restricted access
COLI_MODEL=/path/to/weights COLI_API_KEY=your-key ./coli serve --host 127.0.0.1 --port 8000
Exposing Local Inference Safely
Because coli serve binds to localhost by default for security, accessing the model while away from your workstation requires a secure tunneling solution. Using Pinggy, you can expose your local server to the public internet securely without complex router configurations:
ssh -p 443 -R0:localhost:8000 free.pinggy.io
This generates a temporary public HTTPS URL, allowing you to poll your long-running batch jobs from mobile devices or external clients. For production-like use cases, upgrading to a persistent domain allows for stable API integration and secondary authentication layers.
Best Practices and Caveats
Running large-scale models via disk streaming is ideal for batch processing, document summarization, and long-horizon tasks where real-time chat latency is not the priority. It is recommended to use high-end NVMe drives to ensure the I/O throughput is sufficient to prevent severe bottlenecking during peak router activity. SSD wear concerns are generally mitigated by the read-heavy nature of weight loading, though continuous logging and KV-cache writes should be monitored on resource-constrained systems.
Reference
Self hosting a 744B param LLM with only 25 GB RAM
A single-file C engine called Colibrì streams GLM-5.2's 744B mixture-of-experts weights off an NVMe drive to run the full model on 25 GB of RAM at 0.05-2 tokens/second. Here's how it works, what Hacker News made of it, and how to check on a queued run from your phone with Pinggy.
pinggy.io
Top comments (0)