Liedson Habacuc

Posted on Jul 19

# Colibri Made Me Rethink How We Run LLMs — And There Might Be a Next Step

Over the past few days, I came across a project that made me stop everything to understand how it worked. It wasn't just another framework for running language models, nor another GPU-based optimization. It was something different.

The project is called Colibri.

At first glance, it almost seems impossible. The idea is simple, yet remarkably bold: run a massive Mixture of Experts (MoE) model on an ordinary computer using nothing more than a CPU and an SSD, loading into memory only the experts that are actually needed at each step.

When I first read that, my reaction was probably the same as everyone else's:

"How could that possibly work?"

After studying the architecture, I realized that Colibri's real innovation isn't simply that it can run an enormous model. The real breakthrough is that it completely changes the way we think about inference.

A Shift in Perspective

For years, we've been taught that running a model means loading the entire model into memory.

More VRAM.

More RAM.

More GPUs.

Colibri challenges that assumption.

Instead of keeping hundreds of billions of parameters resident in memory, it treats the model as a living system.

The router selects which experts are needed.

Those experts are loaded from the SSD.

They perform their computation.

And once they're no longer needed, they can be discarded.

It's an incredibly elegant approach.

In practice, storage is no longer just a place where model weights are kept—it becomes an active participant in the inference process.

It's one of those ideas that feels both simple and brilliant.

The Bottleneck Isn't the Architecture

Once I understood how it worked, another question came to mind.

What if the real problem isn't the algorithm?

Because the architecture itself makes perfect sense.

The bottleneck appears somewhere else.

Every time an expert is needed, there's an unavoidable sequence:

SSD → RAM → CPU

That process happens thousands of times during a single generation.

In other words, the real limitation is no longer computational power—it's the time spent waiting for data to move from storage into memory.

That's when I started wondering what the next step for this architecture could be.

What If Colibri Could Predict the Future?

Modern CPUs already do this.

They try to predict which instruction will be executed next.

Operating systems do it.

Databases do it.

Web browser caches do it.

So why couldn't an MoE inference engine do the same?

Imagine that after processing thousands of tokens, the system discovers patterns like these:

Expert 15
 ↓ 92%
Expert 48

Expert 48
 ↓ 87%
Expert 102

Expert 102
 ↓ 95%
Expert 211

In this scenario, the engine wouldn't need to wait for the next token before loading the next expert.

While the CPU is computing the current expert, another thread could already be loading the most likely next experts.

By the time they're needed, they're already sitting in RAM.

The waiting time nearly disappears.

Storage Would No Longer Be the Bottleneck

Today, the SSD is used reactively.

First, the router makes a decision.

Then, the SSD responds.

But what if we reversed that logic?

Instead of reacting to requests, the system begins anticipating them.

That transforms storage from a passive component into a proactive one.

One possible architecture could consist of four independent modules:

An intelligent scheduler, responsible for predicting upcoming experts.
An asynchronous I/O manager, performing disk reads in parallel.
A smart expert cache (LRU/LFU), keeping frequently used experts in memory.
A dedicated inference engine, focused solely on computation.

This separation would allow disk I/O and computation to happen simultaneously, dramatically reducing idle time.

Other Interesting Ideas

As I explored the project further, a few additional possibilities came to mind.

Frequency-Based Caching

Experts that are used hundreds of times shouldn't be evicted from memory as aggressively as experts that are rarely selected.

Intelligent Prefetching

As soon as the router makes its decision, immediately begin loading the most likely upcoming experts.

Memory-Mapped I/O (mmap)

Allow the operating system to manage page caching directly, reducing unnecessary memory copies and making better use of the existing virtual memory subsystem.

Physical Expert Clustering

Experts that are frequently activated together could be stored physically close to one another on disk, enabling much more efficient sequential reads.

A Fully Pipelined Engine

While one group of experts is executing...

Another group is already being loaded.

And a third group is waiting in memory.

Ideally, the CPU would spend almost no time waiting for the SSD.

The Core Idea Remains the Same

The most interesting part is that none of these ideas changes Colibri's original philosophy.

The entire model still isn't loaded into memory.

Inference still runs entirely on CPUs.

The SSD remains an active part of the inference pipeline.

The difference is that the system stops reacting to events and starts anticipating them.

It's an evolution of the same idea.

Not a replacement.

Perhaps This Is Only the Beginning

Colibri demonstrates that there's still plenty of room for innovation in inference engines.

For years, the conversation has revolved around bigger GPUs, more memory, and greater computational power.

Maybe there's another path.

Maybe we can build increasingly intelligent systems for moving data.

Because in the end, the hardest problem may not be computing faster.

It may simply be ensuring that the right data is in the right place at exactly the right time.

That's what made Colibri so fascinating to me.

And perhaps that's exactly where the next major step in this idea lies.

Congratulations to the creator of the project for the creativity and the courage to challenge a paradigm that many considered unquestionable. I hope these ideas inspire further discussion and experimentation, because I believe architectures like this still have tremendous room to evolve.

JustVugg / colibri

Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.

Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM RAM, and storage as one managed memory hierarchy. Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics.

$ ./coli chat
  🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
  ✓ ready in 32s · resident 9.9 GB
  › ciao
  ◆ Ciao! 😊 Come posso aiutarti oggi?

See it running

The web dashboard (./coli web): a 744B model answering at 4+ tok/s end-to-end on 6× RTX 5090 — with live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.