WISP — Stream What Shouldn't Run
Last week JustVugg dropped Colibrì — a 2,400-line pure C
engine that proved a 744B parameter model could run on 25GB
of consumer RAM by streaming expert weights from disk.
It blew my mind. So I built on top of that concept.
This is WISP.
The Problem
The largest open source AI models in the world — GLM-5.2
(744B), DeepSeek-V3 (671B), Kimi K3 (2.8T) — require
datacenter hardware to run. Most people trying to run them
locally hit a wall immediately.
The standard approaches all make the same tradeoff:
quantize aggressively until the model fits, or don't run it.
Both options sacrifice intelligence for accessibility.
The Insight (Credit: JustVugg / Colibrì)
MoE (Mixture-of-Experts) models don't activate all their
parameters for every token. GLM-5.2 activates ~5.4% per
token. Kimi K3 activates ~1.8%.
This means you don't need to fit the whole model in RAM.
You just need to stream the right experts fast enough.
Token arrives
↓
Model's own router selects which experts to activate
↓
Check VRAM first → instant if cached
↓
Then RAM → fast if cached
↓
Then NVMe SSD → stream if cold
↓
LRU cache promotes hot experts upward automatically
After 10-15 minutes in one domain, the cache self-organizes
to 85-92% hit rate. No configuration. Fully automatic.
What WISP Adds on Top
Colibrì proved the concept for one model (GLM-5.2) in pure C.
WISP extends it to every major MoE model with full CUDA
acceleration:
Absorbed MLA Attention
GLM-5.2 and DeepSeek use Multi-head Latent Attention.
WISP implements true absorbed MLA — storing compressed c_kv
instead of expanded K,V tensors.
Result: ~70KB per token KV cache instead of ~5MB.
This is what makes 1M token context feasible in RAM.
Double-Buffer Async Pipeline
While the GPU computes token N (2-8ms), the C engine loads
token N+1's predicted experts from SSD into pinned RAM
(0.1-0.3ms). Transfer is fully hidden inside compute time.
GPU never waits. No GPUDirect Storage needed.
Same-Family Speculative Decoding
Small models from the same family draft 3 tokens
simultaneously. The main model verifies all 3 in one
parallel forward pass. At 39-55% acceptance rate this gives
2.2-2.8x effective throughput with zero quality loss
(Leviathan et al. 2023).
Auto-Config Everything
WISP profiles your hardware once on first run and
calculates the optimal VRAM/RAM/SSD split automatically.
It even detects whether your monitor is on the GPU or
motherboard and reserves VRAM accordingly.
The Stack
Python → orchestration (download, convert, configure)
C → hot path (64-1504 expert fetches per token)
CUDA → math (attention, routing, FFN, speculation)
Three layers. One job each.
What's Supported
| Model | Parameters | Active/token | Status |
|---|---|---|---|
| GLM-5.2 | 744B | 40B | ✅ Ready |
| DeepSeek-V3 | 671B | 37B | ✅ Ready |
| DeepSeek-R1 | 671B | 37B | ✅ Ready |
| Mixtral-8x7B | 47B | 13B | ✅ Verified |
| Mixtral-8x22B | 141B | 39B | ✅ Ready |
| Kimi K3 | 2.8T | ~50B | ⏳ July 27 |
| Qwen3.8 | 2.4T | TBD | ⏳ Soon |
Real Benchmark — Mixtral-8x7B on RTX 5070
Verified on: R7 9800X3D | RTX 5070 12GB |
32GB DDR5-6000 | PCIe 4.0 NVMe (4.34 GB/s)
Cold tok/s: 0.75 tok/s (measured)
Cache hit rate: 68.8% after only 80 tokens
Expert fetches: 64 per token (2 experts × 32 layers)
All 256 experts: Fit entirely in RAM (14.3GB)
After warm-up: Zero SSD reads
Why is Mixtral slower than GLM-5.2 will be?
Mixtral experts = 99MB each.
GLM-5.2 experts = 17.5MB each.
5.7x smaller experts = 5.7x less data per token.
GLM-5.2 is where WISP's architecture truly sings.
Lessons From Building This
Lesson 1: The bottleneck is always the SSD
Every optimization that doesn't improve cache hit rate
or reduce bytes-per-token is noise. CUDA for expert matmul
gives near-zero benefit on a 9800X3D because the CPU is
already fast enough — the NVMe is the wall.
Lesson 2: The model's router is the best scheduler
We don't predict or preload experts manually. The model's
own router fires on every token and tells us exactly what
it needs. Our LRU cache learns from this automatically.
No heuristics. No preset domain modes. Pure adaptation.
Lesson 3: Windows GPU display is a real problem
Running inference on the same GPU that drives your monitor
will black screen your display. The GPU tries to do both
and the display loses. We ship display auto-detection that
reserves VRAM and prevents this automatically.
Learned this the hard way.
Lesson 4: Honest numbers matter more than impressive ones
The README says "We do not publish a bold number we didn't
measure." Every estimate is labeled as an estimate. Every
measured number has the exact conditions listed.
People trust projects that are honest about limitations.
Credits
WISP wouldn't exist without Colibrì by JustVugg.
He built the proof of concept. We built the generalization.
Full credit: github.com/JustVugg/colibri
Get Started
pip install wisp-engine
wisp convert --model glm-5.2 --output ./models/
wisp chat --model ./models/glm-5.2/
Requirements:
- 16GB+ RAM
- Any NVMe SSD with 300GB+ free
- CUDA GPU with 8GB+ VRAM (optional, CPU-only works)
What's Next
July 27 — Kimi K3 day
Weights drop. Technical report publishes. We implement
KDA attention and Quantile Balancing router. WISP streams
2.8 trillion parameters on consumer hardware.
v1.1
- Learning cache (gets faster with use)
- Persistent KV cache (resume conversations)
- Web dashboard with expert heatmap
- ROCm support (AMD R9700)
- OpenAI-compatible API server
73 tests passing. MIT license.
github.com/zeroextub-collab/wisp
Built on the shoulders of Colibrì.
JustVugg showed us what was possible.
Top comments (0)