I Built a CUDA Engine That Streams 744B Parameter AI Models on Consumer Hardware

Zero_planck — Tue, 28 Jul 2026 16:39:05 +0000

WISP — Stream What Shouldn't Run

Last week JustVugg dropped Colibrì — a 2,400-line pure C
engine that proved a 744B parameter model could run on 25GB
of consumer RAM by streaming expert weights from disk.

It blew my mind. So I built on top of that concept.

This is WISP.

The Problem

The largest open source AI models in the world — GLM-5.2
(744B), DeepSeek-V3 (671B), Kimi K3 (2.8T) — require
datacenter hardware to run. Most people trying to run them
locally hit a wall immediately.

The standard approaches all make the same tradeoff:
quantize aggressively until the model fits, or don't run it.

Both options sacrifice intelligence for accessibility.

The Insight (Credit: JustVugg / Colibrì)

MoE (Mixture-of-Experts) models don't activate all their
parameters for every token. GLM-5.2 activates ~5.4% per
token. Kimi K3 activates ~1.8%.

This means you don't need to fit the whole model in RAM.
You just need to stream the right experts fast enough.

Token arrives
↓
Model's own router selects which experts to activate
↓
Check VRAM first → instant if cached
↓
Then RAM → fast if cached
↓
Then NVMe SSD → stream if cold
↓
LRU cache promotes hot experts upward automatically

After 10-15 minutes in one domain, the cache self-organizes
to 85-92% hit rate. No configuration. Fully automatic.

What WISP Adds on Top

Colibrì proved the concept for one model (GLM-5.2) in pure C.

WISP extends it to every major MoE model with full CUDA
acceleration:

Absorbed MLA Attention

GLM-5.2 and DeepSeek use Multi-head Latent Attention.
WISP implements true absorbed MLA — storing compressed c_kv
instead of expanded K,V tensors.

Result: ~70KB per token KV cache instead of ~5MB.
This is what makes 1M token context feasible in RAM.

Double-Buffer Async Pipeline

While the GPU computes token N (2-8ms), the C engine loads
token N+1's predicted experts from SSD into pinned RAM
(0.1-0.3ms). Transfer is fully hidden inside compute time.
GPU never waits. No GPUDirect Storage needed.

Same-Family Speculative Decoding

Small models from the same family draft 3 tokens
simultaneously. The main model verifies all 3 in one
parallel forward pass. At 39-55% acceptance rate this gives
2.2-2.8x effective throughput with zero quality loss
(Leviathan et al. 2023).

Auto-Config Everything

WISP profiles your hardware once on first run and
calculates the optimal VRAM/RAM/SSD split automatically.
It even detects whether your monitor is on the GPU or
motherboard and reserves VRAM accordingly.

The Stack

Python → orchestration (download, convert, configure)
C → hot path (64-1504 expert fetches per token)
CUDA → math (attention, routing, FFN, speculation)

Three layers. One job each.

What's Supported

Model	Parameters	Active/token	Status
GLM-5.2	744B	40B	✅ Ready
DeepSeek-V3	671B	37B	✅ Ready
DeepSeek-R1	671B	37B	✅ Ready
Mixtral-8x7B	47B	13B	✅ Verified
Mixtral-8x22B	141B	39B	✅ Ready
Kimi K3	2.8T	~50B	⏳ July 27
Qwen3.8	2.4T	TBD	⏳ Soon

Real Benchmark — Mixtral-8x7B on RTX 5070

Verified on: R7 9800X3D | RTX 5070 12GB |
32GB DDR5-6000 | PCIe 4.0 NVMe (4.34 GB/s)

Cold tok/s: 0.75 tok/s (measured)
Cache hit rate: 68.8% after only 80 tokens
Expert fetches: 64 per token (2 experts × 32 layers)
All 256 experts: Fit entirely in RAM (14.3GB)
After warm-up: Zero SSD reads

Why is Mixtral slower than GLM-5.2 will be?

Mixtral experts = 99MB each.
GLM-5.2 experts = 17.5MB each.

5.7x smaller experts = 5.7x less data per token.
GLM-5.2 is where WISP's architecture truly sings.

Lessons From Building This

Lesson 1: The bottleneck is always the SSD

Every optimization that doesn't improve cache hit rate
or reduce bytes-per-token is noise. CUDA for expert matmul
gives near-zero benefit on a 9800X3D because the CPU is
already fast enough — the NVMe is the wall.

Lesson 2: The model's router is the best scheduler

We don't predict or preload experts manually. The model's
own router fires on every token and tells us exactly what
it needs. Our LRU cache learns from this automatically.
No heuristics. No preset domain modes. Pure adaptation.

Lesson 3: Windows GPU display is a real problem

Running inference on the same GPU that drives your monitor
will black screen your display. The GPU tries to do both
and the display loses. We ship display auto-detection that
reserves VRAM and prevents this automatically.
Learned this the hard way.

Lesson 4: Honest numbers matter more than impressive ones

The README says "We do not publish a bold number we didn't
measure." Every estimate is labeled as an estimate. Every
measured number has the exact conditions listed.
People trust projects that are honest about limitations.

Credits

WISP wouldn't exist without Colibrì by JustVugg.
He built the proof of concept. We built the generalization.

Full credit: github.com/JustVugg/colibri

Get Started

pip install wisp-engine
wisp convert --model glm-5.2 --output ./models/
wisp chat --model ./models/glm-5.2/

Requirements:

16GB+ RAM
Any NVMe SSD with 300GB+ free
CUDA GPU with 8GB+ VRAM (optional, CPU-only works)

What's Next

July 27 — Kimi K3 day
Weights drop. Technical report publishes. We implement
KDA attention and Quantile Balancing router. WISP streams
2.8 trillion parameters on consumer hardware.

v1.1

Learning cache (gets faster with use)
Persistent KV cache (resume conversations)
Web dashboard with expert heatmap
ROCm support (AMD R9700)
OpenAI-compatible API server

73 tests passing. MIT license.

github.com/zeroextub-collab/wisp

Built on the shoulders of Colibrì.
JustVugg showed us what was possible.

DEV Community: Zero_planck

Please support guys !!!!!!!!