I Built a CUDA Engine That Streams 744B-Parameter AI Models on Consumer Hardware
ai #cuda #opensource #machinelearning
WISP — Stream What Shouldn't Run
Last week, JustVugg released Colibrì — a ~2,400-line pure-C inference engine demonstrating something I couldn't stop thinking about:
A 744B-parameter MoE model doesn't necessarily need 744B parameters sitting in memory at once.
Colibrì showed how expert weights could be streamed from storage instead of forcing the entire model into RAM.
So I started building on that idea.
Meet WISP.
WISP is an experimental inference engine built around one principle:
Don't load the whole model. Load what the current token needs.
The Problem
Frontier-scale open-weight models are becoming enormous.
Models such as GLM-5.2, DeepSeek-V3, and emerging trillion-parameter MoE architectures can be far beyond the memory capacity of a normal workstation.
The usual local-inference solution is simple:
Make the model smaller until it fits.
Quantize it. Offload layers. Reduce context. Use a smaller model.
But MoE models give us another option.
They may contain hundreds of billions of total parameters while activating only a fraction of them for each token.
So instead of asking:
How do we fit the entire model into memory?
WISP asks:
What if we never load the entire model at all?
The Core Insight
Credit here goes to JustVugg and Colibrì.
Mixture-of-Experts models contain many expert networks, but the router activates only a subset for each token.
That changes the memory problem completely.
Instead of:
MODEL
↓
Load hundreds of billions of parameters
↓
Run inference
we can do:
Token
↓
Router selects experts
↓
VRAM cache?
├── HIT → use immediately
↓ MISS
RAM cache?
├── HIT → transfer to GPU
↓ MISS
NVMe SSD
↓
Stream expert into pinned RAM
↓
Promote frequently used experts
WISP uses a hierarchical cache:
VRAM → RAM → NVMe
with LRU-based promotion and eviction.
As workloads develop locality, frequently used experts migrate upward through the hierarchy automatically.
No manually configured "coding mode."
No "math expert preset."
The workload shapes the cache.
What WISP Adds
Colibrì demonstrated the core streaming idea.
WISP attempts to generalize that architecture across multiple MoE families while combining Python, C, and CUDA.
1. Absorbed MLA Attention
Models in the DeepSeek family use Multi-head Latent Attention (MLA).
Instead of storing fully expanded K/V tensors for every token, WISP implements an absorbed MLA path built around the compressed latent representation.
Conceptually:
Traditional KV cache
Token → K tensor
→ V tensor
→ store both
Absorbed MLA
Token → compressed latent state
→ store latent representation
→ reconstruct/absorb projections during attention
This can dramatically reduce KV-cache memory requirements for architectures where MLA is supported.
And that matters because once model weights are streamed, context memory becomes the next wall.
2. Double-Buffered Async Streaming
SSD streaming sounds terrible for inference latency.
And it is — if the GPU waits for it.
So WISP overlaps I/O with compute.
TOKEN N
GPU
████████████████████
Compute token N
CPU + I/O
████████████
Prepare/fetch next experts
While CUDA is computing the current work, the C runtime can prepare upcoming expert data in pinned host memory.
The goal is simple:
Hide transfers behind computation whenever possible.
WISP doesn't require GPUDirect Storage for this design.
Instead, it uses asynchronous I/O, pinned memory, CUDA streams, and double buffering to reduce idle time.
3. Same-Family Speculative Decoding
Streaming solves memory.
It doesn't automatically solve generation speed.
So WISP also supports speculative decoding.
A smaller draft model proposes several tokens:
Draft model
→ token A
→ token B
→ token C
The larger model then verifies the proposal in parallel.
Accepted tokens are committed immediately. Rejected tokens fall back to the target model.
When acceptance is high enough, one expensive target-model pass can advance generation by multiple tokens.
The important property:
the target model still determines the final output.
Speculation changes how inference is executed, not which model ultimately validates the tokens.
4. Hardware Auto-Configuration
I didn't want WISP to require this:
--gpu-cache 8.7GB
--ram-cache 21.3GB
--io-workers 6
--buffer-size 512MB
--please-dont-crash-my-display
So the first-run profiler measures the machine and builds the configuration automatically.
It considers things like:
VRAM
RAM
storage throughput
available memory
GPU configuration
cache capacity
The runtime can then determine how aggressively experts should be cached at each level.
The idea is that you should point WISP at a model and let the engine figure out the machine.
The Stack
WISP intentionally uses three layers.
┌─────────────────────────────┐
│ PYTHON │
│ Download / Convert / Config │
├─────────────────────────────┤
│ C │
│ Cache / I/O / Hot Runtime │
├─────────────────────────────┤
│ CUDA │
│ Attention / FFN / GPU Math │
└─────────────────────────────┘
Python handles orchestration.
C handles the latency-sensitive runtime and expert streaming.
CUDA handles the math that belongs on the GPU.
One job per layer.
Model Targets
| Model | Total Parameters | Active / Token | Status |
|---|---|---|---|
| GLM-5.2 | 744B | ~40B | ✅ Ready |
| DeepSeek-V3 | 671B | ~37B | ✅ Ready |
| DeepSeek-R1 | 671B | ~37B | ✅ Ready |
| Mixtral-8x7B | 47B | ~13B | ✅ Verified |
| Mixtral-8x22B | 141B | ~39B | ✅ Ready |
| Kimi K3 | 2.8T | ~50B | ⏳ Planned |
| Qwen3.8 | 2.4T | TBD | ⏳ Planned |
Real Benchmark: Mixtral-8x7B
Test machine:
CPU: Ryzen 7 9800X3D
GPU: RTX 5070 12GB
RAM: 32GB DDR5-6000
Storage: PCIe 4.0 NVMe
Measured sequential throughput: ~4.34 GB/s
Measured cold throughput:
0.75 tok/s
After only 80 tokens:
68.8% expert-cache hit rate
Mixtral performs:
2 experts × 32 layers
= 64 expert activations/token
On this machine, all 256 experts can fit in RAM after warm-up, occupying roughly 14.3GB in the tested representation.
At that point, the inference path can avoid cold SSD expert reads.
That's exactly the behavior WISP is designed around.
The Interesting Part: Bigger Can Behave Differently
Here's one of the counterintuitive things I learned while building this.
Model size alone doesn't determine streaming performance.
Expert size matters enormously.
In my tested setup:
Mixtral expert: ~99MB
GLM-class expert target: ~17.5MB
Difference: ~5.7×
A much larger total model can therefore present a very different I/O profile if each routed expert is substantially smaller.
For a streaming architecture, the important number isn't only:
How large is the model?
It's also:
How many bytes must move for each token?
What Building WISP Taught Me
1. Storage Can Become the Real Bottleneck
I initially expected CUDA optimization to dominate performance work.
It didn't.
Once compute becomes sufficiently fast, cold expert loading can dominate latency.
That means an optimization that makes a matrix multiplication 20% faster may barely affect end-to-end generation if the runtime is still waiting on hundreds of megabytes of storage traffic.
For this architecture, I keep coming back to:
bytes moved / token
cache hit rate
I/O overlap
expert locality
Those can matter more than another optimized kernel.
2. The Router Already Knows What the Model Needs
I experimented with thinking about expert prediction and domain-specific preloading.
But every MoE already contains something better:
its router.
Every token generates routing decisions.
Those decisions provide a continuous stream of information about which experts the model is actually using.
So WISP can let the model's own behavior drive cache adaptation rather than maintaining manually designed domain modes.
3. GPU Memory Isn't Just Yours
Consumer GPUs are often also driving the display.
Inference code that assumes every byte of reported VRAM belongs to the model can create a terrible desktop experience.
WISP therefore treats available VRAM as a dynamic resource and reserves headroom instead of blindly allocating everything it sees.
This sounds boring compared with CUDA kernels.
It also matters a lot when you're actually running the thing on your daily PC.
4. Benchmark Claims Need Context
Large-model inference produces very tempting numbers.
Estimated throughput.
Theoretical bandwidth.
Projected cache-hit rates.
Best-case model sizes.
So I made a rule for WISP:
Measured numbers are labeled measured. Estimates are labeled estimates.
Benchmarks should include the hardware, model, quantization/representation, cache state, context, and relevant runtime conditions.
A boring number you can reproduce is more useful than an insane number nobody can.
Credit Where It's Due
WISP started because JustVugg's Colibrì demonstrated that this style of inference was worth exploring.
Colibrì proved the core concept.
WISP explores how far that concept can be generalized with a multi-model runtime, hierarchical caching, CUDA acceleration, asynchronous streaming, MLA support, and speculative decoding.
Colibrì:
github.com/JustVugg/colibri
Built on the shoulders of open-source work.
Getting Started
pip install wisp-engine
wisp convert \
--model glm-5.2 \
--output ./models/
wisp chat \
--model ./models/glm-5.2/
Typical target hardware:
16GB+ system RAM
NVMe SSD with sufficient free storage
CUDA GPU with 8GB+ VRAM recommended
CPU-only execution supported
You don't need a datacenter.
You need a memory hierarchy.
What's Next
The next targets are focused on pushing streaming inference further:
• Kimi K3 support
• KDA attention
• Quantile Balancing routing support
• Learning cache
• Persistent KV cache
• Conversation resume
• Expert heatmap dashboard
• ROCm support
• OpenAI-compatible API server
The learning cache is especially interesting.
Instead of starting cold every session, WISP could preserve workload patterns and gradually optimize itself around how you actually use the model.
Today the hierarchy is:
VRAM
↓
RAM
↓
NVMe
Eventually I want it to feel more like:
VRAM
↓
RAM
↓
NVMe
↓
Persistent workload intelligence
73 tests passing. MIT licensed. Open source.
WISP:
github.com/zeroextub-collab/wisp
The goal isn't to pretend a consumer PC suddenly became an H100 cluster.
It's to ask a different question:
How much AI can we run when model size stops being limited by RAM?
Colibrì showed me that the answer might be much bigger than I thought.
So I built WISP to find out.
Top comments (0)