DEV Community: Zero_planck

I Built a CUDA Engine That Streams 744B-Parameter AI Models on Consumer Hardware

Zero_planck — Thu, 30 Jul 2026 14:30:16 +0000

I Built a CUDA Engine That Streams 744B-Parameter AI Models on Consumer Hardware

ai #cuda #opensource #machinelearning

WISP — Stream What Shouldn't Run

Last week, JustVugg released Colibrì — a ~2,400-line pure-C inference engine demonstrating something I couldn't stop thinking about:

A 744B-parameter MoE model doesn't necessarily need 744B parameters sitting in memory at once.

Colibrì showed how expert weights could be streamed from storage instead of forcing the entire model into RAM.

So I started building on that idea.

Meet WISP.

WISP is an experimental inference engine built around one principle:

Don't load the whole model. Load what the current token needs.

The Problem

Frontier-scale open-weight models are becoming enormous.

Models such as GLM-5.2, DeepSeek-V3, and emerging trillion-parameter MoE architectures can be far beyond the memory capacity of a normal workstation.

The usual local-inference solution is simple:

Make the model smaller until it fits.

Quantize it. Offload layers. Reduce context. Use a smaller model.

But MoE models give us another option.

They may contain hundreds of billions of total parameters while activating only a fraction of them for each token.

So instead of asking:

How do we fit the entire model into memory?

WISP asks:

What if we never load the entire model at all?

The Core Insight

Credit here goes to JustVugg and Colibrì.

Mixture-of-Experts models contain many expert networks, but the router activates only a subset for each token.

That changes the memory problem completely.

Instead of:

MODEL
↓
Load hundreds of billions of parameters
↓
Run inference

we can do:

Token
  ↓
Router selects experts
  ↓
VRAM cache?
  ├── HIT → use immediately
  ↓ MISS
RAM cache?
  ├── HIT → transfer to GPU
  ↓ MISS
NVMe SSD
  ↓
Stream expert into pinned RAM
  ↓
Promote frequently used experts

WISP uses a hierarchical cache:

VRAM → RAM → NVMe

with LRU-based promotion and eviction.

As workloads develop locality, frequently used experts migrate upward through the hierarchy automatically.

No manually configured "coding mode."

No "math expert preset."

The workload shapes the cache.

What WISP Adds

Colibrì demonstrated the core streaming idea.

WISP attempts to generalize that architecture across multiple MoE families while combining Python, C, and CUDA.

1. Absorbed MLA Attention

Models in the DeepSeek family use Multi-head Latent Attention (MLA).

Instead of storing fully expanded K/V tensors for every token, WISP implements an absorbed MLA path built around the compressed latent representation.

Conceptually:

Traditional KV cache

Token → K tensor
      → V tensor
      → store both


Absorbed MLA

Token → compressed latent state
      → store latent representation
      → reconstruct/absorb projections during attention

This can dramatically reduce KV-cache memory requirements for architectures where MLA is supported.

And that matters because once model weights are streamed, context memory becomes the next wall.

2. Double-Buffered Async Streaming

SSD streaming sounds terrible for inference latency.

And it is — if the GPU waits for it.

So WISP overlaps I/O with compute.

TOKEN N

GPU
████████████████████
Compute token N


CPU + I/O
      ████████████
      Prepare/fetch next experts

While CUDA is computing the current work, the C runtime can prepare upcoming expert data in pinned host memory.

The goal is simple:

Hide transfers behind computation whenever possible.

WISP doesn't require GPUDirect Storage for this design.

Instead, it uses asynchronous I/O, pinned memory, CUDA streams, and double buffering to reduce idle time.

3. Same-Family Speculative Decoding

Streaming solves memory.

It doesn't automatically solve generation speed.

So WISP also supports speculative decoding.

A smaller draft model proposes several tokens:

Draft model

→ token A
→ token B
→ token C

The larger model then verifies the proposal in parallel.

Accepted tokens are committed immediately. Rejected tokens fall back to the target model.

When acceptance is high enough, one expensive target-model pass can advance generation by multiple tokens.

The important property:

the target model still determines the final output.

Speculation changes how inference is executed, not which model ultimately validates the tokens.

4. Hardware Auto-Configuration

I didn't want WISP to require this:

--gpu-cache 8.7GB
--ram-cache 21.3GB
--io-workers 6
--buffer-size 512MB
--please-dont-crash-my-display

So the first-run profiler measures the machine and builds the configuration automatically.

It considers things like:

VRAM
RAM
storage throughput
available memory
GPU configuration
cache capacity

The runtime can then determine how aggressively experts should be cached at each level.

The idea is that you should point WISP at a model and let the engine figure out the machine.

The Stack

WISP intentionally uses three layers.

┌─────────────────────────────┐
│           PYTHON            │
│ Download / Convert / Config │
├─────────────────────────────┤
│              C              │
│ Cache / I/O / Hot Runtime   │
├─────────────────────────────┤
│            CUDA             │
│ Attention / FFN / GPU Math  │
└─────────────────────────────┘

Python handles orchestration.

C handles the latency-sensitive runtime and expert streaming.

CUDA handles the math that belongs on the GPU.

One job per layer.

Model Targets

Model	Total Parameters	Active / Token	Status
GLM-5.2	744B	~40B	✅ Ready
DeepSeek-V3	671B	~37B	✅ Ready
DeepSeek-R1	671B	~37B	✅ Ready
Mixtral-8x7B	47B	~13B	✅ Verified
Mixtral-8x22B	141B	~39B	✅ Ready
Kimi K3	2.8T	~50B	⏳ Planned
Qwen3.8	2.4T	TBD	⏳ Planned

Real Benchmark: Mixtral-8x7B

Test machine:

CPU:     Ryzen 7 9800X3D
GPU:     RTX 5070 12GB
RAM:     32GB DDR5-6000
Storage: PCIe 4.0 NVMe
Measured sequential throughput: ~4.34 GB/s

Measured cold throughput:

0.75 tok/s

After only 80 tokens:

68.8% expert-cache hit rate

Mixtral performs:

2 experts × 32 layers
= 64 expert activations/token

On this machine, all 256 experts can fit in RAM after warm-up, occupying roughly 14.3GB in the tested representation.

At that point, the inference path can avoid cold SSD expert reads.

That's exactly the behavior WISP is designed around.

The Interesting Part: Bigger Can Behave Differently

Here's one of the counterintuitive things I learned while building this.

Model size alone doesn't determine streaming performance.

Expert size matters enormously.

In my tested setup:

Mixtral expert: ~99MB
GLM-class expert target: ~17.5MB

Difference: ~5.7×

A much larger total model can therefore present a very different I/O profile if each routed expert is substantially smaller.

For a streaming architecture, the important number isn't only:

How large is the model?

It's also:

How many bytes must move for each token?

What Building WISP Taught Me

1. Storage Can Become the Real Bottleneck

I initially expected CUDA optimization to dominate performance work.

It didn't.

Once compute becomes sufficiently fast, cold expert loading can dominate latency.

That means an optimization that makes a matrix multiplication 20% faster may barely affect end-to-end generation if the runtime is still waiting on hundreds of megabytes of storage traffic.

For this architecture, I keep coming back to:

bytes moved / token
cache hit rate
I/O overlap
expert locality

Those can matter more than another optimized kernel.

2. The Router Already Knows What the Model Needs

I experimented with thinking about expert prediction and domain-specific preloading.

But every MoE already contains something better:

its router.

Every token generates routing decisions.

Those decisions provide a continuous stream of information about which experts the model is actually using.

So WISP can let the model's own behavior drive cache adaptation rather than maintaining manually designed domain modes.

3. GPU Memory Isn't Just Yours

Consumer GPUs are often also driving the display.

Inference code that assumes every byte of reported VRAM belongs to the model can create a terrible desktop experience.

WISP therefore treats available VRAM as a dynamic resource and reserves headroom instead of blindly allocating everything it sees.

This sounds boring compared with CUDA kernels.

It also matters a lot when you're actually running the thing on your daily PC.

4. Benchmark Claims Need Context

Large-model inference produces very tempting numbers.

Estimated throughput.

Theoretical bandwidth.

Projected cache-hit rates.

Best-case model sizes.

So I made a rule for WISP:

Measured numbers are labeled measured. Estimates are labeled estimates.

Benchmarks should include the hardware, model, quantization/representation, cache state, context, and relevant runtime conditions.

A boring number you can reproduce is more useful than an insane number nobody can.

Credit Where It's Due

WISP started because JustVugg's Colibrì demonstrated that this style of inference was worth exploring.

Colibrì proved the core concept.

WISP explores how far that concept can be generalized with a multi-model runtime, hierarchical caching, CUDA acceleration, asynchronous streaming, MLA support, and speculative decoding.

Colibrì:
github.com/JustVugg/colibri

Built on the shoulders of open-source work.

Getting Started

pip install wisp-engine

wisp convert \
    --model glm-5.2 \
    --output ./models/

wisp chat \
    --model ./models/glm-5.2/

Typical target hardware:

16GB+ system RAM
NVMe SSD with sufficient free storage
CUDA GPU with 8GB+ VRAM recommended
CPU-only execution supported

You don't need a datacenter.

You need a memory hierarchy.

What's Next

The next targets are focused on pushing streaming inference further:

• Kimi K3 support
• KDA attention
• Quantile Balancing routing support
• Learning cache
• Persistent KV cache
• Conversation resume
• Expert heatmap dashboard
• ROCm support
• OpenAI-compatible API server

The learning cache is especially interesting.

Instead of starting cold every session, WISP could preserve workload patterns and gradually optimize itself around how you actually use the model.

Today the hierarchy is:

VRAM
  ↓
RAM
  ↓
NVMe

Eventually I want it to feel more like:

VRAM
  ↓
RAM
  ↓
NVMe
  ↓
Persistent workload intelligence

73 tests passing. MIT licensed. Open source.

WISP:
github.com/zeroextub-collab/wisp

The goal isn't to pretend a consumer PC suddenly became an H100 cluster.

It's to ask a different question:

How much AI can we run when model size stops being limited by RAM?

Colibrì showed me that the answer might be much bigger than I thought.

So I built WISP to find out.

Please support guys !!!!!!!!

Zero_planck — Tue, 28 Jul 2026 16:40:03 +0000

Zero_planck

Jul 28

I Built a CUDA Engine That Streams 744B Parameter AI Models on Consumer Hardware

#ai #cuda #opensource #machinelearning

4 min read

I Built a CUDA Engine That Streams 744B Parameter AI Models on Consumer Hardware

Zero_planck — Tue, 28 Jul 2026 16:39:05 +0000

WISP — Stream What Shouldn't Run

Last week JustVugg dropped Colibrì — a 2,400-line pure C
engine that proved a 744B parameter model could run on 25GB
of consumer RAM by streaming expert weights from disk.

It blew my mind. So I built on top of that concept.

This is WISP.

The Problem

The largest open source AI models in the world — GLM-5.2
(744B), DeepSeek-V3 (671B), Kimi K3 (2.8T) — require
datacenter hardware to run. Most people trying to run them
locally hit a wall immediately.

The standard approaches all make the same tradeoff:
quantize aggressively until the model fits, or don't run it.

Both options sacrifice intelligence for accessibility.

The Insight (Credit: JustVugg / Colibrì)

MoE (Mixture-of-Experts) models don't activate all their
parameters for every token. GLM-5.2 activates ~5.4% per
token. Kimi K3 activates ~1.8%.

This means you don't need to fit the whole model in RAM.
You just need to stream the right experts fast enough.

Token arrives
↓
Model's own router selects which experts to activate
↓
Check VRAM first → instant if cached
↓
Then RAM → fast if cached
↓
Then NVMe SSD → stream if cold
↓
LRU cache promotes hot experts upward automatically

After 10-15 minutes in one domain, the cache self-organizes
to 85-92% hit rate. No configuration. Fully automatic.

What WISP Adds on Top

Colibrì proved the concept for one model (GLM-5.2) in pure C.

WISP extends it to every major MoE model with full CUDA
acceleration:

Absorbed MLA Attention

GLM-5.2 and DeepSeek use Multi-head Latent Attention.
WISP implements true absorbed MLA — storing compressed c_kv
instead of expanded K,V tensors.

Result: ~70KB per token KV cache instead of ~5MB.
This is what makes 1M token context feasible in RAM.

Double-Buffer Async Pipeline

While the GPU computes token N (2-8ms), the C engine loads
token N+1's predicted experts from SSD into pinned RAM
(0.1-0.3ms). Transfer is fully hidden inside compute time.
GPU never waits. No GPUDirect Storage needed.

Same-Family Speculative Decoding

Small models from the same family draft 3 tokens
simultaneously. The main model verifies all 3 in one
parallel forward pass. At 39-55% acceptance rate this gives
2.2-2.8x effective throughput with zero quality loss
(Leviathan et al. 2023).

Auto-Config Everything

WISP profiles your hardware once on first run and
calculates the optimal VRAM/RAM/SSD split automatically.
It even detects whether your monitor is on the GPU or
motherboard and reserves VRAM accordingly.

The Stack

Python → orchestration (download, convert, configure)
C → hot path (64-1504 expert fetches per token)
CUDA → math (attention, routing, FFN, speculation)

Three layers. One job each.

What's Supported

Model	Parameters	Active/token	Status
GLM-5.2	744B	40B	✅ Ready
DeepSeek-V3	671B	37B	✅ Ready
DeepSeek-R1	671B	37B	✅ Ready
Mixtral-8x7B	47B	13B	✅ Verified
Mixtral-8x22B	141B	39B	✅ Ready
Kimi K3	2.8T	~50B	⏳ July 27
Qwen3.8	2.4T	TBD	⏳ Soon

Real Benchmark — Mixtral-8x7B on RTX 5070

Verified on: R7 9800X3D | RTX 5070 12GB |
32GB DDR5-6000 | PCIe 4.0 NVMe (4.34 GB/s)

Cold tok/s: 0.75 tok/s (measured)
Cache hit rate: 68.8% after only 80 tokens
Expert fetches: 64 per token (2 experts × 32 layers)
All 256 experts: Fit entirely in RAM (14.3GB)
After warm-up: Zero SSD reads

Why is Mixtral slower than GLM-5.2 will be?

Mixtral experts = 99MB each.
GLM-5.2 experts = 17.5MB each.

5.7x smaller experts = 5.7x less data per token.
GLM-5.2 is where WISP's architecture truly sings.

Lessons From Building This

Lesson 1: The bottleneck is always the SSD

Every optimization that doesn't improve cache hit rate
or reduce bytes-per-token is noise. CUDA for expert matmul
gives near-zero benefit on a 9800X3D because the CPU is
already fast enough — the NVMe is the wall.

Lesson 2: The model's router is the best scheduler

We don't predict or preload experts manually. The model's
own router fires on every token and tells us exactly what
it needs. Our LRU cache learns from this automatically.
No heuristics. No preset domain modes. Pure adaptation.

Lesson 3: Windows GPU display is a real problem

Running inference on the same GPU that drives your monitor
will black screen your display. The GPU tries to do both
and the display loses. We ship display auto-detection that
reserves VRAM and prevents this automatically.
Learned this the hard way.

Lesson 4: Honest numbers matter more than impressive ones

The README says "We do not publish a bold number we didn't
measure." Every estimate is labeled as an estimate. Every
measured number has the exact conditions listed.
People trust projects that are honest about limitations.

Credits

WISP wouldn't exist without Colibrì by JustVugg.
He built the proof of concept. We built the generalization.

Full credit: github.com/JustVugg/colibri

Get Started

pip install wisp-engine
wisp convert --model glm-5.2 --output ./models/
wisp chat --model ./models/glm-5.2/

Requirements:

16GB+ RAM
Any NVMe SSD with 300GB+ free
CUDA GPU with 8GB+ VRAM (optional, CPU-only works)

What's Next

July 27 — Kimi K3 day
Weights drop. Technical report publishes. We implement
KDA attention and Quantile Balancing router. WISP streams
2.8 trillion parameters on consumer hardware.

v1.1

Learning cache (gets faster with use)
Persistent KV cache (resume conversations)
Web dashboard with expert heatmap
ROCm support (AMD R9700)
OpenAI-compatible API server

73 tests passing. MIT license.

github.com/zeroextub-collab/wisp

Built on the shoulders of Colibrì.
JustVugg showed us what was possible.