DEV Community

Zero_planck
Zero_planck

Posted on

I Built a CUDA Engine That Streams 744B-Parameter AI Models on Consumer Hardware

I Built a CUDA Engine That Streams 744B-Parameter AI Models on Consumer Hardware

ai #cuda #opensource #machinelearning

WISP — Stream What Shouldn't Run

Last week, JustVugg released Colibrì — a ~2,400-line pure-C inference engine demonstrating something I couldn't stop thinking about:

A 744B-parameter MoE model doesn't necessarily need 744B parameters sitting in memory at once.

Colibrì showed how expert weights could be streamed from storage instead of forcing the entire model into RAM.

So I started building on that idea.

Meet WISP.

WISP is an experimental inference engine built around one principle:

Don't load the whole model. Load what the current token needs.


The Problem

Frontier-scale open-weight models are becoming enormous.

Models such as GLM-5.2, DeepSeek-V3, and emerging trillion-parameter MoE architectures can be far beyond the memory capacity of a normal workstation.

The usual local-inference solution is simple:

Make the model smaller until it fits.

Quantize it. Offload layers. Reduce context. Use a smaller model.

But MoE models give us another option.

They may contain hundreds of billions of total parameters while activating only a fraction of them for each token.

So instead of asking:

How do we fit the entire model into memory?

WISP asks:

What if we never load the entire model at all?


The Core Insight

Credit here goes to JustVugg and Colibrì.

Mixture-of-Experts models contain many expert networks, but the router activates only a subset for each token.

That changes the memory problem completely.

Instead of:

MODEL
↓
Load hundreds of billions of parameters
↓
Run inference
Enter fullscreen mode Exit fullscreen mode

we can do:

Token
  ↓
Router selects experts
  ↓
VRAM cache?
  ├── HIT → use immediately
  ↓ MISS
RAM cache?
  ├── HIT → transfer to GPU
  ↓ MISS
NVMe SSD
  ↓
Stream expert into pinned RAM
  ↓
Promote frequently used experts
Enter fullscreen mode Exit fullscreen mode

WISP uses a hierarchical cache:

VRAM → RAM → NVMe

with LRU-based promotion and eviction.

As workloads develop locality, frequently used experts migrate upward through the hierarchy automatically.

No manually configured "coding mode."

No "math expert preset."

The workload shapes the cache.


What WISP Adds

Colibrì demonstrated the core streaming idea.

WISP attempts to generalize that architecture across multiple MoE families while combining Python, C, and CUDA.

1. Absorbed MLA Attention

Models in the DeepSeek family use Multi-head Latent Attention (MLA).

Instead of storing fully expanded K/V tensors for every token, WISP implements an absorbed MLA path built around the compressed latent representation.

Conceptually:

Traditional KV cache

Token → K tensor
      → V tensor
      → store both


Absorbed MLA

Token → compressed latent state
      → store latent representation
      → reconstruct/absorb projections during attention
Enter fullscreen mode Exit fullscreen mode

This can dramatically reduce KV-cache memory requirements for architectures where MLA is supported.

And that matters because once model weights are streamed, context memory becomes the next wall.


2. Double-Buffered Async Streaming

SSD streaming sounds terrible for inference latency.

And it is — if the GPU waits for it.

So WISP overlaps I/O with compute.

TOKEN N

GPU
████████████████████
Compute token N


CPU + I/O
      ████████████
      Prepare/fetch next experts
Enter fullscreen mode Exit fullscreen mode

While CUDA is computing the current work, the C runtime can prepare upcoming expert data in pinned host memory.

The goal is simple:

Hide transfers behind computation whenever possible.

WISP doesn't require GPUDirect Storage for this design.

Instead, it uses asynchronous I/O, pinned memory, CUDA streams, and double buffering to reduce idle time.


3. Same-Family Speculative Decoding

Streaming solves memory.

It doesn't automatically solve generation speed.

So WISP also supports speculative decoding.

A smaller draft model proposes several tokens:

Draft model

→ token A
→ token B
→ token C
Enter fullscreen mode Exit fullscreen mode

The larger model then verifies the proposal in parallel.

Accepted tokens are committed immediately. Rejected tokens fall back to the target model.

When acceptance is high enough, one expensive target-model pass can advance generation by multiple tokens.

The important property:

the target model still determines the final output.

Speculation changes how inference is executed, not which model ultimately validates the tokens.


4. Hardware Auto-Configuration

I didn't want WISP to require this:

--gpu-cache 8.7GB
--ram-cache 21.3GB
--io-workers 6
--buffer-size 512MB
--please-dont-crash-my-display
Enter fullscreen mode Exit fullscreen mode

So the first-run profiler measures the machine and builds the configuration automatically.

It considers things like:

VRAM
RAM
storage throughput
available memory
GPU configuration
cache capacity
Enter fullscreen mode Exit fullscreen mode

The runtime can then determine how aggressively experts should be cached at each level.

The idea is that you should point WISP at a model and let the engine figure out the machine.


The Stack

WISP intentionally uses three layers.

┌─────────────────────────────┐
│           PYTHON            │
│ Download / Convert / Config │
├─────────────────────────────┤
│              C              │
│ Cache / I/O / Hot Runtime   │
├─────────────────────────────┤
│            CUDA             │
│ Attention / FFN / GPU Math  │
└─────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Python handles orchestration.

C handles the latency-sensitive runtime and expert streaming.

CUDA handles the math that belongs on the GPU.

One job per layer.


Model Targets

Model Total Parameters Active / Token Status
GLM-5.2 744B ~40B ✅ Ready
DeepSeek-V3 671B ~37B ✅ Ready
DeepSeek-R1 671B ~37B ✅ Ready
Mixtral-8x7B 47B ~13B ✅ Verified
Mixtral-8x22B 141B ~39B ✅ Ready
Kimi K3 2.8T ~50B ⏳ Planned
Qwen3.8 2.4T TBD ⏳ Planned

Real Benchmark: Mixtral-8x7B

Test machine:

CPU:     Ryzen 7 9800X3D
GPU:     RTX 5070 12GB
RAM:     32GB DDR5-6000
Storage: PCIe 4.0 NVMe
Measured sequential throughput: ~4.34 GB/s
Enter fullscreen mode Exit fullscreen mode

Measured cold throughput:

0.75 tok/s

After only 80 tokens:

68.8% expert-cache hit rate

Mixtral performs:

2 experts × 32 layers
= 64 expert activations/token
Enter fullscreen mode Exit fullscreen mode

On this machine, all 256 experts can fit in RAM after warm-up, occupying roughly 14.3GB in the tested representation.

At that point, the inference path can avoid cold SSD expert reads.

That's exactly the behavior WISP is designed around.


The Interesting Part: Bigger Can Behave Differently

Here's one of the counterintuitive things I learned while building this.

Model size alone doesn't determine streaming performance.

Expert size matters enormously.

In my tested setup:

Mixtral expert: ~99MB
GLM-class expert target: ~17.5MB

Difference: ~5.7×
Enter fullscreen mode Exit fullscreen mode

A much larger total model can therefore present a very different I/O profile if each routed expert is substantially smaller.

For a streaming architecture, the important number isn't only:

How large is the model?

It's also:

How many bytes must move for each token?


What Building WISP Taught Me

1. Storage Can Become the Real Bottleneck

I initially expected CUDA optimization to dominate performance work.

It didn't.

Once compute becomes sufficiently fast, cold expert loading can dominate latency.

That means an optimization that makes a matrix multiplication 20% faster may barely affect end-to-end generation if the runtime is still waiting on hundreds of megabytes of storage traffic.

For this architecture, I keep coming back to:

bytes moved / token
cache hit rate
I/O overlap
expert locality
Enter fullscreen mode Exit fullscreen mode

Those can matter more than another optimized kernel.


2. The Router Already Knows What the Model Needs

I experimented with thinking about expert prediction and domain-specific preloading.

But every MoE already contains something better:

its router.

Every token generates routing decisions.

Those decisions provide a continuous stream of information about which experts the model is actually using.

So WISP can let the model's own behavior drive cache adaptation rather than maintaining manually designed domain modes.


3. GPU Memory Isn't Just Yours

Consumer GPUs are often also driving the display.

Inference code that assumes every byte of reported VRAM belongs to the model can create a terrible desktop experience.

WISP therefore treats available VRAM as a dynamic resource and reserves headroom instead of blindly allocating everything it sees.

This sounds boring compared with CUDA kernels.

It also matters a lot when you're actually running the thing on your daily PC.


4. Benchmark Claims Need Context

Large-model inference produces very tempting numbers.

Estimated throughput.

Theoretical bandwidth.

Projected cache-hit rates.

Best-case model sizes.

So I made a rule for WISP:

Measured numbers are labeled measured. Estimates are labeled estimates.

Benchmarks should include the hardware, model, quantization/representation, cache state, context, and relevant runtime conditions.

A boring number you can reproduce is more useful than an insane number nobody can.


Credit Where It's Due

WISP started because JustVugg's Colibrì demonstrated that this style of inference was worth exploring.

Colibrì proved the core concept.

WISP explores how far that concept can be generalized with a multi-model runtime, hierarchical caching, CUDA acceleration, asynchronous streaming, MLA support, and speculative decoding.

Colibrì:
github.com/JustVugg/colibri

Built on the shoulders of open-source work.


Getting Started

pip install wisp-engine

wisp convert \
    --model glm-5.2 \
    --output ./models/

wisp chat \
    --model ./models/glm-5.2/
Enter fullscreen mode Exit fullscreen mode

Typical target hardware:

16GB+ system RAM
NVMe SSD with sufficient free storage
CUDA GPU with 8GB+ VRAM recommended
CPU-only execution supported
Enter fullscreen mode Exit fullscreen mode

You don't need a datacenter.

You need a memory hierarchy.


What's Next

The next targets are focused on pushing streaming inference further:

• Kimi K3 support
• KDA attention
• Quantile Balancing routing support
• Learning cache
• Persistent KV cache
• Conversation resume
• Expert heatmap dashboard
• ROCm support
• OpenAI-compatible API server
Enter fullscreen mode Exit fullscreen mode

The learning cache is especially interesting.

Instead of starting cold every session, WISP could preserve workload patterns and gradually optimize itself around how you actually use the model.

Today the hierarchy is:

VRAM
  ↓
RAM
  ↓
NVMe
Enter fullscreen mode Exit fullscreen mode

Eventually I want it to feel more like:

VRAM
  ↓
RAM
  ↓
NVMe
  ↓
Persistent workload intelligence
Enter fullscreen mode Exit fullscreen mode

73 tests passing. MIT licensed. Open source.

WISP:
github.com/zeroextub-collab/wisp

The goal isn't to pretend a consumer PC suddenly became an H100 cluster.

It's to ask a different question:

How much AI can we run when model size stops being limited by RAM?

Colibrì showed me that the answer might be much bigger than I thought.

So I built WISP to find out.

Top comments (0)