DEV Community

AI OpenFree
AI OpenFree

Posted on

How a Sparse 35B MoE Runs on a CPU: Active Params, Memory Bandwidth, and a Reproducible Benchmark

#machinelearning #ai #llm #opensource

How a Sparse 35B MoE Runs on a CPU: Active Params, Memory Bandwidth, and a Reproducible Benchmark

Most "runs on your laptop" LLM claims quietly mean a 7B model. This one is a 34.7B reasoning model — and it also runs on a GPU-less CPU. The interesting part isn't the marketing; it's why the numbers are possible, and that you can reproduce them.

I looked at VIDRAFT's Ourbox-35B-JGOS (served by their proprietary VKUE engine) using only the public model, demos, and leaderboard.

The one-file benchmark

Same GGUF weights, measured across a four-orders-of-magnitude hardware range:

Hardware tok/s
B200 (data center) 18,057
A10G ×1 (cloud) 126
8GB gaming laptop 20
CPU, no GPU ~17

Why it works: memory bandwidth, not FLOPs

Autoregressive decoding is memory-bandwidth bound — the bottleneck is bytes read per token, not compute. For a dense model you stream every weight per token; for a sparse Mixture-of-Experts (MoE) you stream only the active experts.

Ourbox-35B: 34.7B total, ~3B active (think "8 of 256 experts routed per token").

Back-of-envelope bytes per token, same quantization:

  • Dense 34B: ~16.7 GB/token
  • Sparse (~3B active): ~1.45 GB/token → ~11× less memory traffic

That ratio is the whole story: park the cold experts in cheap RAM, keep the hot path on whatever accelerator you have (or none), and an 8GB card — or a CPU — is enough.

Isolating the variable: A/B vs a dense 32B

Speed comparisons usually confound size, quantization, and hardware. VIDRAFT published a controlled A/B — same laptop, same quantization, near-identical file size — so the only variable is active-parameter count:

Model Active params tok/s
Ourbox-35B (MoE) ~3B 20.01
Qwen2.5-32B (dense) 32.8B 5.36

3.7× from sparsity alone.

Fast ≠ dumb

The obvious objection is quality. Public score: GPQA Diamond 86.4% (maj@8) — graduate-level science reasoning, with the method label included so you can compare apples to apples.

Reproduce it yourself

This is the part that matters for engineers: you don't have to trust the table. The model (GGUF), two live demos, and the efficiency leaderboard are public. Run the same prompt on GPU and CPU in the demo and watch the live tok/s. GGUF plus a standard 3-bit quant (Q3_K_M) is the worst-case config; a full 24GB card pushes the same weights to 87–196 tok/s.

Honest limits (stated, not hidden)

  • Numbers are per-machine measurements, not guarantees; long contexts slow everything.
  • CPU at ~17 tok/s proves "it runs," not "it's fast." This is not "CPU beats GPU."
  • The laptop/CPU figures are the 3-bit worst case.
  • The whole trick is contingent on MoE sparsity — it does not transfer to a dense 34B.
  • The VKUE serving engine itself is closed; what's open and verifiable is the model, the demos, and the numbers.

VKAE vs VKUE

VIDRAFT ships two serving profiles: VKAE (maximum throughput on data-center GPUs — the B200 18,057 tok/s) and VKUE (minimum-hardware reach — laptop/CPU). Different objectives: throughput vs access.

Links

Top comments (0)