SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.

Roy Mu — Tue, 13 Oct 2026 18:16:49 +0000

I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs.

Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes using the same GGUF on the same machine, llama.cpp at commit 43fe9c64281):

       SonicBoom                    llama.cpp

Decode (tg64) 51.65 tok/s 45.20 tok/s
Decode @ depth 2048 47.13 tok/s 45.21 tok/s
Prefill 4100 tokens 1051 tok/s ~1944 tok/s
TTFT, 4000-token prefix + 100 new 503 ms (prefix reuse)

Why it matters: the quantized matmul kernels (MMVQ/MMQ) are ported from ggml, so the kernels are the same ones llama.cpp uses. The 14% decode lead comes entirely from the layer above them. A graph planner rewrites the model before it runs: activations are quantized once and reused, norm and activation are fused with quantization, dead computation is removed, and each decode step is captured and replayed as a single CUDA Graph. Code derived from ggml is about 8% of the core. The rest is the runtime.

Prefix reuse is exact: a continuation is token-for-token identical to a full prefill, which makes multi-turn chat and coding-assistant workloads cheap.

What it can’t do yet:

Prefill is about half of llama.cpp. It is a real batched path (int8 tensor-core plus cuBLAS), but there’s more to close.
One verified model: Gemma 4 E4B. The quant formats are Q3_K, Q4_K and Q5_K only, so most Q4_K_M files from Hugging Face won’t load yet because they mix in Q6_K and Q8_0. Those formats are next, along with Qwen.
It has only been tested on sm_86 (RTX 30-series), and sampling is greedy only.

It’s not a drop-in llama.cpp replacement yet. The point of v0.0.1 is to show that the architecture works: once a graph compiler sits on top of the same kernels, the remaining gains come from the compiler instead of more hand-written kernels. Next on the roadmap are MLIR GPU codegen, speculative decoding, and more formats.

GPL-3.0. Benchmark conditions, reproduce commands and the model sha256 are in the README:
https://github.com/hardenedlinux/SonicBoom

DEV Community: Roy Mu

SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.