I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs.
Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes using the same GGUF on the same machine, llama.cpp at commit 43fe9c64281):
SonicBoom llama.cpp
Decode (tg64) 51.65 tok/s 45.20 tok/s
Decode @ depth 2048 47.13 tok/s 45.21 tok/s
Prefill 4100 tokens 1051 tok/s ~1944 tok/s
TTFT, 4000-token prefix + 100 new 503 ms (prefix reuse)
Why it matters: the quantized matmul kernels (MMVQ/MMQ) are ported from ggml, so the kernels are the same ones llama.cpp uses. The 14% decode lead comes entirely from the layer above them. A graph planner rewrites the model before it runs: activations are quantized once and reused, norm and activation are fused with quantization, dead computation is removed, and each decode step is captured and replayed as a single CUDA Graph. Code derived from ggml is about 8% of the core. The rest is the runtime.
Prefix reuse is exact: a continuation is token-for-token identical to a full prefill, which makes multi-turn chat and coding-assistant workloads cheap.
What it can’t do yet:
Prefill is about half of llama.cpp. It is a real batched path (int8 tensor-core plus cuBLAS), but there’s more to close.
One verified model: Gemma 4 E4B. The quant formats are Q3_K, Q4_K and Q5_K only, so most Q4_K_M files from Hugging Face won’t load yet because they mix in Q6_K and Q8_0. Those formats are next, along with Qwen.
It has only been tested on sm_86 (RTX 30-series), and sampling is greedy only.
It’s not a drop-in llama.cpp replacement yet. The point of v0.0.1 is to show that the architecture works: once a graph compiler sits on top of the same kernels, the remaining gains come from the compiler instead of more hand-written kernels. Next on the roadmap are MLIR GPU codegen, speculative decoding, and more formats.
GPL-3.0. Benchmark conditions, reproduce commands and the model sha256 are in the README:
https://github.com/hardenedlinux/SonicBoom
Top comments (1)
This is a nicely controlled experiment — same kernels, same GGUF, same machine — so the delta genuinely isolates the runtime layer, which is exactly how these comparisons should be run. The part that caught my attention is that decode holds its lead at depth 2048: that's the regime where KV-cache traffic starts to dominate, so the graph-level wins (quantize-once activation reuse, fusion, single CUDA Graph replay per step) clearly aren't being washed out by memory bandwidth yet. One question on the prefill side: is the gap mostly in the epilogue/fusion coverage of the batched int8 path rather than the tensor-core GEMM itself, and do you expect the planner's fusion passes to transfer to prefill as well? Also curious how the exact prefix-reuse mechanism will compose with paged KV at longer contexts — keeping token-for-token equivalence there would be a strong property for coding-assistant workloads.