DEV Community: Roy Mu

Squeezing an RTX 3050: fast and exact LLM inference with SonicBoom

Roy Mu — Wed, 11 Nov 2026 08:22:21 +0000

Recently I released SonicBoom v0.0.6. SonicBoom is a PyTorch alternative, and I chose my favorite GNU Guile as its first-class language. I feel comfortable that I can finally do AI things with Scheme. There is no Python anywhere at runtime. The Python ecosystem caused me a lot of pain in product deployment, and now it's gone. The SBOM is reduced to a minimum, and we know what we deploy.

These days most people prefer not to type code by themselves (I still write the compiler and key infra myself), so the meaning of a programming language is the safety and optimization that come from the nature of its features. I'll have another post about programming languages in the AI era.

My idea is a bit old school hacker

I only have an RTX 3050. I once considered buying a 4090 or 5090 for my AI research, that's the mainstream. But I suddenly remembered John Carmack once said in 2013:

"There's so much you can still do on the previous console generation."

Besides, the mainstream autonomous driving chip has a similar config, say, NVIDIA DRIVE Orin-X / Jetson AGX Orin. And on AWS, A10G has the same architecture as 3050 (sm_86) with bigger VRAM. It's good bang for the buck.

So that's it, 3050 is good enough for a start.

However, I'm not going to optimize specifically for the 3050. The optimizations must be general, so you get higher performance on a better GPU.

So what did we get in v0.0.6?

Speculative decoding is on by default, and it is exact. The tokens you get with speculation on are the tokens you get with it off, bit for bit, on every run.

Qwen2.5-7B edits code at up to 523 tok/s on an RTX 3050 (8GB).

Of course you may not want to use Qwen2.5-7B for code editing, it's too weak. But it shows the power of SonicBoom. Next, we're working on MoE, e.g. Qwen3.6-35B-A3B, with flexible memory management to run big MoE models on 8GB VRAM. It seems popular today to do so.

The machine is a desktop RTX 3050 8 GB (sm_86, 20 SMs, roughly 200 GB/s of memory bandwidth) with a Ryzen 5 5500. The model is Qwen2.5-7B-Instruct Q4_K_M.

The workload is code editing: ten requests, each pasting a 100–300-line source file and asking for an edit. Decoding is greedy, and the generation length is the file's length.

The result is 471–523 tok/s for code editing. But when writing brand-new code, the speed drops back to about 41 tok/s. It's reasonable since code editing spends most of the time on copying rather than generation. But I would say code editing is one of the most common needs, so it's worth optimizing for it. The full benchmark is in the README.

But speed is not the point; quality is. Speculative decoding is a double-edged sword: it raises the speed, but in most engines it also changes the output, because the drafted tokens are checked with different kernels than plain decoding uses.

Why speculation usually changes the output

Speculative decoding drafts several tokens cheaply, then checks them in one forward pass of the real model. The pass scores k+1 positions at once, and a drafted token is kept when it matches what the model would have produced. With greedy decoding this is lossless in exact arithmetic: you can only ever accept tokens the model would have picked anyway.

GPUs don't do exact arithmetic. A verify pass is a batch of k+1 rows, while a plain decode step is a batch of one, and engines are built to pick kernels by batch shape. A matrix-vector kernel serves one row, a tiled matrix-matrix kernel serves sixteen, and the attention kernel splits its keys differently for different query counts. Each choice changes the order in which floating-point numbers are added up. That changes the last bits of the logits, and whenever two candidate tokens are close, the last bits decide which one wins. After one flipped token, the rest of the text goes its own way.

That is why, in many engines, turning speculation on can change the output. It is also why a speedup claimed with speculation on is hard to judge. You are no longer comparing the same text.

How SonicBoom keeps every row exact

However, this is just a simple case. In actual engineering, there are more things that make it complex:

kernel selection by batch shape
split-K and stream-K
fused vs. unfused epilogues
split-KV attention partitions
online softmax rescaling
KV cache write paths
rollback of rejected drafts
sliding-window ring reuse
stopping inside a verify batch
timing-based autotuning
......

BTW, each of them has its own test in SonicBoom's test suite.

Making verification cheap enough to matter

Exactness only pays if verification is cheap. On this card, a plain Qwen2.5-7B step is about 24.6 ms, essentially the time to stream the weights once. A 16-row verify costs about 1.25× that. A 64-row verify costs about 2.9×, and when a 63-token draft is accepted, that one step emits 64 tokens.

SonicBoom verifies up to 64 rows per step. Every row bucket (2 to 16, 24, 32, 48 and 64) is built, warmed and captured as a CUDA Graph when the model loads, so a verify step costs one graph replay.

Reproducible, with stated limits

SonicBoom's claim is: same version, same GPU model and same warm-up table give the same output. That includes seeded sampling: draws are keyed by seed, position and token id, so they don't depend on how the tokens were batched.

The warm-up table is measured on your own GPU when a model loads. With the table imported, a second machine with the same GPU model skips measuring entirely and should produce the same tokens. We have checked this on one machine: export, import into an empty cache, zero re-measurements, identical hashes on all ten turns. We would like to see it across machines; see the benchmark issue at the end.

What is not covered yet is batched serving: several requests sharing one batch.

Why reproducibility matters

SonicBoom is an official project of HardenedLinux, an old-school hacker community. We care about security and individual sovereignty. Our philosophy is "for fun and for profit". SonicBoom is one of our choices to help individuals get affordable AI power, and we also use it for serious products that require security. Reproducibility is critical in security and auditing.

When the same input gives the same output, you can replay any answer, explain it, and prove it hasn't changed. That is trustworthy AI.

Run it yourself

Repository: https://github.com/hardenedlinux/SonicBoom

And please help us benchmark v0.0.6 publicly:

https://github.com/hardenedlinux/SonicBoom/issues/5

489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding

Roy Mu — Mon, 09 Nov 2026 13:49:52 +0000

This is the v0.0.5 update. It's about one thing: speculative decoding that's fast and doesn't change the output.

Same RTX 3050 8 GB, Qwen2.5-7B-Instruct Q4_K_M, same GGUF for both runtimes. llama.cpp 43fe9c64281 with -ngl 99 -fa on, using the fastest of llama-server's eight speculative modes (ngram-mod). The workload is 10 turns, each editing a 100–300-line source file, greedy.

SonicBoom llama.cpp
Code editing, speculative 489.8 tok/s
Turns identical, spec on vs off 10 / 10
Decode, spec off, ctx 512 / 4K 40.6 / 38.2
Prefill 512 / 8K tokens (wall) 1557 / 1200
Plain decode and prefill are still slightly behind. The speculative row is 12.7× our own plain decode, and every verified token is exactly what a plain decode step would have produced.

How the outputs stay identical:

the quantized matmul is column-invariant;

verify attention partitions keys by absolute position and reduces rows in a fixed order;

any kernel choice that affects numerics is made once at load and sealed.

So row i of a 64-row verify batch carries the same bits as decode step i. That's why speculation is on by default.

The drafter is a long-match n-gram one (up to 63 tokens per proposal). It wins wherever the output copies the input: edits, revisions, refactors. Coding chat gets 4.4×.

We only have sm_86. If you have another NVIDIA card with 8 GB or more, the benchmark issue has copy-paste commands that run all of the above against llama.cpp. There's also an optional check that two machines with the same GPU produce the same token hash:

https://github.com/hardenedlinux/SonicBoom/issues/4

https://github.com/hardenedlinux/SonicBoom

Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode

Roy Mu — Mon, 19 Oct 2026 10:42:39 +0000

Hi folks!

We are HardenedLinux, an old-school free software community. SonicBoom is our open-source (GPLv3) inference runtime: a compiler-first alternative to the usual Python stack. No Python at build or run time — one shared library, a C ABI, and Guile Scheme as the first-class language.

What v0.0.3 does on a desktop RTX 3050 8 GB (same session, same GGUF, vs llama.cpp -ngl 99):

On real multi-turn agent workloads, time-to-first-token dropped several-fold since v0.0.2 (prefix reuse + plan caching). Output tokens are bit-identical to our golden runs, and prefix reuse is bitwise identical to a full prefill.

How: all graph optimization and kernel generation happen on Boom IR, an MLIR dialect; hot quantized matmuls run as attributed ggml-derived kernels. Every optimization has a switch and an ablation number, so you can see what each one buys.

Where we are honest about gaps: Qwen decode at depth 512 is 0.98×, and we have only one GPU. That is exactly why we need you.

The invitation: every release gets one benchmark issue. If you have an NVIDIA GPU (any generation — Ada, Hopper, Blackwell especially welcome), one command produces a report you can paste. Everyone whose results we publish is credited in the README.

→ v0.0.3 benchmark issue: https://github.com/hardenedlinux/SonicBoom/issues/2

Please post results and problems in the issue rather than here, so everything stays in one place.

SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.

Roy Mu — Tue, 13 Oct 2026 18:16:49 +0000

I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs.

Numbers (Gemma 4 E4B-it Q3_K_M on a laptop RTX 3050 8 GB, both runtimes using the same GGUF on the same machine, llama.cpp at commit 43fe9c64281):

       SonicBoom                    llama.cpp

Decode (tg64) 51.65 tok/s 45.20 tok/s
Decode @ depth 2048 47.13 tok/s 45.21 tok/s
Prefill 4100 tokens 1051 tok/s ~1944 tok/s
TTFT, 4000-token prefix + 100 new 503 ms (prefix reuse)

Why it matters: the quantized matmul kernels (MMVQ/MMQ) are ported from ggml, so the kernels are the same ones llama.cpp uses. The 14% decode lead comes entirely from the layer above them. A graph planner rewrites the model before it runs: activations are quantized once and reused, norm and activation are fused with quantization, dead computation is removed, and each decode step is captured and replayed as a single CUDA Graph. Code derived from ggml is about 8% of the core. The rest is the runtime.

Prefix reuse is exact: a continuation is token-for-token identical to a full prefill, which makes multi-turn chat and coding-assistant workloads cheap.

What it can’t do yet:

Prefill is about half of llama.cpp. It is a real batched path (int8 tensor-core plus cuBLAS), but there’s more to close.
One verified model: Gemma 4 E4B. The quant formats are Q3_K, Q4_K and Q5_K only, so most Q4_K_M files from Hugging Face won’t load yet because they mix in Q6_K and Q8_0. Those formats are next, along with Qwen.
It has only been tested on sm_86 (RTX 30-series), and sampling is greedy only.

It’s not a drop-in llama.cpp replacement yet. The point of v0.0.1 is to show that the architecture works: once a graph compiler sits on top of the same kernels, the remaining gains come from the compiler instead of more hand-written kernels. Next on the roadmap are MLIR GPU codegen, speculative decoding, and more formats.

GPL-3.0. Benchmark conditions, reproduce commands and the model sha256 are in the README:
https://github.com/hardenedlinux/SonicBoom