This is the v0.0.5 update. It's about one thing: speculative decoding that's fast and doesn't change the output.
Same RTX 3050 8 GB, Qwen2.5-7B-Instruct Q4_K_M, same GGUF for both runtimes. llama.cpp 43fe9c64281 with -ngl 99 -fa on, using the fastest of llama-server's eight speculative modes (ngram-mod). The workload is 10 turns, each editing a 100–300-line source file, greedy.
SonicBoom llama.cpp
Code editing, speculative 489.8 tok/s
Turns identical, spec on vs off 10 / 10
Decode, spec off, ctx 512 / 4K 40.6 / 38.2
Prefill 512 / 8K tokens (wall) 1557 / 1200
Plain decode and prefill are still slightly behind. The speculative row is 12.7× our own plain decode, and every verified token is exactly what a plain decode step would have produced.
How the outputs stay identical:
the quantized matmul is column-invariant;
verify attention partitions keys by absolute position and reduces rows in a fixed order;
any kernel choice that affects numerics is made once at load and sealed.
So row i of a 64-row verify batch carries the same bits as decode step i. That's why speculation is on by default.
The drafter is a long-match n-gram one (up to 63 tokens per proposal). It wins wherever the output copies the input: edits, revisions, refactors. Coding chat gets 4.4×.
We only have sm_86. If you have another NVIDIA card with 8 GB or more, the benchmark issue has copy-paste commands that run all of the above against llama.cpp. There's also an optional check that two machines with the same GPU produce the same token hash:
Top comments (0)