This is the v0.0.5 update. It's about one thing: speculative decoding that's fast and doesn't change the output.
Same RTX 3050 8 GB, Qwen2.5-7B-Instruct Q4_K_M, same GGUF for both runtimes. llama.cpp 43fe9c64281 with -ngl 99 -fa on, using the fastest of llama-server's eight speculative modes (ngram-mod). The workload is 10 turns, each editing a 100–300-line source file, greedy.
SonicBoom llama.cpp
Code editing, speculative 489.8 tok/s
Turns identical, spec on vs off 10 / 10
Decode, spec off, ctx 512 / 4K 40.6 / 38.2
Prefill 512 / 8K tokens (wall) 1557 / 1200
Plain decode and prefill are still slightly behind. The speculative row is 12.7× our own plain decode, and every verified token is exactly what a plain decode step would have produced.
How the outputs stay identical:
the quantized matmul is column-invariant;
verify attention partitions keys by absolute position and reduces rows in a fixed order;
any kernel choice that affects numerics is made once at load and sealed.
So row i of a 64-row verify batch carries the same bits as decode step i. That's why speculation is on by default.
The drafter is a long-match n-gram one (up to 63 tokens per proposal). It wins wherever the output copies the input: edits, revisions, refactors. Coding chat gets 4.4×.
We only have sm_86. If you have another NVIDIA card with 8 GB or more, the benchmark issue has copy-paste commands that run all of the above against llama.cpp. There's also an optional check that two machines with the same GPU produce the same token hash:
Top comments (2)
Interesting performance results on an RTX 3050. Have you tested the same setup with code-editing tasks from different programming languages? I'm curious whether domain-specific fine-tuning could improve editing accuracy while maintaining this generation speed.
Thanks! Not yet: the current code-editing set is ten C++ edits taken from SonicBoom’s own source, each a small change where the model returns the whole file. A multi-language set (Python, TypeScript, Rust, Go) is a good idea and we’ll add it.
We’d expect the speed to depend less on the language than on how much of the output is copied from the input: returning a whole file with a small edit lets most drafts be accepted, while diff-only output or large rewrites leave less to copy.
On fine-tuning: speculative decoding in SonicBoom is lossless, so the output is token-for-token the same with it on or off. Editing accuracy is entirely the model’s, and the speedup carries over to any fine-tuned model in GGUF format. A model tuned to reproduce unchanged code faithfully would, if anything, accept more drafts.