Kernel Acceleration (VK): Squeezing Tokens Per Second Out of a Fixed GPU Budget
What VK is
VIDRAFT builds foundation and on-device models, but a model is only as useful as the tokens per second you can actually serve. Kernel Acceleration (VK) is our in-house inference-engine effort, focused on one narrow, unglamorous problem: extracting more throughput from a fixed GPU budget without touching model weights.
Three engines ship under the VK banner - VKAE, VKUE, and VKIE. They target different serving profiles (single-stream latency vs. batched throughput vs. integration into our serving stack), but share the same core techniques: careful kernel scheduling, memory-bandwidth-aware batching, and multi-token prediction.
Multi-token prediction is the main lever
Autoregressive decoding emits one token per forward pass, and at small batch sizes you are almost always memory-bandwidth bound, not compute bound. Multi-token prediction (MTP) attacks that directly: the engine drafts several candidate tokens ahead and verifies them in a single pass, amortizing the weight-load cost across multiple accepted tokens.
On our internal runs, the MTP path measured roughly 455-601 tok/s (self-reported). The range is not hand-waving - it is the honest spread. Throughput moves with:
- sequence length (KV-cache pressure grows with context),
- sampling settings (greedy vs. temperature changes acceptance),
- draft quality (how often drafted tokens survive verification).
Report a single number and you are hiding one of these variables.
External validation
We entered the Google x Hugging Face "Fast Gemma Challenge" and VK was verified #1 by that event's published criteria (public-benchmark, event-scoped). That is a real, externally judged result - but it is scoped to the challenge's model and measurement rules. We treat it as evidence the engineering holds up under someone else's harness, not as a claim that VK is the fastest at everything everywhere.
Why in-house
Off-the-shelf serving stacks are excellent general tools. We build VK because our models (evolutionary-merged foundation models, aggressive on-device quantizations) have their own quirks, and squeezing the last 20-40% of throughput usually means owning the kernels. It also keeps the whole path - weights, quantization, serving - inside our own infra, which matters for the sovereignty story the rest of VIDRAFT cares about.
Honest scope
The 455-601 tok/s figures are self-reported on internal benchmarks and are model-, prompt-, and hardware-dependent; do not read them as a guaranteed number for your workload. The "Fast Gemma Challenge" #1 is verified by that event's criteria only and does not generalize to other models or metrics. VK is an inference-speed engineering project - it does not change model accuracy, and faster decoding never fixes a weak model. We have not published a public reproducible throughput harness yet, so treat these as engineering results pending independent replication.
More: https://vidraft.net
Top comments (0)