Throughput vs. Reach: Why VIDRAFT Ships Two Serving Engines (VKAE x VKUE)

#machinelearning #ai #llm #opensource

Throughput vs. Reach: Why VIDRAFT Ships Two Serving Engines (VKAE × VKUE)

"Serving an LLM" is usually treated as a single optimization target. It isn't. There are two very different problems hiding under that phrase, and VIDRAFT ships a separate engine for each. (Korea's Electronic Times just covered the pair as a set for AI-data-center efficiency.)

Problem 1 — throughput: get more out of the GPU you have

VKAE is a kernel-level acceleration engine. Same GPU, same output quality, more tokens out.

Up to 23.4× higher throughput vs. standard serving
601 tok/s single-stream (Qwen3.5-35B-A3B)
10,000+ tok/s multi-request; 18,000+ tok/s sustained at 256 concurrent users on a B200

This is the batching/scheduling regime: when you're GPU-bound and request-rich, the win is in how efficiently you pack work onto the accelerator.

Problem 2 — reach: remove the GPU requirement

VKUE runs a 34.7B model across a four-orders-of-magnitude hardware range — the same file:

Hardware	tok/s
B200	18,057
A10G ×1	126
8GB laptop	20
CPU, no GPU	~17

The mechanism is memory bandwidth, not FLOPs. Autoregressive decode is bandwidth-bound; a sparse Mixture-of-Experts streams only the active experts. Ourbox-35B-JGOS is 34.7B total but ~3B active per token, so per-token memory traffic drops enough that an 8GB card — or a CPU — is viable. Quality: GPQA Diamond 86.4% (maj@8).

Why two engines instead of one

Because the two problems pull in opposite directions:

	VKAE	VKUE
Optimizes	throughput on data-center GPUs	reach on minimal hardware
Bound by	GPU compute/scheduling	memory bandwidth
Wins when	GPU-bound, request-rich	GPU-scarce or air-gapped
Ships as	OpenAI-compatible API + Docker	open model (GGUF) + demos