VKAE: VIDRAFT's Inference Engine Hits 23 GPU Speedup and ~10K Tokens/sec on a Single Nvidia B200

#llminference #gpuacceleration #vkae #nvidiab200

VKAE: VIDRAFT's Inference Engine Hits 23× GPU Speedup and ~10K Tokens/sec on a Single Nvidia B200

TL;DR: VIDRAFT, a Korean Pre-AGI AI startup, has developed VKAE — an inference acceleration system that achieves up to 23× GPU utilization improvement on a single Nvidia B200, delivering approximately 10,000 tokens per second on the Qwen3.5-35B-A3B model. The system exposes an OpenAI-compatible API, making it a drop-in target for existing LLM toolchains.

What it is

VKAE is VIDRAFT's proprietary inference engine designed to dramatically increase the throughput efficiency of large language model (LLM) serving on modern accelerator hardware. According to reporting from Russia's RusNews, the system was benchmarked on a single Nvidia B200 GPU and demonstrated:

Up to 23× improvement in GPU utilization compared to baseline inference configurations
~10,000 tokens per second throughput when serving Qwen3.5-35B-A3B, a 35B-parameter Mixture-of-Experts model with a 3B active parameter count
A baseline comparison point of approximately 455 tokens per second on the same model under standard serving conditions

VKAE is positioned as a full inference serving stack, not merely a model-level optimization. It targets production deployment scenarios where throughput, latency, and GPU cost-per-token are critical operational concerns.

How it works

The source article does not disclose internal implementation details, and specifics remain proprietary. At a conceptual level, VKAE appears to be a systems-level inference runtime that re-architects how the GPU is utilized during autoregressive decoding and prefill — the two phases where most LLM serving stacks leave significant hardware capacity on the table.

What the reporting indicates at a high level:

Hardware-aware scheduling: The engine is specifically tuned for the Nvidia B200's memory bandwidth and compute characteristics, suggesting tight integration with the hardware's architectural features rather than generic kernel reuse.
Throughput-first design: The 23× figure refers to GPU utilization efficiency, implying that existing frameworks leave substantial compute idle. VKAE reclaims that idle capacity through better batching, memory management, or kernel-level scheduling — though the exact mechanism is not publicly described.
MoE model affinity: The Qwen3.5-35B-A3B result (35B total, 3B active) suggests the engine is optimized for sparse Mixture-of-Experts routing patterns, which are notoriously difficult to serve efficiently due to irregular memory access and load imbalance across experts.
Accuracy preservation: The report notes that output quality is maintained at parity with standard inference — speedups are not achieved by degrading model fidelity.

The article explicitly benchmarks VKAE against well-known open-source inference frameworks such as vLLM and TensorRT-LLM, as well as commercial inference providers Groq and Cerebras — positioning VKAE as competitive across both the open-source and managed-service landscape.

Benchmarks & results

All numbers below come directly from the RusNews source article:

Metric	Value
GPU	Single Nvidia B200
Model	Qwen3.5-35B-A3B
VKAE throughput	~10,000 tokens/sec
Baseline throughput (same model)	~455 tokens/sec
GPU utilization improvement	Up to 23×
Output quality vs. baseline	Reported as equivalent

Comparison baselines cited in the article: vLLM, TensorRT-LLM, Groq, Cerebras.

No additional benchmark dimensions (latency percentiles, batch size configurations, context length sweep, or TTFT figures) are disclosed in the source.

How to try it

VKAE is not yet publicly available as an open-source release. There is no Hugging Face repository, GitHub link, or public API endpoint announced in the source article.

What is confirmed:

The system exposes an OpenAI-compatible API, meaning that if and when access is granted, existing code using the OpenAI Python SDK or any OpenAI-compatible client can point to VKAE with minimal changes.
Enterprise and partnership inquiries appear to be the current access path, based on the article's framing.

If you want to follow availability announcements:

Watch VIDRAFT's official channels for Hugging Face or GitHub releases.
The OpenAI-compatible interface means integration, once available, would look familiar — standard /v1/chat/completions endpoints with your existing client libraries.

FAQ

Q: How does 10,000 tokens/sec on a single B200 compare to what vLLM delivers on the same hardware?
A: The source article states VKAE outperforms vLLM (and TensorRT-LLM, Groq, and Cerebras) on this benchmark, but does not publish the exact per-system token throughput figures for each competitor. The 455 tokens/sec figure appears to represent a baseline serving configuration, not necessarily the peak of any specific named framework.

Q: Does VKAE require VIDRAFT-specific model weights, or does it work with standard Hugging Face checkpoints?
A: The benchmark uses Qwen3.5-35B-A3B, a publicly available model. The article does not indicate that VKAE requires proprietary weight formats, but runtime compatibility details have not been publicly disclosed.

Q: Is the 23× figure a throughput improvement or a latency improvement?
A: Per the source, the 23× figure specifically refers to GPU utilization efficiency — meaning how effectively the hardware is kept busy — rather than a raw latency reduction. Throughput (tokens/sec) improvements are separately stated as the ~10,000 vs. ~455 tokens/sec comparison.

Originally reported by RusNews (러시아) (2026-07-06) — source article.