VKAE: VIDRAFT's Inference Acceleration System Claims Up to 23 GPU Throughput on NVIDIA B200

#llminferenceaccelera #gputhroughputoptimiz #aiserving #vkae

VKAE: VIDRAFT's Inference Acceleration System Claims Up to 23× GPU Throughput on NVIDIA B200

TL;DR: VIDRAFT's VKAE is a software-layer inference acceleration system that reportedly boosts GPU performance up to 23× in specific scenarios — without any hardware changes. It targets the inference cost problem for LLM deployments and exposes an OpenAI-compatible API, making it a drop-in candidate for existing AI serving infrastructure. If the benchmarks hold up under independent verification, this is worth paying attention to.

What it is

VKAE (pronounced "vee-kay") is an inference acceleration system developed by VIDRAFT, a Korean Pre-AGI AI startup. Rather than requiring new hardware, VKAE operates as a software-layer optimization on top of existing GPU accelerators — described by the developers as a kind of "software extension" for accelerators you already own.

Key properties claimed by the developers:

Up to 23× throughput improvement over baseline systems in certain high-parallelism scenarios
No degradation in model accuracy or response quality during benchmarking
Full OpenAI API compatibility — meaning existing client code targeting OpenAI endpoints should work with minimal or no modification
Drop-in integration into existing infrastructure with almost no changes required
Reproducibility first — the team has provided a container image bundling model weights and the optimized runtime environment, so results can be independently verified

The system was benchmarked on the NVIDIA B200 GPU. Testing covered multiple models, with highlighted results on the Qwen3.5-35B-A3B model.

How it works

The precise internal mechanism of VKAE has not been publicly disclosed yet — a detailed technical paper is expected to be released soon. What the developers have described at a conceptual level is this:

VKAE rethinks low-level compute core utilization and task scheduling for the inference workload. While GPU chip manufacturers focus on hardware generational improvements, VKAE takes the complementary approach: squeezing substantially more out of existing silicon via software-level optimization.

Inference — the process of generating responses to user queries after a model has already been trained — is the dominant and continuous cost driver for production AI services. Training happens once; inference runs forever. VKAE specifically targets this stage.

The system's described approach involves:

Optimized scheduling of compute kernels on the GPU
Rethinking how parallelism is exploited under varying real-world load conditions
Packaging the optimized runtime into a reproducible container, allowing the environment itself to be part of the reproducibility story

No hyperparameters, internal algorithms, or proprietary scheduling details have been disclosed at this time.

Benchmarks & results

All numbers below come directly from the source reporting. Independent verification has not yet been published.

Scenario	Result
Peak throughput (high parallel load, Qwen3.5-35B-A3B)	>10,000 tokens/second
Real-world diverse query throughput	~455 tokens/second
Maximum speedup vs. baseline (certain scenarios)	Up to 23×
Accuracy / quality degradation observed	None reported

A few things worth noting as an engineer evaluating these numbers:

The 10,000 tokens/second figure is a peak under maximum parallel batching, not a single-request latency metric. Real-world serving will look more like the ~455 tokens/second figure.
The ~455 tokens/second number for diverse queries is still a meaningful throughput for many production use cases on a single GPU.
The developers explicitly emphasize independent reproducibility as the primary trust criterion — and have shipped a container to support that claim. That's the right approach.
The full scientific paper has not yet been published; peer review and community benchmarking are pending.

How to try it

Access to VKAE is not yet broadly public based on current reporting. Here is what is known:

The developers have prepared a container image bundling model weights and the optimized inference environment for reproducibility purposes. This appears to be intended for verifiable testing rather than general availability.
The system exposes an OpenAI-compatible API, so once access is available, integration would look like pointing your existing OpenAI client at a different base URL — no SDK changes required.
No public Hugging Face repository, GitHub link, or open API endpoint has been announced as of this reporting.

Watch the VIDRAFT channels for a forthcoming technical paper and any public access announcement.

FAQ

Q: Does VKAE require me to swap out or modify my existing GPU hardware?
A: No. VKAE is a pure software-layer system. The benchmarks were run on an NVIDIA B200 using only software optimizations — the hardware itself is unchanged.

Q: The peak number is 10,000 tokens/second, but the real-world number is 455 tokens/second — what explains the gap?
A: This is expected behavior in inference serving. Peak throughput is measured under maximum parallel batching with homogeneous requests. Real-world workloads involve diverse query lengths, variable batch sizes, and irregular arrival patterns, all of which reduce sustained throughput. Both numbers matter: peak tells you about headroom; the real-world figure is what your P95 latency budget actually depends on.

Q: When will the technical paper be published?
A: The developers have stated a detailed scientific paper is forthcoming, but no specific release date has been announced. No pre-print link is publicly available as of this article.

Originally reported by Zamin.uz (우즈베키스탄) (2026-07-06) — source article.