VIDRAFT VKAE Engine: Kernel-Level LLM Inference Acceleration Hitting 23.4× Throughput Gains

TL;DR: VIDRAFT has publicly released VKAE (VIDRAFT Kernel Acceleration Engine), a kernel-level inference acceleration engine for large language models that delivers up to 23.4× higher throughput compared to standard serving under identical GPU conditions. It ships as a unified Docker container bundling model weights and serving environment, exposes an OpenAI-compatible API, and is externally verifiable — meaning you can reproduce the published benchmark numbers on your own hardware.

What it is

VKAE — short for VIDRAFT Kernel Acceleration Engine — is a kernel-level acceleration layer designed to dramatically increase LLM inference throughput without degrading output quality. Key characteristics:

• Scope: Operates at the GPU kernel level, below the framework/runtime layer, targeting raw compute efficiency during inference.
• Deployment unit: A single Docker container that bundles model weights and the full serving environment, so there is no multi-step setup to wire together.
• API surface: Exposes an OpenAI-compatible API, which means any application already talking to the OpenAI SDK or any OpenAI-compatible client can switch to VKAE-backed endpoints with minimal code changes.
• Quality preservation: VIDRAFT states that throughput and latency improvements are achieved while maintaining output quality — the acceleration does not involve lossy output-level shortcuts.
• GPU targets: Primarily supports NVIDIA Blackwell (B200) and Hopper (H100, H200) series GPUs, with support being extended to smaller cards such as the A10G.

VKAE is positioned as VIDRAFT's inference infrastructure layer within a broader full-stack AI portfolio that also includes the FINAL Bench (AI metacognitive evaluation), the MARL runtime middleware (hallucination reduction), and model families including Darwin, Chimera, and Aether.

How it works

VIDRAFT has explicitly designated the internal kernel implementation as a trade secret, so the following is limited to what is publicly disclosed at a conceptual level:

• The engine operates at the kernel level, meaning optimizations are applied at the level of GPU compute kernels rather than purely at the orchestration or batching layer.
• The same set of optimization techniques scales across model sizes — from mid-range models (~35B parameters, mixture-of-experts architecture) up to the ultra-large JGOS-398B model — suggesting the approach is architecture-agnostic at a high level rather than hand-tuned per model.
• The container design (weights + serving environment co-packaged) implies that the kernel optimizations are tightly coupled to the serving stack, reducing the surface area for misconfiguration that often degrades real-world inference performance.
• External verification is explicitly supported: users are expected to run the container on their own GPU hardware and reproduce the benchmark figures themselves, rather than take vendor-reported numbers at face value.

The reproducibility-first design philosophy is notable: VIDRAFT CEO Kim Min-sik is quoted stating that "speed competition without reproducibility is meaningless," framing verifiability as a core product property.

Benchmarks & results

All figures below are from VIDRAFT's publicly released benchmark. The test configuration uses an NVIDIA B200 GPU with FP8 precision:

Model	Mode	Result
Qwen3.5-35B-A3B	Single-stream	Up to 601 tokens/second
Qwen3.5-35B-A3B	Multi-request (batched)	Throughput exceeds 10,000 tokens/second

• Peak throughput improvement: Up to 23.4× higher than standard serving under the same GPU and conditions.
• The same optimization techniques have been applied to the JGOS-398B ultra-large model, though specific per-token numbers for that model are not listed in the public benchmark release.
• All figures are described as internally measured and externally reproducible via the public container.

How to try it

VIDRAFT has made the benchmark and Docker container publicly available. Based on what the source discloses:

• Docker container: A unified container integrating model weights and the serving environment has been deployed publicly. Users are encouraged to pull and run it on a compatible GPU (B200, H100, H200, or A10G) to reproduce benchmark results directly.
• OpenAI-compatible API: Once the container is running, you can connect any OpenAI-compatible client to it. A generic example using the OpenAI SDK pattern would work without code changes beyond pointing the base URL at your local container endpoint.

# Generic OpenAI-compatible client call — point base_url at your running VKAE container
curl http://<your-container-host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "your-deployed-model", "messages": [{"role": "user", "content": "Hello"}]}'

Note: Specific container registry URLs, Hugging Face model IDs, and GitHub repositories were not listed in the source article. Check VIDRAFT's official channels for the current public links.

FAQ

Q: Do I need NVIDIA Blackwell hardware (B200) to use VKAE?
A: No, but the headline benchmark figures (including the 23.4× figure and 601 tokens/second result) were measured on a B200 with FP8 precision. VKAE also supports Hopper-generation GPUs (H100, H200), and support for smaller cards like the A10G is actively being extended. Results on other hardware will vary.

Q: Does the OpenAI-compatible API mean I can drop VKAE into an existing LangChain or LlamaIndex pipeline?
A: In principle, yes — any framework or SDK that accepts a configurable base URL for OpenAI-compatible endpoints should work without modification. You point your client at the running VKAE container endpoint instead of the OpenAI endpoint.

Q: Is the kernel implementation open source?
A: No. VIDRAFT explicitly classifies the internal kernel implementation as a trade secret. What is public is the benchmark methodology and the Docker container, which are designed to allow external performance verification without exposing implementation details.

---

Originally reported by Wedoany (영문) (2026-07-07) — source article.