VIDRAFT VKAE: A Kernel-Level LLM Inference Engine Claiming Up to 23.4 Throughput on the Same GPU

#llminference #kernelacceleration #gpuoptimization #vkae

VIDRAFT VKAE: A Kernel-Level LLM Inference Engine Claiming Up to 23.4× Throughput on the Same GPU

TL;DR: VIDRAFT's VKAE (VIDRAFT Kernel Acceleration Engine) is a kernel-level inference acceleration engine for large language models that reportedly delivers up to 23.4× throughput over standard serving on the same GPU hardware. It ships as a unified Docker container bundling model weights and the serving environment, exposes an OpenAI-compatible API, and allows engineers to reproduce benchmark results directly on their own GPU hardware.

What it is

VKAE — short for VIDRAFT Kernel Acceleration Engine — is a kernel-level toolchain designed to accelerate LLM inference without degrading output quality. Rather than modifying a model's weights or architecture, VKAE operates at the system and kernel layer beneath the model serving stack.

Key characteristics from the announcement:

Packaging: Ships as a public Docker container that bundles model weights and the full serving environment together, reducing deployment friction.
API surface: Implements an OpenAI-compatible API, meaning it can be swapped into existing pipelines that already call OpenAI-style endpoints with minimal code changes.
Target hardware: Primarily targets NVIDIA Blackwell (B200) and Hopper (H100, H200) GPU series, with expanding support for smaller GPUs such as the A10G.
Reproducibility-first design: VIDRAFT explicitly frames external verification as a core feature — users can run the container on their own hardware and check the numbers themselves rather than trusting vendor-reported figures alone.
Supported models: The announcement references the Qwen3.5-35B-A3B model as a representative benchmark target, and notes that the ultra-large-scale JGOS-398B model applies the same family of optimization techniques.

VKAE fits into VIDRAFT's broader full-stack AI infrastructure, which also includes the FINAL Bench (a metacognition evaluation benchmark), the MARL runtime middleware (aimed at hallucination reduction), and model families under the Darwin, Chimera, and Aether series.

How it works

VIDRAFT explicitly states that the kernel's internal implementation is a trade secret and will not be disclosed. What can be said at a conceptual level, based on public statements:

VKAE operates at the kernel level, meaning optimizations happen below the model-serving framework layer — think custom GPU kernel implementations rather than framework-level batching tweaks.
The engine is positioned as a drop-in acceleration layer: it accelerates inference throughput and response latency while preserving output quality, implying the modifications do not alter the model's numerical outputs in a meaningful way.
The FP8 precision format is used in at least one documented configuration (B200 + FP8), which is consistent with modern low-precision inference techniques that trade marginal numerical range for significant compute and memory bandwidth gains on supported hardware.
The unified container approach suggests VKAE handles environment parity concerns — the same image that was benchmarked is the one users deploy, reducing the "works on my machine" gap for performance claims.

The throughput improvement being tied to "the same GPU conditions" means the gains are attributed to software-side kernel optimization rather than hardware upgrades.

Benchmarks & results

All figures below are VIDRAFT's own measurements as reported in the press coverage. Independent third-party verification has not been cited.

Configuration	Metric	Reported Value
GPU: NVIDIA B200, Precision: FP8, Model: Qwen3.5-35B-A3B	Single-stream token generation	Up to 601 tokens/sec
Same config, multi-request concurrency	Aggregate throughput	Exceeds 10,000 tokens/sec
vs. standard serving baseline (same GPU)	Throughput multiplier	Up to 23.4×

VIDRAFT emphasizes four properties of this release:

Verifiability — results are reproducible by users on their own hardware via the public container.
Cost reduction — higher throughput per GPU translates directly to lower per-token serving cost.
Quality preservation — acceleration does not degrade model output quality.
Immediate integration — OpenAI-compatible API means no bespoke client changes.

How to try it

According to the announcement, VIDRAFT has deployed a public Docker container that integrates model weights and the serving environment. The container supports an OpenAI-compatible API.

If you have access to a supported NVIDIA GPU (B200, H100, H200, or A10G), you can in principle pull and run the container in your own environment. A generic OpenAI-compatible request would follow the standard pattern:

curl http://<your-container-host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-api-key>" \
  -d '{
    "model": "<model-name-as-served>",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

⚠️ Note: The specific Docker image name, registry URL, port, and model identifiers are not included in the source article. Refer to VIDRAFT's official channels for the exact pull command and endpoint configuration before deploying.

FAQ

Q: Does VKAE require retraining or fine-tuning the model?
A: Based on the announcement, no. VKAE is described as a kernel-level acceleration layer that operates beneath the model itself and is stated to preserve output quality — implying the model weights are used as-is.

Q: Can I use VKAE if I'm not on a Blackwell or Hopper GPU?
A: Primary support targets NVIDIA B200, H100, and H200. The announcement notes that support is expanding to smaller GPUs such as the A10G, but broad multi-GPU compatibility is not confirmed at this time.

Q: Is the 23.4× figure comparable to vLLM or TGI baselines?
A: The source states the comparison is against "standard serving" on the same GPU, but does not specify which serving framework was used as the baseline. Engineers should run the public container on their own hardware to form an independent judgment.

Q: What models are confirmed to work with VKAE?
A: The source explicitly names Qwen3.5-35B-A3B (benchmark reference) and JGOS-398B (ultra-large scale). Other models are not specified in this announcement.

Originally reported by Wedoany (포르투갈어) (2026-07-07) — source article.