Training Transformers on 'Inference-Only' NPUs: How VKORE Reached 0.9999 Parity

#ai #llm #machinelearning #hardware

TL;DR: NPUs are sold as inference-only accelerators. VKORE, a hardware-independent acceleration layer, leaves the chip untouched and (1) raises single-user inference 3.8x and server throughput 4.1x on the same NPU server, and (2) runs real transformer pretraining on that NPU at numerical parity with standard training. Measured pretraining throughput reached 170% of a single Nvidia H100 and about 75% of a single B200 per device. Measured on a Korean NPU server (8 cards) with Google's open Gemma 4 26B.

What problem does VKORE solve?

Data center economics are shifting from "buy more GPUs" to "get more intelligence from the same power and the same silicon." NPUs win on power efficiency for inference, which is why they are spreading fast in inference fleets. The industry treats them as inference-only. Training, the thinking goes, belongs to GPUs.

VKORE challenges that split. If NPUs can also train, operators avoid a separate GPU build-out just for training. For teams on sovereign (non-Nvidia) silicon, it means the full loop, build, serve, and improve, can run without Nvidia in the critical path.

How much faster is inference on the same NPU?

We applied VKORE to a Korean NPU server (8 cards) and measured Gemma 4 26B before and after. Nothing about the chip changed. Only the software layer did.

Single-user response: 14.1 to 53.9 tokens per second (3.8x).
Server throughput: 214 to 881 tokens per second (4.1x).
Under 64 concurrent users (overload), per-token generation time stayed at 29 ms with zero failed requests.

The last point matters for operators: when demand exceeds capacity, each user still felt a steady speed. That makes service-level promises easier to keep.

Can an inference-only NPU really pretrain?

Yes, with VKORE. We ran transformer pretraining (attention plus feed-forward blocks) across the NPU server, and the numerical behavior matched standard training, so this is real training rather than an approximation.

Pretraining throughput per device reached 170% of a single H100.
The same workload reached about 75% of a single B200 per device.
Numerical behavior matched standard training.

An NPU that most of the industry treats as inference-only outran an H100 on pretraining throughput in our setup, and came within range of a B200, the current flagship.

Why does the ZTC decision engine pair well with NPUs?

A decision engine returns a judgment, not generated text. VIDRAFT's Darwin ZTC finishes a judgment in a single forward pass with zero output tokens. That is a large number of short, deterministic computations rather than long token-by-token generation, which maps cleanly onto a power-efficient NPU. On the S1MB public leaderboard, Darwin-27B-ZTC-v5 ranks first overall among 152 models, so putting a top decision engine on NPUs increases how many judgments the same power budget can serve.

The night-shift loop

Put the pieces together and one data center can run a cycle:

Serve inference by day.
Let the decision engine score the day's traffic and pick what to learn from.
Let the same NPUs pretrain or fine-tune at night.

The decision engine selects data, self-improvement trains the model, and the acceleration layer runs all of it on NPUs. The method is patent-pending and the internal details are not disclosed. Results will be published in a paper.

FAQ

Is VKORE tied to one chip vendor?
No. It is designed as a hardware-independent layer intended for data centers that mix GPUs and NPUs.

Does pretraining on NPU lose accuracy?
In our measurement, the numerical behavior matched standard training.

An NPU beating an H100 on pretraining?
Per device, pretraining throughput with VKORE reached 170% of an H100 in our configuration. Against a B200 it reached about 75%.

What model and hardware were used?
Google's open Gemma 4 26B on a Korean NPU server with 8 cards.

Where can I read more?
VIDRAFT plans to publish the measurement methodology as a paper.

DEV Community