TL;DR: NPUs are sold as inference-only accelerators. VIDRAFT built VKORE, a software acceleration layer that leaves the chip untouched and (1) raises single-user inference 3.8x and server throughput 4.1x on the same NPU server, and (2) runs real transformer LoRA training on that NPU at 0.9999 numerical parity with standard PyTorch, about 170x faster than the same server's CPU. Measured on a Korean NPU server (8 cards) with Google's open Gemma 4 26B.
What problem does VKORE solve?
Data center economics are shifting from "buy more GPUs" to "get more intelligence from the same power and the same silicon." NPUs win on power efficiency for inference, which is why they are spreading fast in inference fleets. The industry treats them as inference-only. Training, the thinking goes, belongs to GPUs.
VKORE challenges that split. If NPUs can also train, operators avoid a separate GPU build-out just for fine-tuning.
How much faster is inference on the same NPU?
We applied VKORE to a Korean NPU server (8 cards) and measured Gemma 4 26B before and after. Nothing about the chip changed. Only the software layer did.
- Single-user response: 14.1 to 53.9 tokens per second (3.8x).
- Server throughput: 214 to 881 tokens per second (4.1x).
- Under 64 concurrent users (overload), per-token generation time stayed at 29 ms with zero failed requests.
The last point matters for operators: when demand exceeds capacity, each user still felt a steady speed. That makes service-level promises easier to keep.
Can an inference-only NPU really train?
Yes, with VKORE. We ran LoRA training of a standard transformer block (attention plus feed-forward) across all 32 chips of one server.
- Accuracy: 0.9999 agreement with standard PyTorch training.
- Speed: about 170x the same server's CPU.
- Versus one NVIDIA B200: about 54% of its training throughput, on comparably priced hardware.
LoRA attaches a small correction layer to an existing model and tunes it for a task. It is the most common training method in enterprise practice. Getting it to run correctly on an NPU means an inference fleet can fine-tune during low-traffic hours instead of sitting idle.
Why does the ZTC decision engine pair well with NPUs?
A decision engine returns a judgment, not generated text. VIDRAFT's Darwin ZTC finishes a judgment in a single forward pass with zero output tokens. That is a large number of short, deterministic computations rather than long token-by-token generation, which maps cleanly onto a power-efficient NPU. On the S1MB public leaderboard, Darwin-27B-ZTC-v5 ranks first overall among 152 models, so putting a top decision engine on NPUs increases how many judgments the same power budget can serve.
The night-shift loop
Put the pieces together and one data center can run a cycle:
- Serve inference by day.
- Let the decision engine score the day's traffic and pick what to learn from.
- Let the same NPUs train at night.
The decision engine selects data, self-improvement trains the model, and the acceleration layer runs all of it on NPUs. The method is patent-pending and the internal details are not disclosed. Results will be published in a paper.
FAQ
Is VKORE tied to one chip vendor?
No. It is designed as a hardware-independent layer intended for data centers that mix GPUs and NPUs.
Does training on NPU lose accuracy?
In our measurement, LoRA training matched standard PyTorch at 0.9999 agreement.
What model and hardware were used?
Google's open Gemma 4 26B on a Korean NPU server with 8 cards (32 chips total).
How does this compare to a B200 on cost?
The NPU server reached about 54% of a single B200's training throughput on comparably priced hardware, while also being the fleet you already bought for inference.
Where can I read more?
VIDRAFT plans to publish the measurement methodology as a paper.
Further reading
- VIDRAFT Engineering Notes: Decision models and why forward-pass-only judgment scales
- VIDRAFT Engineering Notes: Running large models on-device without a GPU
Top comments (0)