TL;DR
Yes, an inference-only NPU can pretrain a language model, if the training stack is built for it. We pretrained VKORE-325M-NPU-Base (325.6M parameters) from random initialization on the Rebellions ATOM-Max, a commercially available Korean NPU released in 2025 and designed for inference. We trained the identical model on a B200 GPU as a reference. On 1,024 windows of text neither model ever saw, the NPU model scores 0.7993 and the GPU model 0.7994 (cross entropy per byte). Both weights and the evaluation script are public.
Why is "an inference-only NPU cannot train" the default assumption?
Inference accelerators ship with a compiler and runtime built to execute a forward pass of a model trained elsewhere. Pretraining needs much more: a backward pass through every layer, optimizer state, and numerically well-behaved updates for tens of thousands of steps. None of that comes with an inference stack, so the common conclusion is that these chips are for serving only.
That conclusion matters commercially. Inference NPUs are cheaper and draw less power than training GPUs, and many organizations already own them. If they can also train, one class of hardware can do both jobs.
What exactly did we train?
| Item | Value |
|---|---|
| Model | VKORE-325M-NPU-Base, decoder-only Transformer |
| Parameters | 325.6M (20 layers, width 1024, FFN 3584, GQA 8 query / 2 KV heads) |
| Vocabulary | 256, byte level (token id = byte value) |
| Data | 5.0B byte tokens of FineWeb-Edu |
| Run | 76,292 steps, about four days, random initialization |
| Pretrained weights used | none |
| Hardware | Rebellions ATOM-Max NPU (2025) |
The released checkpoint is a standard LlamaForCausalLM, so it loads with plain transformers.
How do you prove "GPU quality" instead of just claiming it?
You need a reference that differs only in the hardware. We trained the same model twice with identical initialization, data, batch and learning-rate schedule: once on the NPU, once on a single B200 GPU.
Do the training curves agree?
They overlap for the whole run. Both fall from about 5.8 to about 0.80. Local wiggles differ only because each run draws its random windows in a different order.
What happens on text the model has never seen?
This is the test that matters. The training corpus was built from the first two shards of FineWeb-Edu sample/10BT. We evaluated both published models on 1,024 random windows from shard 013, which was never part of training.
| Model | Loss, unseen text | Bits per byte |
|---|---|---|
| Rebellions ATOM-Max NPU | 0.7993 | 1.1531 |
| B200 GPU reference | 0.7994 | 1.1533 |
Because both models saw identical windows, we can compare them pair by pair. The mean difference is -0.0001 (NPU minus GPU) with a standard error of 0.0007, and the NPU model is lower on 50.1% of windows. At this resolution the two are indistinguishable.
How does this compare with VKORE-46M?
VKORE-46M-NPU proved the backward pass could run at all. VKORE-325M is the step to real pretraining.
The hard part between the two was not the backward pass. It was keeping a long run stable to the end. That stability technique was developed in house and is under patent application, so we do not describe it here.
Can I just buy the NPU and train?
No. The ATOM-Max and its standard software are built for inference. Training on it depends on our own training stack and stability technique. Buying the chip does not by itself enable training.
How do I reproduce the result?
from transformers import AutoModelForCausalLM
npu = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base")
gpu = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base", subfolder="b200_reference")
evaluate_unseen.py in the repository downloads shard 013, draws the same 1,024 windows and prints both losses and the paired difference. It needs only transformers, numpy and pyarrow.
FAQ
Is this the first language model pretrained on an inference-only NPU?
It is the second model in our own VKORE line, after VKORE-46M-NPU, and the first one at real pretraining scale (5B tokens) with a GPU twin for comparison.
Which NPU was used?
The Rebellions ATOM-Max, a commercially available NPU released in 2025 and designed for inference.
Is the NPU model as good as the GPU model?
On unseen FineWeb-Edu text, yes: 0.7993 vs 0.7994 loss, a paired difference of -0.0001 with a standard error of 0.0007.
Is VKORE-325M a chat model?
No. It is a 325M base model trained on 5B tokens, released as a training-hardware result.
Is the training method open?
The weights, data recipe and evaluation are open. The training stack and stability technique are in house and under patent application.
Top comments (0)