Disclosure: the case study below is VIDRAFT's own result. VIDRAFT is a technology partner of QuantID. This post reviews the measurement design; it is not an independent verification.
TL;DR
To claim two training runs reached the same quality, you need two things: an evaluation set the models truly never saw, and a paired comparison on identical inputs. The case here is VIDRAFT's VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and, as a reference, on a B200 GPU. On 1,024 unseen windows the losses are 0.7993 and 0.7994; the paired difference is -0.0001 with a standard error of 0.0007.
What makes a fair comparison between two runs?
Hold everything fixed except the variable you test. In this case the model, initialization, data, batch and learning-rate schedule were identical, and only the hardware differed.
Matching training curves are encouraging but not proof. Training loss is measured on data the model is fitting.
Is "held out" really held out?
This case contains a useful lesson. The first evaluation used 256 fixed windows from the end of the training file and was called "held out". But training sampled random windows from the whole file, about 0.72 passes over it, so those windows were partly seen during training. The comparison between the two runs was still fair, because both saw the same file, but the label was wrong.
The fix was to evaluate on a shard of the source dataset that was never part of the training corpus. The training file came from FineWeb-Edu sample/10BT shards 000 and 001; the new evaluation used shard 013.
Check: before calling a set held out, read the training sampler, not only the data split. Random-window samplers cover the whole file.
Why compare windows in pairs instead of averages?
Loss varies far more between text windows than between two good models. If you compare two averages, that window-to-window variation dominates the error bar.
Evaluating both models on identical windows and taking the per-window difference removes it. Here the per-model standard error is about 0.0049, while the paired standard error is 0.0007, about seven times tighter.
| Quantity | Value |
|---|---|
| NPU loss, unseen (1,024 windows) | 0.7993 |
| GPU loss, unseen | 0.7994 |
| Paired mean difference (NPU minus GPU) | -0.0001 |
| Paired standard error | 0.0007 |
| Windows where NPU is lower | 50.1% |
The honest reading is "no detectable difference at a resolution of about 0.0014 (two standard errors)". That is a stronger statement than "the averages look close", and a weaker one than "identical".
What else should a reader check?
- Reproducibility. Both weights and the evaluation script are public, so the paired test can be rerun.
- Scope. The result concerns a 325M base model on 5B tokens. It says nothing about larger scales.
- What is not measured. Speed and cost per run are separate questions. The case also notes that the NPU alone does not enable training; it depends on the developer's own training stack.
FAQ
Why bits per byte? With a byte-level vocabulary, loss per byte divided by ln 2 gives bits per byte (1.1531 vs 1.1533 here), which is comparable across tokenizers.
Is 1,024 windows enough? For a paired test with this variance, yes: the standard error of the difference is 0.0007.
Could a single seed hide differences? Possibly. A stronger design repeats each run with several seeds; this case reports one run per hardware.
Who measured this? VIDRAFT measured its own models. This post reviews the design.
Further reading
- Source report: Pretraining a 325M Language Model from Zero on an Inference-Only NPU
- Data and scripts: FINAL-Bench/VKORE-325M-NPU-Base
Top comments (0)