Description: "Introducing Ruitong: the first public accuracy delta table for LLMs across CUDA, Ascend, and AMD. Same model, same weights, same prompt — different hardware gives different answers. Nobody publishes this number. We do."
tags: [llm, inference, hardware, benchmarks, gpu]
canonical_url: https://ruitong.io
cover_image: https://ruitong.io/matrix-banner.jpg
Everyone knows that moving an LLM across hardware changes the cost and the latency. What nobody talks about is that it also changes the answers.
The same model, the same weights, the same prompt, the same seed — different accelerators produce different output. This isn't a bug. Floating-point arithmetic is not associative, so different kernels reduce in a different order and the logprobs shift. It's expected, unavoidable, and almost never measured.
Until now.
The Problem
If you move inference from NVIDIA CUDA to Huawei Ascend, AMD MI300X, or even change precision (fp16 → bf16) on the same GPU, the output distribution shifts. The industry has good benchmarks for throughput and latency. Nobody publishes a reference number for accuracy equivalence.
Huawei's msprobe tells you operator-level tensor diffs. vllm-ascend's CI accepts 5% drift against a hardcoded YAML value with no GPU baseline. CAICT's national adaptation programme scores five dimensions — accuracy equivalence is not one of them. MLCommons publishes 99%/99.9%-of-FP32 for a handful of benchmark models.
Every team migrating inference independently discovers the same drift. And independently decides what tolerance is acceptable. That's a lot of duplicated effort.
What Ruitong Is
Ruitong is a cross-accelerator equivalence reference — a dated, versioned, reproducible accuracy delta table that answers one question: how much does the output actually change?
| Model | Reference | Candidate | Metric | Status |
|---|---|---|---|---|
| Qwen3-8B | A100 fp16 | A100 bf16 | top-k max Δlogprob | ✅ Collected |
| Qwen3-8B | A100 bf16 | Ascend 910B bf16 | top-1 agreement | 🔄 In progress |
| Llama-3.1-8B | A100 bf16 | MI300X bf16 | prob mass shift | 🔄 Planned |
The three rows that matter most
- Same hardware, different precision (fp16 vs bf16) — establishes the noise floor. Everything else is read against it.
- Different vendor, same precision (NVIDIA vs Ascend/AMD) — the actual question a migrating customer has.
- Same hardware, different batch size — published research shows bf16 accuracy varying up to 9% from GPU count and batch size alone.
Key Finding So Far
On A100, across fp16 vs bf16 — same hardware, same vendor, different precision — 18 out of 61 prompts produced different output text. All calibrated equivalence gates still passed.
This is the noise floor. The baseline against which every cross-vendor delta must be evaluated.
Beyond Token-Level Metrics
We run calibrated equivalence gates — end-to-end tests that check whether the candidate output is functionally equivalent to the reference (same classification, same choice, same numeric output within tolerance). Our finding: even when per-token metrics show drift, calibrated gates often pass. This has implications for how inference benchmarks should be designed.
The Methodology
- Models: Qwen3-8B (primary), Llama-3.1-8B, DeepSeek variants
- Hardware: NVIDIA A100 80GB (reference), Ascend 910B 64GB, Ascend 910C 128GB, AMD MI300X
- Precisions: fp16, bf16, W8A8
- Metrics: top-k max Δlogprob, prob mass shift, top-1 agreement, top-5 set agreement, calibrated equivalence gates
- Tooling: Open-source Python pipeline, reproducible via command line
Why This Matters
Every company migrating inference from CUDA to Ascend (or any accelerator) needs to answer one question: "Does the model still work correctly?"
Without a published reference, every team independently discovers the same drift — and independently decides what tolerance is acceptable. Ruitong publishes the reference so the industry has a single number to cite, argue with, and improve against.
Current Status
| Component | Status |
|---|---|
| Benchmark spec | ✅ Complete |
| Calibration harness | ✅ Complete |
| A100 fp16/bf16 baseline | ✅ Collected |
| Ascend data collection | 🔄 In progress |
| Public accuracy table | 📝 Draft |
| Site | ✅ Live at ruitong.io |
| GitHub | ✅ github.com/loopeywho/ruitong |
Contribute
The method and tooling are open. If you run inference on different hardware and want your bench to have a comparable number, this is the reference. Pull requests, issue reports, and hardware contributions welcome.
瑞通 Ruitong — auspicious flow between ecosystems.
Top comments (0)