Comparing AI inference accelerators is harder than it looks, and for a specific reason. The industry benchmark everyone points to does not contain the silicon from the two companies most likely to compete with NVIDIA on cost.
That gap turned out to be the interesting finding. Once you build a comparison that survives it, the picture that emerges is less about which chip is fastest and more about which axis you chose to divide by.
The outcome first
Four accelerators, one benchmark cohort, one day of pricing, four ways of looking at the same question.
Figure 1: Performance against price
Each dot is one chip. Moving right is more tokens per second per GPU. Moving down is a lower hourly price, so the lower right corner is where a chip is both fast and cheap. Only one chip lands in that shaded region, and it is not the flagship.
The B300 and the MI355X sit almost on top of each other on the horizontal axis, 0.4 percent apart on speed. Vertically they are $5.26 apart per GPU-hour. This is the single most important thing the research turned up: the two chips are interchangeable on raw performance and completely different on cost.
Figure 2: The ranking inverts when you divide by price
Same two chips, same cohort, two currencies. On raw speed the B300 is first. On tokens per dollar the MI355X is first, by a factor of three. Whichever number a comparison starts from, it decides the answer, so the currency has to be stated before the winner.
Figure 3: No chip wins every axis
Each cell is a chip scored out of 100 on one axis, with the row weight in brackets. The filled cell in each row is the winner on that axis. Every chip wins something and loses something: the MI355X takes throughput and memory per dollar, the B300 takes energy, the GB200 takes scale-up domain. This is why a single weighted score is a summary and not a verdict.
Figure 4: Where the differentiation actually lives
Across these four chips the price axis separates them by 4.5 times. The energy axis separates them by 1.3 times. A comparison that only argues about efficiency is arguing about the axis that barely distinguishes these parts.
This is the environmental finding with a caveat attached. Joules per token still matters for the grid and for the bill, and the ordering on that axis is real. But between four chips of this class, the decision gets made on price and on scale-up domain, not on watts.
The problem with chip comparisons
A vendor's own benchmark result is a measurement of one system, on one model, under one scenario, at one moment. It is a real number. It is also not a comparison.
Read a chip launch and you will find FLOPS, memory bandwidth, memory capacity, TDP, and a throughput chart, sometimes on the same page. These are five different currencies. Dividing one by another changes what the number means, and most comparison articles do it silently.
There is a second problem. The industry benchmark that exists to settle these arguments, MLPerf Inference, ran 30 organisations and 120 systems in its September 2026 round. Of the 492 published result rows, 459 name an accelerator. Of those, NVIDIA accounts for 335, AMD for 105, Intel for 19. Google TPU, AWS Trainium and Intel Gaudi do not appear at all in the latest round.
So the framework below does two things. It fixes the axes so a comparison is valid. And it says plainly which chips cannot be scored on a given axis, rather than leaving the gap to be filled with a vendor's marketing number.
The five axes
Axis 1: Throughput per dollar
Raw speed is the axis everyone quotes, and it is close to useless on its own, because speed without price says nothing about cost. Divide it by what you actually pay per hour.
tokens/second per accelerator / price per GPU-hour
This is the axis that decides almost every practical comparison, because inference cost is a function of both. Two chips with identical silicon performance can differ threefold in tokens per dollar on price alone.
Source: MLPerf result divided by accelerator count, divided by the provider's hourly list price. Get the token rate from a matched MLPerf cohort. Get the price from the provider, not from a blog.
Axis 2: Memory per dollar
Decoding is bound by how fast weights and the KV cache move, not by how fast the tensor cores multiply. Peak FLOPS mostly buy you prefill. Bandwidth buys you decode.
memory per chip in GB / price per GPU-hour
Memory per dollar also decides whether the model fits at all, and how much context and concurrency you can afford on top of it. For a 70B model at FP8, roughly 70 GB of weights sit on the card before any KV cache.
Source: Vendor specification sheet for memory and bandwidth, provider list price for cost.
Axis 3: Energy per token
Every generation adds more compute per card, and every generation adds more watts. The number that matters for a fleet is not watts per card, it is joules per output token, because that is what you pay for and what the grid sees.
board power in watts / tokens per second per accelerator
Turn it into kWh per million tokens to make it comparable to a utility bill. One joule per token is 0.278 kWh per million tokens.
Source: Vendor board power rating (TBP or TGP) divided by measured throughput. This is an upper-bound estimate at rated board power, not a wall measurement. A true wall measurement needs a certified power analyser at the socket, which is what MLPerf Power does and why so few rows carry it.
Axis 4: Scale-up domain
The number of accelerators that share memory at full interconnect speed sets the largest model you can serve without crossing a network hop. A network hop changes your cost and your latency, and it happens silently in most comparisons.
accelerators per scale-up domain x memory per accelerator = memory in the domain
An 8-GPU node with 270 GB cards gives you 2.2 TB of fast memory. A 72-GPU rack with 186 GB cards gives you 13.4 TB. For a 405B model at FP8, that difference decides whether you serve it as one model or as a sharded one.
Source: Vendor platform documentation for the scale-up domain size and memory per accelerator.
Axis 5: Software readiness
Two chips with the same silicon behave differently because the software stacks differ in maturity, in which models they support, and in which serving paths are production-grade. This axis is the one most often skipped and most often regretted.
Three things to check. Does the serving framework support your model at production quality, or only in beta. Does the vendor publish benchmark results, because a vendor who does not submit cannot be compared on any axis. And are the documented performance caveats written down somewhere you can find them.
Source: Release notes and model support matrices. The score below is a 0 to 100 judgement, and the specific reason is printed next to every chip.
The rule that keeps the axes honest: score each axis only inside a cohort with the same benchmark, the same model, the same scenario, the same division, the same availability class and the same unit. If the cohorts do not match, the number is not comparable, and the honest output is "not measured" rather than a rank.
A worked example on four chips
All four chips below are scored on the same cohort: MLPerf Inference v6.1, Closed division, Available category, Datacenter, gpt-oss-120b, Server scenario. The token rate is the best submitted per-accelerator result in that cohort. Weights are throughput per dollar 30 percent, memory per dollar 25 percent, energy per token 20 percent, scale-up domain 15 percent, software readiness 10 percent.
| Rank | Chip | Tokens/s per GPU | Price per GPU-hour | Tokens/s per $ | GB per $ | Joules per token | kWh per Mtok | Scale-up domain | Software | Score |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | AMD Instinct MI355X | 14,155 | $2.59 | 5,465 | 111 | 0.099 | 0.028 | 8 accelerators, 2.3 TB | 85 | 82.3 |
| 2 | NVIDIA GB200 NVL72 | 12,774 | $10.50 | 1,217 | 18 | 0.094 | 0.026 | 72 accelerators, 13.4 TB | 100 | 52.1 |
| 3 | NVIDIA B300 | 14,211 | $7.85 | 1,810 | 34 | 0.077 | 0.022 | 8 accelerators, 2.2 TB | 100 | 50.4 |
| 4 | NVIDIA B200 | 11,276 | $6.69 | 1,685 | 27 | 0.089 | 0.025 | 8 accelerators, 1.4 TB | 100 | 42.8 |
AMD Instinct MI355X matches NVIDIA's flagship on silicon, sells at a third of the hourly rate, and carries the same 288 GB of HBM3E at the same 8 TB/s. The scale-up domain is 8 accelerators, so a 405B model does not fit in one domain. Throughput: MLPerf v6.1 best per-accelerator result in cohort. Price: Vultr API, $20.72/hr for an 8-accelerator node, $2.59 per GPU-hour. Specs: AMD MI355X product page, 1400W TBP.
NVIDIA GB200 NVL72 has the strongest scale-up domain in the set at 72 accelerators and 13.4 TB of memory, so the largest model you can serve in one domain. The lowest memory per dollar in the set. Throughput: MLPerf v6.1 best per-accelerator result in cohort. Price: CoreWeave inference single-GPU price for GB200 NVL72, $10.50 per GPU-hour. Specs: 186 GB per GPU, 1200W TGP per GPU.
NVIDIA B300 is the fastest chip in the cohort on tokens per second per GPU and the most efficient on joules per token, at 0.077 J per token. It ranks third only because of price and an 8-GPU scale-up domain. Throughput: MLPerf v6.1 best per-accelerator result in cohort. Price: Nebius list price, 9 September 2026. Specs: 270 GB HBM3E, 8 TB/s, 1100W TGP per GPU.
NVIDIA B200 is the previous generation, and still competitive per dollar because its price fell further than its throughput. 180 GB per card and a 1.4 TB domain. Throughput: MLPerf v6.1 best per-accelerator result in cohort. Price: Lambda list price, 9 September 2026. Specs: 180 GB HBM3e, 8 TB/s, 1000W TGP per GPU.
The one comparison that carries the argument
On tokens per second per GPU, the B300 and the MI355X are 0.4 percent apart, 14,211 against 14,155. On tokens per dollar, the MI355X is at 5,465 and the B300 at 1,810, a gap of 202 percent.
Same silicon performance. Three times the money. That gap exists only because one of the two chips is priced at $2.59 per GPU-hour and the other at $7.85. Any comparison that stops at throughput will never surface it, which is exactly why throughput needs a price attached before it means anything.
The chips the benchmark cannot score
The five axes need comparable data, and for several important chips that data does not exist in the standard benchmark. Saying so is part of the framework.
- Google TPU Ironwood and TPU v7. No MLPerf inference submission in v6.0 or v6.1. Google submitted in both rounds, on NVIDIA GB200 and GB300 hardware. Third-party testing puts TPU v7 ahead on cost per million tokens at high concurrency on some models, and behind on FP4 workloads, because TPU v7 has no native FP4. Scored on axes 2, 3, 4 and 5. Not scored on axis 1 from the standard benchmark.
- AWS Trainium3 and Trainium2. No MLPerf inference submission in any round through v6.1. The vLLM Neuron plugin is still beta, and DeepSeek V3 and R1 are not production models on it. Priced at $2.235 per chip-hour on AWS Capacity Blocks. Scored on axes 2, 3, 4 and 5. Not scored on axis 1.
- Intel Gaudi 3. No MLPerf inference submission in any round through v6.1. Intel wrote off $1.3 billion of Gaudi inventory across 2024 and 2025 and its 2026 earnings calls do not mention the product. Not scored on axis 1 or 5.
- AMD MI455X Helios and NVIDIA Vera Rubin. Both shipping or ramping with no available MLPerf inference result. Neither has a public hourly price. Not scored on axis 1.
- NVIDIA GB300, AMD MI350X, MI350P, MI300X and RTX PRO 6000. All have MLPerf throughput results in the cohort but no public hourly price from any provider, so they cannot be placed on axis 1.
A chip without a benchmark result is not a slow chip. It is an unmeasured chip. Any framework that ranks it anyway has quietly substituted a vendor claim for a peer-reviewed number, and the reader cannot see that substitution.
How to use the framework
- Pick the workload before the chip. A 70B chat model and a 405B reasoning model do not want the same hardware. The scale-up domain axis decides that before any price comparison happens.
- Fix the cohort. One benchmark, one model, one scenario, one division, one availability class. Take the token rate from that cohort only.
- Divide by the accelerator count, and say you did. A per-accelerator figure is descriptive bookkeeping, not a deployable design. The full system, its host processors, its network and its software are part of the result.
- Attach the price from the provider's own page on the day you check it. Prices move, and cloud prices move more than hardware prices. Note the date and the commitment term, because a three-year Google commitment and on-demand Google pricing differ by more than 2x on the same silicon.
- Report what is missing. A chip with no submission, no public price, or no production-grade serving path gets marked as not measured. That is the honest output and it is more useful than a false rank.
- Re-run it each round. MLPerf publishes twice a year. The cohort changes, the software moves, and the gap between the price and the silicon moves with it. In the v6.1 round, software-only changes were worth 8.9 to 37.9 percent per GPU on the same hardware.
References
- MLCommons, MLPerf Inference v6.1 results repository (30 submitters, 120 systems, published 16 September 2026), Closed division, Available category, Datacenter suite, gpt-oss-120b, Server scenario: https://github.com/mlcommons/inference_results_v6.1
- MLCommons, MLPerf Inference v6.0 results repository, used to confirm no Google TPU, AWS Trainium or Intel Gaudi accelerator appears in either round: https://github.com/mlcommons/inference_results_v6.0
- Miro Hodak and Frank Han, "Where the Industry Is Investing: A Look at MLPerf Inference v6.1", MLCommons, 17 September 2026: https://mlcommons.org/2026/09/chairs-mlperf-inference-v6-1/
- MLCommons, MLPerf Inference Datacenter benchmark rules, including the Closed and Open division definitions and the power measurement method: https://mlcommons.org/benchmarks/inference-datacenter/
- MLCommons, "MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results", 16 September 2026: https://mlcommons.org/2026/09/mlperf-inference-v6-1-results/
- AMD, Instinct MI355X product specifications (288 GB HBM3E, 8 TB/s, 1400W TBP): https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html
- AMD, Instinct MI350 Series microarchitecture reference (MI350X 1000W, MI355X 1400W, MI350P 600W configurable, 288 GB HBM3E, 8 TB/s): https://rocm.docs.amd.com/en/latest/reference/gpu-arch/mi350.html
- AMD, Instinct MI300X product specifications (192 GB HBM3, 5.325 TB/s, 750W TBP): https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html
- NVIDIA, RTX PRO 6000 Blackwell Server Edition specifications (96 GB GDDR7, 1597 GB/s, up to 600W): https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/
- Google Cloud, Cloud TPU pricing (Ironwood $12.00 per chip-hour on demand in us-central1, $8.40 one-year commitment, $5.40 three-year commitment, checked 10 November 2026): https://cloud.google.com/tpu/pricing
- Amazon Web Services, Amazon EC2 Capacity Blocks pricing (trn2.48xlarge $35.7608 per hour for 16 AWS Trainium2 chips, $2.235 per chip-hour, checked 10 November 2026): https://aws.amazon.com/ec2/capacityblocks/pricing/
- CoreWeave, GPU and inference pricing (GB200 NVL72 inference single-GPU price $10.50 per hour, checked 10 November 2026): https://www.coreweave.com/pricing
- SemiAnalysis, InferenceX performance-per-dollar comparisons for TPU v7 versus MI355X, including the native FP4 gap on TPU v7: https://inferencex.semianalysis.com/compare-per-dollar
- Tesla, "MLPerf Power: Addressing the AI Power Wall", IEEE Micro, 2026, on samples per joule and tokens per joule methodology and the 1,841-measurement corpus from 60 systems: https://doi.org/10.1109/mm.2026.3700126
- Emad Barsoum, "Energy per Token: Measuring LLM Inference Efficiency", GPUSmith, on the four energy-measurement boundaries and why joules per token figures are not interchangeable: https://gpusmith.com/articles/en/energy-per-token-measurement-mlperf-power
Notes on method
Throughput figures come from the published MLPerf Inference v6.1 results file, filtered to the Closed division, Available category, Datacenter suite, gpt-oss-120b model, Server scenario, with each system result divided by its accelerator count. The same filter was applied to the v6.0 results file to confirm that no Google TPU, AWS Trainium or Intel Gaudi accelerator appears in either round. Counts of rows by accelerator vendor were computed directly from those two files.
Memory, bandwidth and board power come from vendor product pages and from the accelerator names recorded in the MLPerf submissions themselves. Prices come from the providers' own pricing pages and APIs on 10 November 2026, except where the page shows a September 2026 date. Software readiness scores are judgements against vendor release notes and model support matrices, and the reason is printed next to each chip.
Joules per token is an upper-bound estimate at rated board power, not a wall measurement. No row in the v6.1 or v6.0 results file carries a certified wall-power measurement, so a true energy comparison needs your own metering.
All derived figures are arithmetic on the sources above. Nothing was measured directly.
Top comments (0)