A higher TFLOPS number looks faster. A pricier card has more memory. Neither tells you which one answers your users quicker. The biggest number on a GPU spec sheet almost never predicts real AI performance, because AI is not one workload. It is two, and they break on different things.
Ravi had two GPU spec sheets open, and both were misleading him. One card led on TFLOPS, so it looked faster. The other carried more memory and more bandwidth and cost more. He was building an app that serves a language model to users, and he could not tell from the numbers which card would answer a prompt quicker. He is not alone, and the answer is counterintuitive: the headline compute number is the wrong thing to optimize for, and a card with fewer TFLOPS often wins.
Here is how to compare GPUs for AI, the four numbers that decide it, the one piece of math that explains why the slower-looking card wins, and how to settle any matchup in about a minute instead of an afternoon of spec-sheet guessing.
AI is two jobs, not one
An AI model is a huge pile of numbers, called weights, arranged in layers. Running it means multiplying your input by those weights, over and over, which is why a GPU with thousands of parallel cores beats a CPU with a few fast ones. Modern GPUs add Tensor Cores built specifically for that matrix math, and when you see an AI performance figure, it is almost always the Tensor Cores doing the work.
The whole comparison hinges on one fork. There are two AI jobs, and they stress the GPU differently. Training teaches a model by running data through it and adjusting the weights, for days or weeks, and it is math-heavy from start to finish. Inference is using the finished model to answer a request, which happens millions of times a day every time someone prompts a chatbot. At scale, inference is where most GPU time and money go. Because they are different jobs, they are limited by different parts of the card. Miss that, and every comparison you make is wrong.
The four numbers that decide it
Ignore the marketing and a GPU's fitness for AI comes down to four measurable things. A good benchmark reports all four, because any one alone can mislead.
Compute, in TFLOPS, is the raw rate the Tensor Cores multiply. Higher compute finishes training faster. This is the number spec sheets lead with, and the one most often over-weighted. Memory capacity, in gigabytes of VRAM, is whether the model fits on the card at all. A model that does not fit either fails or spills to system memory, where performance collapses, so capacity is a hard gate. Memory bandwidth, in terabytes per second, is how fast weights travel from memory into the cores, and it is the number that quietly decides inference speed, the one buyers most often overlook. Precision support, FP16, FP8, FP4, is whether the card can represent each weight in fewer bits to shrink the model and roughly double throughput, but only if the hardware supports the format. Blackwell's native FP4 is an example older cards cannot use.
Why the biggest TFLOPS number lies
This is the part that trips up almost everyone. Picture a factory. The cores are workers who assemble products incredibly fast. The weights are raw materials in a warehouse, delivered by truck, the memory bus. During training there is so much math per delivery that the workers stay busy, so more workers, more TFLOPS, means more output. During inference, specifically generating text token by token, the math per delivery is tiny. The GPU reads the entire model's weights just to produce one token, does a little arithmetic, and waits for the next truckload. The workers sit idle. The factory is delivery-bound, not worker-bound.
That gives a rule precise enough to reason with: for token generation, tokens per second is roughly the memory bandwidth divided by the size of the model in memory. Add compute and nothing changes. Add bandwidth and it speeds up.
This explains the industry's favorite example. The H200 uses the same compute die as the H100, so their TFLOPS are identical, yet the H200 serves large models roughly 40 percent faster. The only thing that changed is memory bandwidth, from 3.35 to 4.8 terabytes per second. Rank them by compute and they look equal. In production they are not close. And it compounds: the KV cache, the running memory of a conversation, grows with every token and every user, so longer prompts and more traffic make a workload more memory-bound, not less.
Measure real work, don't trust the datasheet
If raw specs mislead, the fix is to measure real work. The industry standard is MLPerf, a suite that runs the same models on every accelerator so results are comparable, with inference tests refreshed through 2026.
A practical tool makes that usable. The MillionMiner GPU and AI benchmark scores every card on an AI Inference Index normalized to the RTX 3090 at 100, alongside VRAM, compute, and training throughput. The normalized index is the key move: instead of asking you to weigh terabytes against teraflops in your head, it expresses real inference performance as one relative number, so a card scoring 400 does about four times the inference work of the 3090 baseline. Capacity and compute sit beside it for the cases where they, not throughput, decide. It is the spec sheet corrected for how GPUs behave.
Read the benchmark through your workload
The right GPU is entirely a function of the job. Serving a language model to users is memory-bound: lead with the inference index and VRAM, confirm the model plus KV cache fits, then take the highest throughput you can afford. Training or fine-tuning is compute-bound: lead with TFLOPS and training throughput, and the interconnect if the job spans multiple GPUs, since NVLink moves data between cards at around 900 gigabytes per second while PCIe becomes the bottleneck the moment you split a model.
Running a large model on one card makes capacity the gate: a 70B model in full precision needs roughly 140 GB, which rules out most cards before speed even matters. Prototyping or budget work is where a consumer card like the RTX 5090 is strong value, until the index shows you have outgrown it.
Comparing two cards then takes about a minute. Name your workload first, training or inference, model size, single card or not, because that decides which column you read. Open the benchmark tool, pick any two GPUs, and put them side by side.
*Read the column that matches the job and ignore the rest: *
inference index for serving, TFLOPS and training throughput for training, VRAM for fit. Sanity-check that the model fits and weigh performance against price, power, and cooling. Then act, on the card's product page or, if you would rather not own and cool the hardware, on hosting. The wider questions of picking within a budget and how the flagships stack up are covered in the guide to choosing a GPU for AI and the H100 vs H200 vs B200 comparison.
The same trap catches Bitcoin miners
If this pattern feels familiar, it should. The mistake Ravi almost made with TFLOPS is the exact mistake buyers make with mining hardware and hashrate. The highest terahash number is not the most profitable miner, because profit is output minus the power it burns, and efficiency, not raw hashrate, decides the winner. The fix is identical: stop reading the headline spec and measure the real result. Our mining profit calculator ranks machines by actual daily profit at your electricity rate the same way the benchmark ranks GPUs by real inference, not by the number the box wants you to read. One lesson, two kinds of hardware: the biggest number on the sheet is marketing, and the metric that pays is the one you have to measure.
The bottom line
Ravi was serving a model, so he stopped reading the TFLOPS line and sorted by the inference index and VRAM. The card with the lower headline compute won on both, fit his model, and answered users faster for less money. The spec sheet would have sent him the wrong way.
Comparing GPUs for AI is not about the biggest number. It is about knowing which number your workload depends on, then measuring real performance instead of trusting a datasheet. Name the job, read the right column, and the choice is obvious. When you are ready to see the figures for your shortlist, run them through the benchmark tool and browse the cards in the AI hardware catalog. If power and cooling are the real constraint, that is a question of the site, not the silicon, and one MillionMiner is built to solve.
See the real inference numbers for any two of 78 GPUs, normalized to the RTX 3090 at 100: millionminer.com/gpu-ai-benchmarks.
Top comments (0)