Three NVIDIA GPUs, three completely different bottlenecks. Most of the money wasted on AI hardware comes from treating them as a simple "faster" ladder. They are not.
Most people read NVIDIA's data center lineup as a straight line: the H100 is fast, the H200 is faster, the B200 is fastest, so buy the newest thing the budget allows. That instinct is wrong, and it is one of the most expensive mistakes in AI infrastructure. These three chips do not clear the same bottleneck harder. They clear three different bottlenecks, and the right choice depends entirely on which one your workload hits first.
Here is what each one is, the numbers that matter, and a simple way to pick the right one instead of overpaying for a spec you will never use.
The H100: the workhorse that built the boom
The H100, on NVIDIA's Hopper architecture, is the chip the current AI wave was trained and served on. It carries 80 GB of HBM3 memory at 3.35 terabytes per second of bandwidth, and its fourth-generation Tensor Cores deliver about 989 teraflops of dense FP16 and roughly double that in FP8. It draws 700 watts, links to other H100s over NVLink at 900 gigabytes per second, and still sets the baseline every newer chip is measured against.
Its compute was never really the problem. Its ceiling is capacity. Eighty gigabytes fits a lot, but a 70-billion-parameter model in full precision needs around 140 GB, which means two H100s and the overhead of splitting a model across them. As models grew, that 80 GB wall, not the math, became the thing people kept hitting.
The H200: not a new chip, a bigger memory system
Here is the fact that reframes the entire lineup: the H200 uses the exact same compute die as the H100. Identical Tensor Cores, identical FP16 and FP8 throughput. On raw compute, they are the same chip.
What NVIDIA changed is the memory. The H200 carries 141 GB of faster HBM3e at 4.8 terabytes per second, which is 76 percent more capacity and 43 percent more bandwidth than the H100, in the same 700-watt envelope and the same socket. In most H100 servers it is close to a drop-in upgrade.
Why build a "new" GPU that only touches memory? Because the workload that dominates AI spending, inference, is limited by memory bandwidth, not compute. Generating each token means reading the model's weights out of memory, doing a little math, and repeating; most of the time is spent moving data, not calculating. The H100 already had enough compute for that job. It did not have enough memory speed. The H200 fixes the part that was the real bottleneck, and the result is roughly 40 percent faster inference on large models for zero extra power, a gain that tracks its 43 percent bandwidth increase almost exactly.
The B200: a real generational jump, and a different class of machine
The B200 is where "newer" finally means fundamentally different. It is built on NVIDIA's new Blackwell architecture, and it is not one die but two, 208 billion transistors joined into a single package by a 10 terabyte-per-second link. It carries 192 GB of HBM3e at 8 terabytes per second, about 2.4 times the H100's bandwidth, and its fifth-generation Tensor Cores add native FP4, a four-bit precision mode that roughly doubles inference throughput again for models that tolerate it.
The numbers are a leap, not a memory bump. Around 2,250 teraflops of dense FP16, 4,500 in FP8, and 9,000 in FP4, alongside NVLink 5 at 1.8 terabytes per second, double the Hopper interconnect. In practice a single B200 serves large models at roughly 2.3 to 2.5 times an H100's token rate, trains about twice as fast, and with FP4 can push inference throughput far higher still. One B200 fits a 70-billion-parameter model that used to need two or three H100s.
But it is a different class of machine, and that is the catch. The B200 draws 1,000 watts, uses a new SXM6 baseboard rather than a card that drops into an existing server, and cannot go into most H100-era, air-cooled infrastructure without new power delivery and cooling. It is also where the lineup stops being about single GPUs at all: the B200 is the building block of the GB200 rack, which packs 72 of them with 36 Grace CPUs into one liquid-cooled system that behaves like a single enormous GPU. Demand has run well ahead of supply, with Blackwell reportedly backlogged into mid-2026.
The one idea that makes the choice obvious
Forget the ladder. The right way to choose is to find the bottleneck your workload hits first, then buy the chip that clears it. Ask three questions, in this order.
First, does your model fit? Capacity is a hard gate. If the model plus its working memory does not fit on the card at your precision, nothing else matters. A 70-billion-parameter model needs roughly 140 GB in full precision, which rules out a single H100 before speed is even a question.
Second, is your workload memory-bandwidth-bound? Almost all inference is. If you are serving a model and it fits, bandwidth decides your token speed, which is exactly why the H200 beats the H100 at the same model size despite identical compute.
Third, do you need more compute or FP4? This is the last question, not the first. Heavy training, very large models, or high-traffic FP4 serving are what justify a B200. If your job is memory-bound, paying for Blackwell's compute buys you a number you will not use.
The most expensive mistake in this whole category is buying for peak compute when the workload was starved for memory the entire time. Match the chip to the bottleneck, not to the top line on the spec sheet.
What the numbers look like
On real inference, serving a Llama-class model in FP8, a single H100 generates on the order of 115 to 135 tokens per second, an H200 around 160 to 185, and a B200 roughly 300 to 360. The H200's lead over the H100 tracks its bandwidth advantage; the B200's lead tracks its own, plus a new architecture and FP4 on top.
Price and availability tell the rest. The H100 has fallen sharply as supply caught up, and is now the cheap, proven option. The H200 sits in the sweet spot for memory-bound inference: more capacity and speed for the same power. The B200 costs several times more per hour and is hard to get, but for the largest models the right measure is cost per result, not cost per hour, and there it can win decisively. For anything that fits comfortably on Hopper, it usually does not.
The part nobody puts on the spec sheet: power
There is a reason large GPU deployments stall, and it is almost never the chips. It is power and cooling. An H100 or H200 at 700 watts is demanding; a B200 at 1,000 watts is more so; and a full GB200 rack pulls well over 100 kilowatts and requires liquid cooling. Securing that much power, at a workable price, in a facility built to move that much heat, is the hard part of AI infrastructure, and the part that quietly decides whether a build is viable at all.
This is the same lesson that has always governed mining, where the machine is the easy purchase and the power and the site are the real constraint. Whether you run Hopper or Blackwell, the silicon is available to anyone with a purchase order. Industrial power and the cooling to match are not, which is exactly the problem MillionMiner's data-center and AI hosting side is built to solve.
How to compare them for your workload
Spec sheets flatter the newest chip. The only comparison that matters is real performance on the job you are running, which is why it helps to look at measured throughput rather than peak teraflops. The MillionMiner GPU and AI benchmark scores every card, including the H100, H200, and B200, on real inference performance rather than the number on the box, alongside memory capacity and compute, so you can see which one clears your bottleneck before spending anything. Browse the cards themselves in the AI hardware catalog, and if you are choosing within a budget or for a specific use case, the guide to choosing a GPU for AI walks through it.
It is the same trap we see on the mining side, where the highest hashrate is not the most profitable machine, and only real, measured output tells the truth. The profit calculator applies that logic to mining hardware the way the benchmark applies it to GPUs: measure the result, not the headline number.
The bottom line
The H100, H200, and B200 are not three rungs on one ladder. The H100 is the proven workhorse with an 80 GB ceiling. The H200 is the same chip with a much bigger, faster memory system, built for the memory-bound reality of inference. The B200 is a real generational jump and a new class of machine, with the power and cooling demands to match.
Choose by bottleneck, not by generation. Find the resource your workload runs out of first, capacity, then bandwidth, then compute, and buy the chip that clears it. The newest GPU is only the best one if the thing it does better is the thing you needed.
Frequently asked questions
What is the real difference between the H100 and the H200?
Only the memory. The H200 uses the identical compute die as the H100, so their FP16 and FP8 throughput are exactly the same. The H200 adds 141 GB of HBM3e at 4.8 TB/s, versus the H100's 80 GB at 3.35 TB/s, for 76 percent more capacity and 43 percent more bandwidth in the same 700-watt envelope. Because inference is limited by memory bandwidth, that makes the H200 roughly 40 percent faster at serving large models, despite no change in compute.
Is the B200 just a faster H200?
No. The B200 is a full architecture change, Blackwell rather than Hopper. It uses a dual-die design with 208 billion transistors, 192 GB of memory at 8 TB/s, native FP4 precision, and NVLink 5 at 1.8 TB/s, and it draws 1,000 watts. It delivers roughly 2 to 2.5 times an H100's performance, and more with FP4, but it needs new baseboards, power, and cooling, so it is not a drop-in upgrade the way the H200 is.
Which GPU should I use for LLM inference?
Start with whether the model fits. If it fits in 80 GB at your precision and batches are healthy, the H100 is the cheapest option. If it fits but decode is memory-bandwidth-bound, which most inference is, the H200 is faster at the same size. If the model exceeds 141 GB, or you need FP4 throughput for high-traffic serving, you need a B200. Capacity first, then bandwidth, then compute.
Why does memory bandwidth matter more than TFLOPS for AI?
Because generating text is memory-bound. To produce each token, the GPU reads the entire model's weights from memory, does a small amount of math, and repeats, so token speed scales with how fast weights move, not with peak compute. This is why the H200 beats the H100 on identical compute, and why buying a GPU for its headline teraflops often means paying for performance the workload cannot use.
Do I need liquid cooling for these GPUs?
The H100 and H200 at 700 watts run in well-designed air-cooled facilities. The B200 at 1,000 watts pushes most air-cooled infrastructure past its limits, and rack-scale Blackwell such as the GB200 NVL72, which draws well over 100 kilowatts per rack, requires liquid cooling. Power availability and cooling capacity, not the GPUs themselves, are usually the real constraint on large deployments.
Sources and image credit
Specifications and pricing current as of mid-2026, from NVIDIA documentation and multiple independent benchmark and cloud-provider analyses; memory, bandwidth, and throughput figures are widely corroborated. Real-world token rates are representative of Llama-class FP8 serving and vary by model, batch size, and software. Hero image graded to brand navy; credit to be added on publish.
The only comparison that matters is real throughput, not peak teraflops. See the H100, H200, and B200 scored on real inference at millionminer.com/gpu-ai-benchmarks, and browse the cards in the AI hardware catalog.
Specs are mid-2026 accurate and move fast; refresh before publishing on major NVIDIA releases.
Top comments (0)