H100 vs H200: which GPU should you use for LLM training?

For most LLM training, use the H100: the NVIDIA H200 has exactly the same BF16 and FP8 compute, so a compute-bound job runs at about the same speed on both. Nodus lists the H100 SXM at $2.60/hr and the H200 SXM at $3.593/hr, so the H200 must run 1.38 times faster to cost less per finished job. Pick the H200 when your model or batch does not fit in 80 GB, or for long-context serving, where its memory bandwidth does the work.

The H200 is an H100 with a bigger, faster memory system. That makes the choice simpler than most GPU comparisons: you are not buying speed, you are buying room. The rest of this post shows when room is worth 38% more per hour, with a priced 100-hour run and the memory math for full fine-tuning.

What is actually different between the H100 and the H200?

Only the memory. Both are Hopper GPUs with the same Tensor Cores, and NVIDIA's own spec tables list the same peak math for the SXM versions.

	H100 SXM	H200 SXM	H200 / H100
BF16 Tensor Core (sparse)	1,979 TFLOPS	1,979 TFLOPS	1.00x
FP8 Tensor Core (sparse)	3,958 TFLOPS	3,958 TFLOPS	1.00x
GPU memory	80 GB HBM3	141 GB HBM3e	1.76x
Memory bandwidth	3.35 TB/s	4.8 TB/s	1.43x
NVLink	900 GB/s	900 GB/s	1.00x
Max power	up to 700 W	up to 700 W	1.00x
Listed on Nodus	$2.60/hr	$3.593/hr	1.38x

Specs from NVIDIA's H100 and H200 pages (the H200 figures are marked preliminary there). Prices checked November 6, 2026 on the Nodus pricing page.

The H200 costs 1.38x as much per hour and is only ahead on memory.

Read the last column as the whole argument. Compute is 1.00x and price is 1.38x, so any job limited by math is cheaper on the H100. Bandwidth is 1.43x, just above the price ratio, so a job limited purely by memory traffic can come out slightly ahead on the H200.

Is an H200 cheaper than an H100 for the same training run?

Only if it finishes the run at least 1.38 times faster. Take a job that runs 100 hours on one 8 x H100 node. At listed rates that is 8 x $2.60 x 100 = $2,080.

H200 speedup over H100	Hours on 8 x H200	Cost on 8 x H200
1.00x (compute-bound)	100	$2,874.40
1.20x	83.3	$2,395.33
1.38x (breakeven)	72.4	$2,080.00
1.43x (bandwidth ratio)	69.8	$2,006.09

The same job on 8 x H100 costs $2,080. We computed each row as 8 x $3.593 x (100 / speedup).

A compute-bound run costs $794.40 more on H200s for the same result.

We expect most dense LLM training at a healthy batch size to sit near the top row. The large matrix multiplies are limited by Tensor Core math, which is identical on both GPUs. The bandwidth row is a ceiling, not a forecast: it assumes every second of the step is waiting on memory, which training steps rarely are.

When does the H200's extra memory make training faster?

When 80 GB forces you to give up compute to fit. There are three common ways that happens, and each one is a place where the H200 can earn back its higher rate.

Activation checkpointing is the most common. If activations do not fit, you recompute the forward pass during the backward pass.

A training step costs about 6 FLOPs per parameter per token, 2 forward and 4 backward, following Kaplan et al.. Full recomputation adds another 2, so the step does 8 units of work instead of 6. Dropping it on a GPU with room gives at most 8 / 6 = 1.33x, which is still under the 1.38x breakeven on its own.

Batch size is the second. A larger micro-batch can keep the Tensor Cores busier and cut gradient-accumulation steps. How much that helps depends on your model, so measure it.

CPU offloading is the third. Offloading moves optimizer state over PCIe, which NVIDIA lists at 128 GB/s against 3.35 TB/s of on-GPU memory bandwidth, so removing it can be worth more than 1.38x. If your H100 run offloads and an H200 run does not, the H200 is the likely winner.

Our rule: pick the H200 for training when it removes CPU offloading, or when it lets you drop recomputation and raise the batch size together. Dropping recomputation alone tops out at 1.33x.

How much model can you fully fine-tune on one 8-GPU node?

About 40B parameters on H100s and about 70B on H200s, before activations. Mixed-precision training with Adam keeps 16 bytes per parameter: 2 for BF16 weights, 2 for gradients and 12 for FP32 master weights and the two Adam moments. That figure comes from the ZeRO paper.

Sharded across a node with FSDP or ZeRO-3, 8 x H100 holds 640 GB / 16 = 40B parameters. 8 x H200 holds 1,128 GB / 16 = 70.5B. Activations need room on top, so the practical limits are lower.

The H200 node holds 1.76x the training state of an H100 node.

Take a 50B model. Its training state is 50 x 16 = 800 GB, which does not fit on one H100 node. On 8 x H200 it is 100 GB per GPU, leaving 41 GB each for activations.

The H100 answer is two nodes, 16 GPUs at $41.60/hr, against one H200 node at $28.744/hr. Two H100 nodes have twice the compute, so they win per finished job if they run at least 41.60 / 28.744 = 1.45 times faster than the H200 node. That means keeping at least 72% scaling efficiency across the network link between nodes.

Measure that efficiency before you count on it. One node is also simpler to run, and multi-node training on Nodus is still in Beta. If you have not measured your scaling, start on the single H200 node.

For parameter-efficient methods the memory gap matters less. Our 70B fine-tuning cost guide finds the cheapest 70B fine-tunes on one H100 with QLoRA or one B200 with LoRA, with the H200 only making sense for full fine-tuning.

Where is the H200 the better choice?

Long-context serving and memory-bound inference. Generating tokens reads the weights and KV cache for every step, so it is limited by memory bandwidth and by how many sequences fit.

NVIDIA's H200 page reports 1.9x faster Llama 2 70B inference and 1.6x faster GPT-3 175B inference than the H100, using larger batches on the H200. Both are above the 1.38x breakeven, so at those speedups the H200 is cheaper per token on Nodus. They are NVIDIA's own benchmarks, so check yours.

The H200 also wins when a model fits on one H200 but needs two H100s. Our VRAM guide for 70B models works through the serving case: about 73 GB of FP8 weights for a 70B model, plus KV cache. On one 80 GB H100 that leaves almost no room for context. On one 141 GB H200 it leaves about 68 GB.

Where is the H100 the better choice?

Almost every training job that already fits in 80 GB per GPU. Pretraining and continued pretraining at a normal batch size, full fine-tuning of models under about 30B on a node, and LoRA or QLoRA runs are all compute-bound. For those, the H100 does the same work for 28% less per hour.

The H100 is also easier to scale out cheaply. If your job already runs well on several nodes, adding H100s buys compute, while adding H200s mostly buys memory you are not using. Our A100 vs H100 comparison has a 30-minute probe for measuring speedup between two GPUs. The same method works for H100 against H200.

How do you let Nodus choose between the H100 and the H200?

List both and let the scheduler price them. The Nodus GPU guide documents type as a list of every accelerator you accept, and the scheduler picks among them by expected cost to finish:

spec:
  resources:
    gpu: {type: [H100, H200], count: 8}
  placement:
    interruptible: Allow          # use interruptible capacity only when it is cheaper to finish
    maxRateUSDPerHour: "30.00"

Check the plan before anything starts. The dry run shows the offering Nodus would use and the expected cost, without creating the job:

$ nodus apply -f job.yaml --dry-run=server -o estimate

If your job needs the H200's memory, do not list the H100. Write type: H200, or ask for memory instead with minMemory. Then a run that would not fit never lands on an 80 GB card. Pair either choice with checkpoints so an interrupted run resumes instead of starting over.

FAQ

Is the H200 faster than the H100 for training?

Not for compute-bound training: both list 1,979 TFLOPS of sparse BF16 and 3,958 of sparse FP8. The H200 is faster when the job is limited by memory capacity or bandwidth.

Can H100 code run on an H200 unchanged?

Yes in normal use, because both are Hopper GPUs with the same CUDA compute capability, 9.0. Nothing needs to change in the code. You may want to raise batch size or turn off activation checkpointing to use the extra memory.

How does the H200 compare with the B200 on Nodus?

Nodus lists the B200 at $5.50/hr with 180 GB, against $3.593/hr with 141 GB for the H200. The B200 is a newer Blackwell GPU with more compute: our 70B fine-tuning cost guide prices dense BF16 peak at $2.44 per petaFLOPS-hour on the B200 against $3.63 on the H200. See our B300 vs B200 comparison for Blackwell pricing.

Is the H200 NVL the same as the H200 SXM?

No. NVIDIA lists the H200 NVL as a PCIe card with the same 141 GB and 4.8 TB/s but lower compute (1,671 TFLOPS sparse BF16) and up to 600 W. The prices in this post are for the SXM version.

Originally published on the Nodus blog.