Why memory bandwidth matters more than TFLOPS for LLM inference

#llm #gpu #nvidia #machinelearning

Your GPU's TFLOPS rating does not decide how fast it generates tokens. Memory bandwidth does. Here's the spec that actually separates A100-era inference from H200-era inference.

Key takeaways

HBM3e reaches up to 9.6 Gb/s per pin versus HBM2e's 3.2 Gb/s, tripling per-pin signalling speed with the same 1,024-bit interface width.
Total GPU memory bandwidth jumps from 2.0 TB/s on the A100 (HBM2e) to 4.8 TB/s on the H200 (HBM3e), a 140% increase, and 8.0 TB/s on the B200.
LLM token generation (decode phase) is memory-bound: tokens per second tracks memory bandwidth far more closely than TFLOPS.
H200 achieves 31,712 tokens/sec on Llama 2 70B with TensorRT-LLM, 45% faster than H100's 21,806 tokens/sec, driven entirely by the HBM3e memory upgrade.
HBM3e is 2.5x more power-efficient per watt than HBM2e, delivering higher bandwidth at lower voltage (1.1V versus 1.2V).

What is HBM and how does it work?

High Bandwidth Memory is a form of stacked DRAM co-packaged with the GPU die on a shared silicon interposer, connected through Through-Silicon Vias (TSVs). Unlike GDDR6 memory sitting on separate chips across a PCB trace, HBM stacks sit microns from the compute die. The interface is wider, latency is lower, and bandwidth per watt is substantially better.

For reference: GDDR6X on a consumer GPU achieves roughly 1.0 TB/s. HBM2e on the A100 doubles that at 2.0 TB/s. HBM3e on the H200 SXM more than doubles HBM2e again at 4.8 TB/s. Each generation achieves this by increasing per-pin signalling speed, not by widening the interface bus.

HBM3e vs HBM2e: the numbers

Specification	HBM2e	HBM3	HBM3e
Flagship GPU	A100 SXM	H100 SXM	H200 / B200 SXM
Total bandwidth	2.0 TB/s	3.35 TB/s	4.8 TB/s (H200) / 8.0 TB/s (B200)
Pin speed	~3.2 Gb/s	6.4 Gb/s	up to 9.6 Gb/s
Memory capacity	80 GB	80 GB	141 to 192 GB
Operating voltage	1.2V	1.1V	1.1V
Power efficiency vs HBM2e	1x	~2x	2.5x

Why LLM decode is memory-bound, not compute-bound

LLM inference runs in two phases. The prefill phase processes the full input prompt in one forward pass and is compute-bound. The decode phase generates one token per forward pass and is memory-bound: every token requires loading the complete model weight matrix and accumulated KV cache from HBM before a small amount of arithmetic work.

At batch size 1, arithmetic intensity falls as low as 1 to 2 FLOP per byte — far below the compute-to-bandwidth ratio of even the A100. Memory bandwidth is the first bottleneck for real-world serving workloads at batch sizes below roughly 64 to 128, which covers most production inference deployments.

This is why upgrading from an A100 (HBM2e, 2.0 TB/s) to an H200 (HBM3e, 4.8 TB/s) improves inference throughput by 2 to 3x on 70B parameter models.

Benchmark: H200 vs H100 on Llama 2 70B