`NVIDIA's spec sheet says the H100 delivers 989 TFLOPS of FP16 compute. The A100: 312. The T4: 65. Simple arithmetic says the H100 is 15× faster.
So a 7-billion-parameter LLM should be 15× faster on an H100, right?
It's 150× faster.
The T4 struggles at ~15 tok/s. The H100 cruises at ~2,200 tok/s — and with continuous batching, north of 3,500.
The 15× TFLOPS gap doesn't explain the 150× throughput gap. The missing variable is the one thing NVIDIA's marketing pages bury on line three of the spec table:
Memory bandwidth.
The T4 has 300 GB/s. The H100 has 3,350 GB/s. That's an 11× gap in raw bandwidth, and closer to 100–150× in effective throughput once cache size and clock speed are factored in.
This post traces the arithmetic from first principles: why memory bandwidth is the bottleneck, how quantization turns it into a lever, and what tok/s you should actually expect across 12 real GPU × model combinations.
1. Memory Bandwidth: The Bottleneck Nobody Talks About
Autoregressive LLM inference has two phases:
| Phase | What Happens | Bottleneck |
|---|---|---|
| Prefill | Process the input prompt (all tokens in parallel) | Compute |
| Decode | Generate output tokens (one at a time) | Memory bandwidth |
The decode phase dominates total latency for any response longer than a few tokens. And in decode, the GPU spends 98%+ of its time doing nothing — waiting for the next chunk of model weights to arrive from VRAM.
The Arithmetic, Step by Step
Each generated token requires reading every single parameter from VRAM. For a 7B model at FP16:
`plaintext
Weights: 7,000,000,000 params × 2 bytes = 14 GB per token
H100: 14 GB ÷ 3,350 GB/s = 4.18 ms → 239 tok/s (theoretical)
T4: 14 GB ÷ 300 GB/s = 46.7 ms → 21 tok/s (theoretical)
`
Now consider the compute cost: ~14 TFLOPs per token. The H100's 989 TFLOPS could execute that 70 times in 4.18 ms. The compute units finish their work and sit idle, waiting for the next 14 GB of weights to trickle in.
You cannot compute faster than you can read.
Every meaningful LLM optimization — quantization, KV cache compression, FlashAttention, speculative decoding — is fundamentally about reducing bytes moved per token.
The Full GPU Landscape
| GPU | Bandwidth | 7B FP16 | 7B INT4 | 70B INT4 | VRAM |
|---|---|---|---|---|---|
| NVIDIA B200 | 8,000 GB/s | 571 | 2,286 | 229 | 192 GB |
| NVIDIA H200 | 4,800 GB/s | 343 | 1,371 | 137 | 141 GB |
| NVIDIA H100 | 3,350 GB/s | 239 | 957 | 96 | 80 GB |
| NVIDIA A100-80GB | 2,039 GB/s | 146 | 583 | 58 | 80 GB |
| NVIDIA RTX 4090 | 1,008 GB/s | 72 | 288 | 28.8 | 24 GB |
| NVIDIA L40S | 864 GB/s | 62 | 247 | — | 48 GB |
| NVIDIA T4 | 300 GB/s | 21 | 86 | — | 16 GB |
Theoretical tok/s at 100% bandwidth utilization. Real-world: multiply by 0.6–0.85. "—" = doesn't fit in VRAM.
2. Quantization: 4× Faster Without Changing GPUs
If your bottleneck is bytes-per-token, and you can't change the GPU's memory bandwidth, there's exactly one lever left: reduce bytes per parameter.
| Precision | Bytes/Param | 7B Model | 70B Model | Speedup | Quality |
|---|---|---|---|---|---|
| FP16 | 2.0 | 14 GB | 140 GB | 1× | Reference |
| INT8 | 1.0 | 7 GB | 70 GB | ~1.8× | Negligible |
| INT4 (GPTQ/AWQ) | 0.5 | 3.5 GB | 35 GB | ~3.5× | 1–3% perplexity |
| INT3 | 0.375 | 2.6 GB | 26 GB | ~4.5× | 3–8% loss |
The rule of thumb: the larger the model, the harder it is to break with quantization. A 405B model at INT2 often outperforms a 70B model at FP16 on knowledge tasks — despite using fewer bytes per token.
What quantization doesn't improve: time-to-first-token (prefill latency). Prefill is compute-bound. Quantization can even make it slower due to dequantization overhead. The gains apply almost entirely to the decode phase.
3. Real Benchmarks
Theoretical ceilings are clean math. Reality includes framework overhead, attention kernel efficiency, and KV cache management. These numbers are from vLLM 0.6.x, continuous batching, bare-metal H100 instances — mid-2026.
| Model | Precision | GPU(s) | Batch 1 | Batch 8 | Batch 32 |
|---|---|---|---|---|---|
| Llama 4 Scout (8B) | FP16 | 1× H100 | 185 | 1,200 | 3,200 |
| Llama 4 Scout (8B) | INT4 | 1× H100 | 620 | 3,800 | 8,500 |
| Llama 4 Scout (8B) | INT4 | 1× RTX 4090 | 95 | 310 | — |
| Mistral Small 3 (7B) | FP16 | 1× H100 | 195 | 1,250 | 3,500 |
| Llama 4 Maverick (70B) | FP16 | 2× H100 | 35 | 190 | 480 |
| Llama 4 Maverick (70B) | INT4 | 1× H100 | 78 | 440 | 1,050 |
| Mistral Large 2 (123B) | FP16 | 4× H100 | 15 | 70 | 140 |
| DeepSeek-V3 (671B MoE) | INT8 | 8× H100 | 18 | 85 | 180 |
| Qwen 2.5 (72B) | INT4 | 1× H100 | 72 | 410 | 960 |
| Phi-4 (14B) | INT4 | 1× RTX 4090 | 78 | — | — |
| Gemma 3 (27B) | INT4 | 1× RTX 4090 | 55 | — | — |
"TP=N" = Tensor Parallelism across N GPUs. "—" = VRAM insufficient at listed context length. Output token decode only.
The headline: INT4 turns a 70B model from "needs 2 GPUs and is still slow" into "runs on 1 GPU and is faster than FP16 on 2."
4. Batch Size: The Latency–Throughput Tradeoff
Batching fills the idle time between memory reads. While one request waits for weights, another's compute can run. But per-request latency climbs with batch size.
| Workload | Batch | Latency Target |
|---|---|---|
| Real-time chat / copilot | 1–4 | <200ms TTFT |
| Code completion | 1–2 | <20ms/tok |
| Support chatbot | 4–16 | <1s TTFT |
| Summarization (async) | 16–32 | Throughput priority |
| Dataset labeling | 32–128 | Throughput only |
Continuous batching (vLLM, TensorRT-LLM, SGLang) is the critical innovation. Traditional static batching waits for all requests to finish. Continuous batching evicts completed requests and admits new ones at every decode step — eliminating the "one slow request holds up 31 others" problem.
If you deploy without continuous batching, you're leaving 50–70% of your GPU throughput on the table.
5. Multi-GPU: The Scaling Efficiency Problem
When one GPU isn't enough, you split the model with tensor parallelism. But all-reduce communication between GPUs eats into your gains:
| Configuration | Batch 32 tok/s | Per-GPU Efficiency | NCCL Tax |
|---|---|---|---|
| 2× A100 (TP=2) | 380 | 92% | ~8% |
| 4× A100 (TP=4) | 610 | 80% | ~20% |
| 8× A100 (TP=8) | 840 | 63% | ~37% |
At TP=8, nearly 40% of interconnect bandwidth goes to synchronization.
This is why expert-parallel inference for MoE models is revolutionary: each expert lives on a dedicated GPU, and only the active experts participate in each forward pass. DeepSeek-V3 has 671B total parameters but only 37B active — so it's faster than a 70B dense model despite having 10× more total weights.
The One-Sentence Takeaway
LLM inference throughput = GPU memory bandwidth ÷ bytes per token. Everything else is overhead management.
Try the interactive calculator at jslet.com/llm-inference-latency — plug in your model, GPU, and quantization level. All 100% client-side, no signup.
`
Top comments (0)