The NVIDIA H100 is the default answer to "what GPU should I serve this model on?" The RTX PRO 6000 Blackwell costs about two-thirds as much per hour, has more memory, and less than half the memory bandwidth. Which of those facts wins depends on one question that most comparisons skip: does your model fit on a single card?
This post works through that question using independent benchmarks, current market prices, and arithmetic you can check. No new measurements were taken; sources and a reproduction script are at the end.
TL;DR
- Single-GPU models: the RTX PRO 6000 is cheaper per token — about 30–40% cheaper than an H100 at October 2026 median on-demand prices.
- Multi-GPU models: the advantage shrinks and then reverses. Roughly 8% cheaper at 4-way tensor parallelism, roughly 8% more expensive at 8-way. The RTX PRO 6000 has no NVLink.
- The H100 has 2.1× the memory bandwidth. The RTX PRO 6000 has 16 GB more memory, which turns into 1.8–2.6× more KV-cache room for 30B–70B models.
- Rule of thumb: cost-per-token ratio = price ratio × throughput ratio. Today's price ratio is about 0.63, so the RTX PRO 6000 wins whenever an H100 is less than ~1.6× faster on your workload.
The specs that matter for inference
| RTX PRO 6000 Blackwell Server Edition | H100 SXM | |
|---|---|---|
| Memory | 96 GB GDDR7 | 80 GB HBM3 |
| Memory bandwidth | 1,597 GB/s | 3,350 GB/s |
| Multi-GPU interconnect | PCIe Gen 5 only — no NVLink | NVLink 900 GB/s, plus PCIe Gen 5 |
| Lowest native tensor precision | FP4 | FP8 |
| MIG partitions | up to 4 | up to 7 |
| Max power | up to 600 W | up to 700 W |
| On-demand price, median, Oct 2026 | $2.20 / GPU-hr (53 providers) | $3.49 / GPU-hr (57 providers) |
Two notes on that table.
Peak TFLOPS are left out on purpose. NVIDIA's product pages quote them with different sparsity conventions, so putting them side by side invites a wrong conclusion. For LLM serving, where the decode phase is usually limited by how fast weights and KV cache can be read, memory bandwidth is the more predictive spec.
"RTX PRO 6000" also covers more than one card. The Workstation Edition lists 1,792 GB/s of bandwidth; the Server Edition most clouds rent lists 1,597 GB/s. That difference shows up in the benchmarks below.
What independent benchmarks show
The most useful public data comes from two vLLM benchmark runs published by CloudRift. Both compare the RTX PRO 6000 and the H100 directly, on the same models and settings.
Run A — November 2025. Single GPU, GLM-4.5-Air (AWQ 4-bit), 1,000 input / 1,000 output tokens, 256–512 concurrent requests, RTX PRO 6000 Workstation Edition.
| GPU | Total throughput |
|---|---|
| RTX PRO 6000 (Workstation) | 3,140 tok/s |
| H100 SXM | 2,987 tok/s |
Run B — January 2026. Google Cloud 8-GPU nodes (G4 with RTX PRO 6000, A3 High with H100), 8,000 input / 8,000 output tokens.
| Workload | RTX PRO 6000 | H100 | H100 advantage |
|---|---|---|---|
| GLM-4.5-Air AWQ, one model per GPU | 2,290.69 tok/s | 2,556.03 tok/s | +11.6% |
| Qwen3-Coder-480B-A35B AWQ, 4-GPU tensor parallel | 1,602.96 tok/s | 2,328.63 tok/s | +45.3% |
| GLM-4.6 FP8, 8-GPU tensor parallel | 1,651.67 tok/s | 2,833.77 tok/s | +71.5% |
Both machines are 8-GPU nodes running identical configurations, so the ratios hold however the throughput was aggregated across the node.
The two runs disagree on the single-GPU case: the RTX PRO 6000 is 5% ahead in Run A, while the H100 is 12% ahead in Run B. They differ in three ways at once — card edition, sequence length, and host — so neither result is wrong. The most plausible reading is that Run A's Workstation card had about 12% more bandwidth than a typical cloud Server Edition, and Run B's 8K-token sequences make each decoded token read much more KV cache, which favours the H100's bandwidth.
The multi-GPU rows are not ambiguous. As tensor parallelism widens, the H100's NVLink pulls away: +45% at 4-way, +72% at 8-way.
Cost per million tokens
Throughput alone does not answer a cost question. Combining it with price:
cost per token (RTX / H100) = (RTX $/hr ÷ H100 $/hr) × (H100 tok/s ÷ RTX tok/s)
At median on-demand prices ($2.20 vs $3.49, price ratio 0.63):
| Scenario | H100 throughput vs RTX PRO 6000 | RTX PRO 6000 cost per token | RTX break-even price |
|---|---|---|---|
| 1 GPU, 1K/1K (Run A) | 0.95× | 40% cheaper | $3.67/hr |
| 1 GPU, 8K/8K (Run B) | 1.12× | 30% cheaper | $3.13/hr |
| 4-GPU tensor parallel (Run B) | 1.45× | 8% cheaper | $2.40/hr |
| 8-GPU tensor parallel (Run B) | 1.72× | 8% more expensive | $2.03/hr |
The break-even column is the RTX PRO 6000 hourly price at which it matches an H100 at $3.49/hr. Rent one below that price and it is the cheaper option for that workload.
In absolute terms, Run A works out to about $0.195 per million tokens on the RTX PRO 6000 and $0.325 on the H100, counting input and output tokens together.
CloudRift's own cost tables differ from these. Each run used a different price basis — Runpod list prices in November 2025, and an estimated cost of ownership in January 2026. The table above re-prices both runs with one consistent, current source, so the scenarios are comparable with each other.
What the extra 16 GB actually buys
Twenty percent more memory sounds marginal. It is not, because every extra gigabyte lands after the weights are loaded, in the space left for KV cache. For models that nearly fill an 80 GB card, that space roughly doubles.
KV cache per token for a standard attention model:
2 (K and V) × layers × KV heads × head dim × bytes per element
With an FP8 KV cache and 4 GB reserved for the runtime:
| Model | Weights | KV per token | Free for KV on H100 80 GB | Free for KV on RTX PRO 6000 | 32K sequences: H100 → RTX |
|---|---|---|---|---|---|
| Qwen3-Coder-30B-A3B (BF16) | 56.8 GB | 48 KB | 19.2 GB | 35.2 GB | 12 → 23 |
| Qwen3-32B (BF16) | 61.1 GB | 128 KB | 14.9 GB | 30.9 GB | 3 → 7 |
| Llama-3.3-70B (FP8) | 65.8 GB | 160 KB | 10.2 GB | 26.2 GB | 2 → 5 |
| Llama-3.3-70B (BF16) | 131.5 GB | 160 KB | does not fit | does not fit | — |
A 70B model at FP8 is the clearest case. On an H100 it fits, but leaves room for only two concurrent 32K-token conversations. On an RTX PRO 6000, five. For long-context, RAG-heavy, or agentic workloads, that headroom decides how much traffic a single card can take.
The last row matters just as much. A 70B model at BF16 needs two cards on either GPU — and once two cards have to talk to each other, the interconnect comes back into play, and with it the H100's advantage.
When the H100 is the better buy
- Your model needs four or more GPUs. Once weights plus KV cache outgrow two cards (roughly 160–190 GB), you are into 4-way tensor parallelism or wider, where NVLink beats PCIe by a margin that outweighs the price gap.
- Single-stream decode speed is the product. If latency for one user matters more than total throughput, 2.1× the memory bandwidth is hard to argue with.
- You are serving very long sequences on a model that barely fits. Run B suggests the bandwidth gap grows with sequence length.
- You need many small isolated tenants. MIG gives you up to seven partitions on an H100 versus four on an RTX PRO 6000.
When the RTX PRO 6000 is the better buy
- The model and its KV cache fit on one 96 GB card. That covers most 7B–35B models at BF16 and 70B-class models at FP8 or below.
- You scale out with replicas rather than tensor parallelism. Data-parallel serving never touches the interconnect, so the PCIe limitation does not apply.
- Concurrency on long contexts is the constraint. The KV-cache table above is the whole argument.
- You want to serve FP4-quantized models. FP4 tensor cores are a Blackwell feature; Hopper stops at FP8.
A short decision checklist
- Add up weights and the KV cache your target concurrency needs, at the precision you plan to serve.
- If the total fits in 96 GB minus a few GB of overhead, the RTX PRO 6000 is very likely cheaper per token. Benchmark it against your real prompt lengths.
- If you need 2 GPUs, test both — the result could go either way.
- If you need 4 or more GPUs, start from the H100 (or H200), unless your RTX PRO 6000 rate is below the break-even price for your workload.
- Re-run the cost formula with the prices you are actually quoted. Market medians move every month.
FAQ
Is the RTX PRO 6000 faster than the H100 for LLM inference?
Not usually per card. In independent vLLM tests the H100 was between 5% slower and 12% faster on a single-GPU model, and 45–72% faster once tensor parallelism across 4–8 GPUs was involved. The RTX PRO 6000's case is cost per token, not raw speed.
How much does it cost to rent an RTX PRO 6000 versus an H100?
As of October 7, 2026, the median on-demand price across tracked providers was $2.20 per GPU-hour for the RTX PRO 6000 and $3.49 for the H100, per getdeploying.com.
What is the RTX PRO 6000's memory bandwidth?
1,597 GB/s for the Server Edition and 1,792 GB/s for the Workstation Edition, both with 96 GB of GDDR7. The H100 SXM has 3,350 GB/s of HBM3.
How many tokens per second can an RTX PRO 6000 serve?
It depends heavily on model, precision and concurrency. One public data point: about 3,140 total tokens/s on GLM-4.5-Air (AWQ 4-bit) at 256–512 concurrent requests, 1K input / 1K output, on a Workstation Edition card.
Is the "RTX 6000 Blackwell" the same GPU as the "RTX PRO 6000"?
Yes — both names refer to the 96 GB Blackwell card, sold in Workstation, Max-Q and Server editions. It is a different card from the RTX 6000 Ada (48 GB) and the older Quadro RTX 6000 (24 GB).
Method and caveats
-
Nothing here was measured by the author. Throughput comes from CloudRift's published benchmarks, prices from getdeploying.com's public listings, specs from NVIDIA's product pages, and model shapes from each model's
config.json. - Prices are medians. Your actual price determines the answer, which is why the formula and break-even prices are included.
- Throughput ratios are workload-specific. The benchmarks used GLM and Qwen3-Coder models; a different model, quantization, or serving stack will shift them.
- Run A likely flatters the RTX PRO 6000 slightly, because it used the Workstation Edition, with about 12% more bandwidth than the Server Edition most clouds rent.
-
Units: in the VRAM table, GB means GiB (2³⁰ bytes), the unit
nvidia-smireports.
Reproduce the numbers
GiB = 1024**3
# Prices: getdeploying.com on-demand medians, 2026-10-07
RTX_HR, H100_HR = 2.20, 3.49
# (rtx tok/s, h100 tok/s) from CloudRift's published runs
RUNS = {
"1 GPU, 1K/1K": (3140.00, 2987.00),
"1 GPU, 8K/8K": (2290.69, 2556.03),
"4-GPU TP": (1602.96, 2328.63),
"8-GPU TP": (1651.67, 2833.77),
}
for name, (rtx, h100) in RUNS.items():
rel = (RTX_HR / H100_HR) * (h100 / rtx)
print(f"{name:<14} RTX cost/token = {rel:.0%} of H100 | break-even ${H100_HR / (h100 / rtx):.2f}/hr")
# KV-cache headroom: FP8 KV cache, 4 GiB runtime reserve, 32K-token sequences
MODELS = [ # name, params (B), bytes/param, layers, kv_heads, head_dim
("Qwen3-Coder-30B-A3B BF16", 30.5, 2, 48, 4, 128),
("Qwen3-32B BF16", 32.8, 2, 64, 8, 128),
("Llama-3.3-70B FP8", 70.6, 1, 80, 8, 128),
]
for name, p, wb, L, kvh, hd in MODELS:
weights = p * 1e9 * wb / GiB
kv_seq = 2 * L * kvh * hd * 1 * 32_768 / GiB
fits = {cap: int((cap - weights - 4) / kv_seq) for cap in (80, 96)}
print(f"{name:<26} H100: {fits[80]:>2} seqs | RTX PRO 6000: {fits[96]:>2} seqs")
Sources
- NVIDIA, RTX PRO 6000 Blackwell Server Edition and Workstation Edition datasheet
- NVIDIA, H100 Tensor Core GPU
- CloudRift, RTX PRO 6000 vs H100, H200, and L40S: LLM Inference (Nov 2025)
- CloudRift, Benchmarking LLM Inference on B200, H200, H100, and RTX PRO 6000 (Jan 2026)
- getdeploying.com, RTX PRO 6000 rental prices and H100 rental prices (checked 2026-10-07)
- Model configs: Qwen3-Coder-30B-A3B-Instruct, Qwen3-32B, Llama-3.3-70B-Instruct
Top comments (0)