Thurmon Demich

Posted on Jul 28 • Originally published at bestgpuforllm.com

RTX 5090 vs H100 for LLM in 2026 ($2K vs $30K Debate)

#rtx5090 #h100 #datacenter #comparison

Cross-posted from Best GPU for LLM — visit the original for our VRAM calculator, GPU comparison table, and current Amazon pricing.

Quick answer: If you're a hobbyist reading this, the answer is RTX 5090. Move on. Don't overthink it. The H100 SXM only earns its 15× price tag in three narrow scenarios — FP8 training, serving 50+ concurrent users, or 256K+ context windows. For everything else, a $2,000 5090 does the same job at consumer wattage.

Who this comparison is actually for

I keep seeing the same question in Reddit threads and Discord servers: "Should I stretch my budget to an H100?" The person asking is almost always a solo developer running Llama 3 8B in Ollama. So let me split the audience up front, because "5090 vs H100" is really two very different questions.

You're in the wrong debate if you're:

A hobbyist running 7B–34B models at home
A solo indie dev serving one client through an API
Anyone whose "server" is a workstation under their desk

You're in the right debate if you're:

A small lab or startup sizing inference infrastructure
Someone planning FP8 training runs on 70B+ models
A developer serving batched requests to dozens of concurrent users
Anyone who thinks "TCO" and "SLA" without having to look them up

The first group buys the 5090. The second group might not. That's the whole article, really — but the details are where the money hides.

Specs side-by-side

Numbers first, opinions second. Here's the raw hardware, no marketing fluff.

Spec	RTX 5090	H100 SXM
VRAM	32GB GDDR7	80GB HBM3
Memory bandwidth	1,792 GB/s	3,350 GB/s
Architecture	Blackwell	Hopper
TGP / power	575W	700W
FP8 support	Native	Native
FP16 TFLOPS	~105	~1,979 (sparse)
NVLink / interconnect	PCIe Gen5 x16	NVLink 900 GB/s
Form factor	Consumer PCIe	SXM5 module
Price (2026)	~$2,000	~$30,000+

Two numbers matter most on that table. Bandwidth (3,350 vs 1,792 GB/s) governs token throughput on large models. VRAM (80 vs 32) governs which models fit at all. The H100 wins both — but by roughly 2× on each axis, at 15× the price. That's the whole cost-per-perf story in one sentence.

VRAM chart available at the original article

Real workload comparison

Specs are a starting point. Real inference numbers are what you actually feel. Here's what I've measured (or pulled from the vLLM and llama.cpp community benchmarks) at the workloads people actually run:

Workload	RTX 5090	H100 SXM
Llama 3 70B Q4_K_M (single request)	~30–40 tok/s	~55–70 tok/s
Llama 3 70B FP8 (batched, 16 users)	~180 tok/s aggregate	~800 tok/s aggregate
Llama 3.1 405B Q4	Needs 4× 5090 (multi-GPU)	Fits single card
1M context KV cache	Caps around ~64K in practice	Handles 256K+ comfortably
FP8 training (7B LoRA)	Works but slow	Native, ~4× faster

Single-user 70B inference on the 5090 is perfectly usable — 30–40 tok/s is faster than most people read. The H100 pulls ahead once you batch requests: the extra bandwidth and Tensor cores let it serve many concurrent users at once instead of one very fast one. That's the split most benchmarks bury.

The 405B row is where the 5090 story gets messy. You physically cannot load Llama 3.1 405B on a single 5090 at Q4 — you're looking at four cards and tensor parallelism through PCIe Gen5. The H100 loads it and moves on. If your product depends on frontier-class dense models, the H100 stops being a luxury and starts being the only path.

Here's the batched-serving math that most single-user benchmarks skip. Under vLLM with continuous batching, the H100's 3,350 GB/s bandwidth and Hopper Tensor cores keep dozens of KV caches hot in parallel. The 5090 has the compute to serve batches too, but 32GB fills fast: at Llama 70B FP8, weights alone eat ~70GB — you can't fit that on a 5090 at all, so you drop to Q4 and lose quality per batched user. The H100 keeps you at FP8 and stacks users. That is the actual moat, not raw tok/s.

Which should YOU buy?

I'll give you the decision tree I actually use when friends ask.

You should buy the RTX 5090 if:

You're a single user running local LLMs for coding, chat, or hobby projects — see our best GPU for LLM server guide for the small-scale serving story
Your workloads top out around 70B Q4 and 32K context, which is where our best GPU for Llama 70B article lives
You want to fine-tune small models (7B–13B LoRAs) without paying cloud rates
You're comparing consumer flagships and want the honest sibling story — read RTX 5090 vs 4090 for LLM next

You should look at the H100 (buy or rent) if:

You need to serve 50+ concurrent users through vLLM — see our best GPU for vLLM writeup for the throughput math
You want native FP8 training runs on 70B+ models without gradient checkpointing acrobatics
Your product hinges on 256K–1M context windows for long documents or codebases
You already run a rack: PSU, cooling, and 208V power are solved problems, not new problems

There's a third path that gets ignored too often: rent the H100. If you only need Hopper class hardware for a specific training run or a launch spike, renting on RunPod costs $2–4/hour and takes exactly zero rack space in your apartment.

Common mistakes I keep seeing

Buying an H100 for solo hobbyist use. I've watched people drop $30K on an H100 to run Qwen 3 32B in Ollama for personal coding assistance. That workload doesn't even saturate a 4070. The 80GB sits mostly empty, the 700W blows a home breaker, and the SXM form factor won't fit any consumer motherboard without a $5K server chassis. Please do not do this.

Underestimating total cost of ownership. The H100 sticker price is $30K, but that's the beginning, not the end. Add a Supermicro or Dell chassis ($4K–8K), redundant 2000W+ PSUs, a proper rack (or rented colocation), 208V circuits if you're serious, and cooling that doesn't sound like a jet engine in your bedroom. Real all-in cost for a single-H100 workstation runs $40K–50K. Compare that to a 5090 in a $1,500 PC — under $4,000 total. That's your actual cost multiplier: closer to 12–13× than 15×.

Ignoring what "context" costs in VRAM. The 5090's 32GB is generous for weights but tight for KV cache. A 70B model at Q4 uses ~35GB with weights alone at long context. The how much VRAM for local LLM guide breaks this down — if you need 128K+ context on a big model regularly, the 5090 will disappoint you and the H100 won't.

Assuming FP8 is a 5090 story. Both cards support FP8 natively. Only the H100 has the tensor throughput and HBM3 bandwidth to make training runs feel snappy — expect roughly 4× the throughput on FP8 gradient passes at 70B. For pure inference, though, 5090 FP8 is genuinely good and gets you most of the way there.

Forgetting about depreciation. A $30K H100 is worth $30K today and quite possibly $12K in eighteen months when the next Hopper successor hits general availability. A $2K 5090 is a $2K 5090 that sells used for $1,400 in the same window. Absolute dollar loss on the H100 dwarfs the entire cost of the 5090. If you're not going to fully amortize the H100 through revenue-generating workloads, you're just holding a very expensive space heater.

Final verdict

Your situation	Best pick	Why
Solo hobbyist, any model up to 70B Q4	RTX 5090	32GB and 1,792 GB/s at consumer price
Small team, occasional big model	5090 + rent H100 on demand	Best cost per launch
Serving 50+ concurrent users	H100 SXM	Batched throughput crushes the 5090
FP8 training on 70B+	H100 SXM	Tensor throughput is the moat
256K+ context production workloads	H100 SXM	KV cache doesn't fit anywhere else
Anyone unsure which bucket they're in	RTX 5090	If you have to ask, it's the 5090

One-sentence verdict: Buy the RTX 5090 unless you're serving batched production traffic or doing FP8 training on 70B+ models — and if you're not sure which of those you are, you're not, so buy the 5090.

Related guides on Best GPU for LLM

Continue on Best GPU for LLM for the complete guide with interactive calculators and current GPU prices.

DEV Community