This article was originally published on Best GPU for LLM. The full version with interactive tools, FAQ, and live pricing is on the original site.
You already know which model you want to run. The only question is which GPU to rent for it — so here is the whole answer up front.
| Model size | Rent this | VRAM | Approx $/hr (mid-2026) |
|---|---|---|---|
| 7B-13B (Q4) | RTX 4090 | 24GB | from ~$0.35 |
| 34B (Q4) | A100 80GB or RTX 6000-class | 48-80GB | ~$0.75-1.50 |
| 70B (Q4) | A100 80GB / H100 80GB, or 2x RTX 4090 | 80GB | ~$2.00-3.00 |
| 70B long context / 100B+ MoE | H200 | 141GB | ~$2.60+ |
| Frontier throughput | B200 | 192GB | ~$3.50 where available |
These are GPU-first cloud rates (RunPod, Vast.ai and similar marketplaces) as of mid-2026 — prices move weekly, so treat them as anchors, not quotes. Hyperscalers like AWS and GCP charge 2-3x more for the same silicon, and spot or interruptible pricing runs 50-80% below on-demand if you can tolerate getting reclaimed mid-run.
Think in tokens per dollar, not dollars per hour
The hourly rate is the wrong number to optimize. What you actually buy is throughput, and a faster GPU that finishes sooner often costs less per million tokens than a cheap one that grinds all night.
A concrete example. An A100 80GB at roughly $1.50/hr pushing a 70B Q4 model through vLLM with batching can chew through a large summarization job in a third of the time a pair of budget cards would take. The meter runs at double the rate but for a third of the hours — you come out ahead, and you get your results the same afternoon.
The math flips for interactive chat. If you are the only user, a single conversation stream never saturates an H100, so most of that expensive compute sits idle between your keystrokes. Batch inference rewards big iron; single-user chat rewards the cheapest card that fits the model. Our tokens-per-second predictor lets you estimate throughput for a given GPU and model before you spend a cent.
7B-13B: rent an RTX 4090 and nothing bigger
A 13B model at Q4 needs about 10GB of VRAM. A 4090's 24GB swallows that with room for a generous context window, and marketplace rates start around $0.35/hr — sometimes under $0.25/hr on Vast.ai when supply is loose.
I'll say it plainly: most people renting H100s for 13B models are burning money. The H100 is maybe 2x faster for single-stream inference at 5-8x the price. Unless you are batch-serving hundreds of concurrent users, the 4090 wins tokens-per-dollar by a landslide at this size.
34B: the awkward middle tier
A 34B at Q4 technically squeezes into 24GB, but the context window gets claustrophobic fast. The comfortable move is a 48GB card — an RTX 6000 Ada-class rental at roughly $0.75-1.00/hr — or an A100 80GB at $1.10-1.50/hr if you want long context and headroom for KV cache.
My pick here depends on the job. Interactive coding assistant on a 34B? RTX 6000-class, cheapest that fits. Batch processing thousands of documents? A100, because the bandwidth advantage compounds across every request in the batch.
70B: A100 or H100, plus the 2x 4090 trick
A 70B model at Q4 wants roughly 40-45GB for weights plus KV cache, which means one 80GB card. On-demand A100 80GB rents for about $2/hr on GPU-first clouds; H100 80GB runs $2.50-3.00/hr and delivers meaningfully faster prefill and generation. If the job is throughput-bound, the H100 usually wins per-token despite the higher rate.
The budget route is two RTX 4090s with tensor parallelism, often $0.70-0.90/hr combined. It works well in vLLM and exllama-family stacks, though you eat some inter-GPU overhead and more setup friction. For the full hardware picture at this size — local and rented — see our best GPU for Llama 70B breakdown.
Frontier MoE and long context: H200 and B200
Once you cross into 100B+ MoE territory or need 128K+ context on a 70B, the 80GB cards run out of room for the KV cache. The H200's 141GB is the practical answer at roughly $2.60+/hr — the extra capacity goes straight into context length and batch size.
The B200 at 192GB and around $3.50/hr is the throughput king when you can find one. Supply is still constrained mid-2026, so availability is spotty on marketplaces and queues are common. Rent it for serious batch workloads; don't wait around for one just to run a chat session.
For provider choice at any of these tiers — reliability, pricing consistency, and which marketplace to trust with a long job — our RunPod vs Vast.ai comparison covers the tradeoffs in detail.
When renting is the wrong answer
Cloud GPUs shine for burst work: a weekend of experiments, a monthly fine-tune, occasional 70B runs. But if you run models every single day, the rental meter never stops and ownership starts winning fast. A daily 7B-34B workload on a rented 4090 at 4+ hours a day costs more per year than the card itself.
If that describes you, buy a 24GB card and rent only for the big stuff. The full break-even math is in our cloud vs local cost breakdown.
See the recommended pick on the original guide
Fine-tuning is the exception where I rent even as a hardware owner. It demands far more VRAM than inference and runs in bounded bursts, which is exactly the workload rental pricing was made for — we cover the setup in renting a GPU for LLM fine-tuning.
Common mistakes that inflate your bill
- Renting an H100 for a small model. A 7B on an H100 is a sports car in a school zone. Match the tier to the model size in the table above and pocket the difference.
- Ignoring storage and egress fees. A 70B checkpoint is 40GB+ of downloads, and persistent volumes bill around the clock even while the GPU is off. Budget for data movement, not just compute.
- Leaving instances running idle. An H100 forgotten overnight is $25+ gone by morning. Set auto-stop timers, use serverless endpoints for spiky workloads, and check the dashboard before bed.
- Running spot instances without checkpointing. Spot pricing at 50-80% off is a genuine bargain right up until your instance gets reclaimed at hour nine of a ten-hour job. Save state every 30-60 minutes or pay for on-demand.
Final verdict
| Your model | Rent this | Expect to pay |
|---|---|---|
| 7B-13B | RTX 4090 24GB | from ~$0.35/hr |
| 34B | RTX 6000-class or A100 80GB | ~$0.75-1.50/hr |
| 70B | H100 80GB (throughput) or 2x 4090 (budget) | ~$0.70-3.00/hr |
| 70B long context / 100B+ MoE | H200 141GB | ~$2.60+/hr |
| Frontier batch throughput | B200 192GB | ~$3.50/hr |
Start on a marketplace, start one tier smaller than you think you need, and move up only when VRAM or throughput actually forces you to.
Rent the smallest GPU that fits your model with context headroom — every dollar above that tier buys speed you should only pay for when batching.
Frequently asked questions
What is the best cloud GPU for LLM inference?
For most people, a rented RTX 4090 is the best value for 7B-13B models at roughly $0.30-0.45 per hour. Step up to an A100 80GB-class card for 34B, and an A100 or H100 80GB for 70B. The H200 only earns its rate for very long context windows or 100B+ MoE models.
How much does it cost to run a 70B model in the cloud?
Expect roughly $2-3 per hour for a single A100 or H100 80GB on GPU-first clouds like RunPod or Vast.ai, which runs a 70B model comfortably at Q4 quantization. A pair of RTX 4090s brings that down to roughly $0.70-0.90 per hour if you can handle tensor parallelism. Hyperscalers charge two to three times more for the same silicon.
Is a rented RTX 4090 good enough for LLMs?
Yes — for 7B-13B models it is the tokens-per-dollar king of the rental market, and it can squeeze a 34B at Q4 with a shorter context window. At roughly $0.30-0.45 per hour it beats every datacenter card on value for small and mid-size models. Rent bigger only when your model genuinely will not fit in 24GB.
Should I rent an H100 or H200?
Rent an H100 for 70B-class models at normal context lengths — it is cheaper and much easier to find. The H200 makes sense when you need its 141GB for long context windows, large KV caches, or 100B+ MoE models that overflow 80GB. If your workload fits in the H100, the H200's premium buys you very little.
Related guides on Best GPU for LLM
- RunPod vs Vast.ai for LLM Inference in 2026 (Compared)
- Cloud vs Local GPU for LLM: Real Cost Breakdown
- Rent a GPU for LLM Fine-Tuning: The $30 Weekend Project
The full version lives on Best GPU for LLM — VRAM calculator, GPU comparison table, and live Amazon pricing.
Top comments (0)