DEV Community

Alex Chen
Alex Chen

Posted on

Stop Overpaying for GPUs: Spot vs On-Demand for LLM Inference (With Real Numbers)

#machinelearning #llm #gpu #cloud

Stop Overpaying for GPUs: Spot vs On-Demand for LLM Inference (With Real Numbers)

If you're running LLM inference on rented GPUs, you're probably overpaying. Not by a little — by 2-3x. The culprit isn't which cloud you picked. It's how you're paying for it.

I spend my days comparing GPU cloud pricing (it's literally what I do), and the spot vs on-demand decision is where most people leave the most money on the table. Here's the practical breakdown.

The Price Gap Is Bigger Than You Think

Take an RTX 4090 (24GB) — the workhorse for 7B-32B model inference:

  • On-demand: ~$0.34–$0.74/hour depending on the cloud
  • Spot/preemptible: ~$0.15–$0.30/hour for the same card

That's a 50-70% discount for the exact same silicon. For an A100 80GB, the gap is similar: ~$1.20/hr on-demand vs ~$0.50/hr spot.

But cheap isn't free. Here's what you're actually trading.

What "Spot" Really Means in Practice

Spot instances can be reclaimed by the provider with little warning (typically 30 seconds to 2 minutes). For LLM inference, that means:

  1. Model reload time: A 70B model takes 3-5 minutes to load into VRAM. If you get preempted twice an hour, you've lost more than you saved.
  2. Request interruption: In-flight requests die. You need retry logic or a queue.
  3. Availability variance: During peak hours, spot capacity dries up. Your "cheap" GPU might simply not exist when you need it.

The Decision Framework

Use spot when:

  • You're doing batch inference (embeddings, bulk classification, offline evals)
  • Your workload is checkpoint-friendly or stateless
  • You can tolerate 5-10 minute interruptions
  • You're running experiments and cost matters more than latency

Use on-demand when:

  • You're serving real users with latency SLAs
  • Your model takes >2 minutes to load (large models)
  • It's peak hours in US/EU and spot markets are thin
  • The cost of an interruption exceeds the savings (do the math!)

The hybrid most people should use:
Run your baseline traffic on 1-2 on-demand instances, burst with spot. Most inference workloads have a predictable floor and a spiky ceiling. Pay full price for the floor, spot price for the spikes.

The Math People Skip

Don't just compare hourly rates. Calculate effective cost per 1K tokens:

effective_hourly = (hourly_rate × hours_run + reload_cost) / hours_run
Enter fullscreen mode Exit fullscreen mode

Where reload_cost = preemptions × reload_time_hours × hourly_rate.

Example: $0.20/hr spot, preempted 3x/hour, 4-min reloads on a 70B model:

  • Reload overhead: 3 × (4/60) × $0.20 = $0.04/hr extra
  • Effective rate: $0.24/hr — still way cheaper than $0.74 on-demand

But with 10 preemptions/hour? Effective rate hits $0.33/hr plus the latency pain. At that point, on-demand wins.

One More Thing: Egress Fees

The sneakiest cost isn't compute — it's data transfer out. Some clouds charge $0.05-0.09/GB for egress. If you're doing RAG with large context windows or serving multimodal models, egress can exceed your GPU bill. Always check the egress pricing before committing.

I track GPU cloud prices weekly and post updates when prices move significantly. If you want the short version without reading pricing pages, I share them on my Telegram channel: https://t.me/qisuancloud

And if you want to skip the spreadsheet entirely, our free GPU Advisor picks the cheapest option for your workload in about 30 seconds: https://qisuanai.com/advisor

Prices referenced are from public pricing pages, October 2026. Always verify before you rent — cloud pricing changes fast.

Top comments (0)