Every GPU cloud comparison I see is a table of dollars per hour. Provider A is $2.10 an hour for an H100, provider B is $1.89, the newcomer is $1.40, therefore the newcomer wins. I understand the appeal, one number, easy to sort. It is also how teams end up paying more for GPUs while believing they got a deal, because $/hour is the least important number in GPU economics once you are running real workloads.
Here is what the hourly rate hides, and what I actually compare instead.
The number that matters is $/unit-of-work, not $/hour
You do not buy GPU time to have GPU time. You buy it to train a model, serve inference, run a batch. The honest metric is cost per unit of work done, cost per token served, per training run, per job. A GPU that is 20% cheaper per hour but delivers 40% less throughput on your actual workload is more expensive, full stop.
Throughput per dollar depends on things the hourly rate says nothing about: the exact GPU generation, memory bandwidth, interconnect, and how well your framework is tuned for that hardware. Two "H100" listings can perform differently depending on networking and host. Benchmark your workload, not the spec sheet.
The four costs the $/hour hides
1. Utilization. This is the big one. A GPU billed by the hour costs you the full hour whether it runs at 90% or 9%. If your workload is bursty and the GPU sits idle between bursts, your effective cost per unit of work is a multiple of the sticker rate. A slightly pricier provider with better autoscaling or faster provisioning can beat a cheap one you leave idle. Utilization, not rate, is where most GPU money leaks.
2. Data transfer and storage. GPUs eat data. If your training data or model weights live in one place and your GPUs in another, egress and cross-region transfer can quietly rival the compute bill. The cheap-GPU provider that charges premium egress, or forces you to move terabytes in and out, is not cheap. Price the whole data path, not the accelerator.
3. Provisioning time and cold starts. If it takes minutes to get a GPU node up and pull a large model into memory, and you churn nodes often, you pay that startup repeatedly, in both wall-clock and instance-hours. A provider with instant capacity can deliver more useful work per dollar than a cheaper one you spend half your time waiting on.
4. Availability and eviction. The cheapest GPU capacity is often the least available, spot or preemptible pools that vanish when demand spikes, which right now is often. If evictions kill your jobs and you restart from scratch, the cheap rate bought you expensive re-work. Reliability is a cost input, not a footnote.
The comparison I actually run
Instead of sorting a table by hourly rate, I answer these:
- What is the cost per unit of my actual work (token, epoch, job) on this provider, from a real benchmark, not the spec sheet?
- What utilization can I realistically hit here, given provisioning speed and autoscaling? A cheap rate at 30% utilization loses to a fair rate at 80%.
- What is the fully-loaded path cost, compute plus storage plus egress plus the ops time to run it?
- How available is the capacity at the tier I need, and what does an eviction cost me in re-work?
Only after those does the hourly number mean anything, and by then it is usually not the deciding factor.
The FinOps habit underneath this
The reason $/hour is seductive is the same reason idle instances persist: the sticker price is visible and the waste is not. GPU cost discipline is the same discipline as the rest of FinOps, pointed at the most expensive resource you rent. Measure utilization. Kill idle capacity. Schedule non-production GPU pools so they are not burning the sticker rate at 3am doing nothing (we schedule GPU node groups the same way we schedule any other non-prod resource, that is part of what ZopNight handles, but a scheduled scale-down does the crude version). The provider you pick matters less than whether you actually use what you rent.
The take
Comparing GPU clouds by $/hour is like comparing cars by fuel-tank size. It is a real number that tells you almost nothing about what the thing costs to run. Benchmark your workload, price the whole data path, factor utilization and availability, and compute cost per unit of work. Do that and the "expensive" provider often turns out cheaper, and the cheap one turns out to be the one quietly draining the budget.
How do you compare GPU providers, do you benchmark your own workload or trust the spec sheet? I trusted spec sheets exactly once, and the throughput gap on my real job was not close to what the numbers promised.
Top comments (0)