Every few months someone posts a confident take on whether running your own AI inference hardware beats paying a cloud API, and both the 'on-premise is obviously cheaper' camp and the 'cloud always wins' camp are making the same mistake. They are compressing a five-variable optimisation problem into a one-line opinion. I have had to build this calculation properly more than once, for actual purchasing decisions with real money attached. The honest answer takes a formula with five inputs, and those inputs change the answer completely depending on your traffic pattern.
The shortcut most people take is comparing a GPU's purchase price directly to a cloud bill and calling it done. That shortcut skips the variable that matters most of all, which is how much of that hardware's capacity you are actually using once you have bought it. Everything below follows from that one point.
The five variables that actually decide it
Hardware amortisation period. The capital cost of a GPU server is not a one-off number, it is a monthly cost that depends entirely on how long you plan to run the thing before it is replaced or written off. Assume an 18-month amortisation window instead of a 48-month one and your monthly hardware cost roughly triples. This single assumption moves most on-premise versus cloud arguments more than any other variable in the spreadsheet, and it is the one people state least explicitly when they want to win an argument online.
Power draw. Not the GPU's rated TDP on the spec sheet, the actual measured draw at your real utilisation, plus cooling, plus the rest of the chassis (CPU, memory, networking, storage). A multi-GPU box sitting mostly idle still pulls a surprising fraction of its peak draw. Price electricity off peak TDP and you will get a number that is wrong in both directions depending on how hard the box is actually working.
Utilisation rate. This is the one everyone forgets, and it is the one that quietly decides the whole argument. A GPU running at 15 per cent utilisation because you provisioned for peak traffic is not 'mostly idle for free', it is capital you have already paid for that is not earning anything. Cloud's entire pricing model is a bet that most customers run low utilisation and will happily pay a premium for elasticity instead of carrying that idle capacity themselves. If your utilisation is genuinely high and steady, that bet stops being profitable for the provider and starts being profitable for you.
Request volume. This is what utilisation is actually measuring, so it earns its own line. Low, spiky volume favours cloud, because you are not stuck paying for idle hardware between bursts. High, steady volume favours on-premise, because you are finally filling the capacity you already bought.
Model size. This decides how much hardware a single request needs, which decides how many requests you can push through a GPU per second, which is the denominator in every cost-per-request number you will ever calculate. A model that needs four GPUs per replica changes your fixed cost base in a completely different way to one that needs one.
A formula you can actually compute
Put those five variables together and you get something you can calculate rather than argue about. It is rough, it ignores a few second-order effects such as resale value and staff time, but it gets you from vibes to a number.
# illustrative structure, plug in your own measured numbers
onprem_monthly_cost = (hardware_capex / amortisation_months)
+ (avg_power_kw * 24 * 30 * electricity_rate_per_kwh)
+ fixed_ops_overhead
max_throughput_per_month = gpus_per_box * requests_per_gpu_per_second * seconds_per_month
onprem_cost_per_request = onprem_monthly_cost / actual_requests_served_per_month
cloud_cost_per_request = cloud_rate_per_request # or derive from a cloud GPU hourly rate
breakeven_requests_per_month = onprem_monthly_cost / cloud_cost_per_request
A worked example (illustrative only, not a benchmark)
Say you are looking at an eight-GPU server costing somewhere around 200,000 pounds, amortised over 36 months. That is roughly 5,556 pounds a month in hardware cost alone. Add power: call it 6kW average under real load including cooling overhead, at a commercial electricity rate of 0.25 pounds per kWh, which is about 1,080 pounds a month. Add a modest 2,000 pounds a month for rack space, networking and a slice of someone's time keeping it running. Total fixed cost: roughly 8,636 pounds a month, largely independent of how many requests you actually serve, up to the box's ceiling.
Now say the cloud alternative costs, illustratively, 0.0015 pounds per request for a comparable model size. Plug in your own actual quote here, this is purely to show the shape of the curve, not a real price.
At 1 million requests a month, your on-premise cost per request is 8,636 divided by 1,000,000, about 0.0086 pounds. The cloud option at 0.0015 pounds per request is nearly six times cheaper, because you are paying the box's full fixed cost to serve a trickle of traffic.
At 50 million requests a month, the same box now costs 8,636 divided by 50,000,000, about 0.00017 pounds per request. The cloud option at that volume would cost roughly 75,000 pounds a month. The on-premise box just won by a wide margin, because you are finally using the capacity you paid for.
Solve for the exact crossover and you get roughly 5.8 million requests a month, which is 8,636 divided by 0.0015. Below that, cloud is cheaper. Above it, on-premise is cheaper. Everything in this calculation depends on where your actual traffic sits relative to that one number, not on which option sounds more serious in an argument.
The sawtooth nobody mentions
The crossover point above is not a smooth line in practice, it is a sawtooth. The moment your request volume exceeds one box's capacity, you do not scale smoothly, you buy a second box, and your fixed cost jumps by another 8,636 pounds a month in one step before per-request cost starts falling again as that new capacity fills up. If your growth curve crosses a capacity boundary right before a slow quarter, you can end up paying for two boxes' worth of fixed cost while only filling one and a half boxes' worth of demand. Cloud does not have this problem, it scales in whatever increment the provider sells, usually much finer than 'another entire server'. That smoothness is part of what you are actually paying the cloud premium for, and it rarely shows up as a line item in anyone's TCO spreadsheet.
Where this actually matters
This is the exact spreadsheet I build with people evaluating whether to run inference on their own hardware instead of staying on a cloud API, including people looking at an on-premise deployment of what I build at Mickai. I always ask for the same two numbers first, measured request volume and utilisation, then show exactly where the crossover sits for those numbers and let the decision follow from the number rather than from an opinion. Anyone willing to answer the general question without asking for your request volume first is guessing.
The variable I see people get wrong most often is utilisation, because nobody measures it honestly before they buy hardware. They estimate it optimistically, then wonder a year later why the on-premise box never paid for itself. What is the messiest fixed cost you have had to fold into a calculation like this, and has anyone else actually hit the sawtooth problem in production and had to plan a hardware purchase around it?
Top comments (0)