The GPU sticker price is 30 to 40 per cent of what an inference cluster really costs. The rest is electricity at facility prices rather than nameplate watts, cooling, interconnect fabric, storage, people, and the spare capacity you hold for traffic peaks. Here is the full three-year model, a fillable formula, and a worked example for an 8x H100 node in Dubai, showing why utilization swings the answer more than every other variable combined.
The total cost of ownership of a GPU inference cluster is set by three things: PUE-adjusted electricity, operations headcount and utilization, not by the GPU price. The GPU line on the purchase order is the smallest, most negotiable part of the bill. Recent TCO analysis puts the silicon at 30 to 40 per cent of true self-hosted infrastructure cost, with real monthly spend running 2.5 to 3 times the hardware cost once staffing and operations are counted (HostRunway). If you have already decided to buy inference GPUs, the number you take to the board has to cover electricity at facility prices rather than nameplate watts, cooling, the interconnect fabric, storage, people, and the spare capacity you hold for traffic peaks. This article gives you the whole model: a fillable formula, every line item with a source you can defend, a worked example for an 8x H100 node in Dubai, and the one variable that swings the answer more than all the others combined: utilization.
This is deliberately not the buy-versus-rent question. Whether on-prem beats an API at your token volume is a separate decision, and the sources below cover it well. Here we assume the build decision has been made, often for reasons that override price: data residency, sovereignty, latency floors or model control. Your job now is to produce a three-year number that survives a finance review.
GPU inference cluster TCO: the fillable formula
The cleanest formulation comes from Kentino's on-prem cost model: three-year total cost divided by tokens actually served, where the denominator depends on utilization and the numerator is mostly fixed (Kentino):
cost per Mtok =
(capex + 3y electricity + 3y maintenance + 3y ops)
/ (sustained tok/s x utilization x 3 x 8,760 x 3,600 / 1e6)
In code, so you can drop it into your own model:
def tco_per_mtok(
capex, # servers, GPUs, fabric, storage, rack, install
cluster_kw_it, # total IT draw of the cluster, kW
pue, # measured or assumed facility PUE
kwh_price, # all-in price per kWh: tariff + surcharges + VAT
ops_fte, ops_cost_fte, # headcount and fully loaded annual cost each
software_per_year, # OS, orchestration, monitoring, support contracts
years=3,
sustained_tok_s=3500, # measured throughput at saturation
utilization=0.60, # fraction of wall-clock time at that throughput
):
seconds = years * 8_760 * 3_600
energy_kwh = cluster_kw_it * pue * 8_760 * years
tokens_mtok = sustained_tok_s * utilization * seconds / 1e6
total = (capex
+ energy_kwh * kwh_price
+ years * (ops_fte * ops_cost_fte + software_per_year))
return total / tokens_mtok, total
Two modelling choices are built in and worth stating openly in your paper:
- Electricity is charged at full wall-clock hours, not utilization-scaled hours. An inference GPU at low load still draws most of its power; an H100 server at 10 per cent request volume is not a 600 W machine. Charging full hours is the honest, conservative assumption.
- Everything except electricity is fixed. That is precisely why utilization dominates, which we quantify below.
Hardware: the easy 30 to 40 per cent
Start with the numbers that are actually quotable. An NVIDIA H100 SXM currently runs $25,000 to $35,000 per unit; a production deployment with redundancy and reasonable throughput for a 70B-class model typically means 8 GPUs, putting the accelerator bill at $200,000 to $280,000 before chassis, networking and storage (Corporate AI Consultants). Fully configured 8-GPU H200 systems quote at $400,000 to $500,000, and HGX B200 nodes around $450,000 when you can secure allocation; those stickers exclude networking uplift, storage, rack power distribution and cooling retrofits (LLM.co).
The trap is treating the sticker as the capex. Add, at minimum:
- server chassis, CPUs, host RAM and NVSwitch backplane
- top-of-rack and spine networking, plus high-speed fabric if you go multi-node (InfiniBand connectivity alone adds roughly $2,000 to $5,000 per node)
- storage for model weights, checkpoints and logs, plus NVMe scratch if you do any fine-tuning
- racks, PDUs, UPS and installation
A workable planning rule: total infrastructure capex lands at 2.5 to 3 times the raw GPU investment. A "modest 8-GPU cluster" that looks like $240,000 in silicon is realistically a $600,000 to $800,000 project before it processes a single token (MPT Solutions).
PUE and electricity: raw watts are not the number
This is the line item most budgets get wrong twice: once on power draw, once on facility overhead.
Power draw. Eight H100 GPUs alone consume over 5.6 kW, and once you add CPUs, NICs and the rest of the node, draw pushes past 10 kW (LLM.co). Most colocation cages and enterprise data halls were provisioned for 4 to 8 kW racks, so densification, PDU upgrades and rear-door heat exchangers become line items that never appear on the GPU quote.
PUE. Power usage effectiveness is the ratio of total facility power to IT power. A PUE of 1.6 means every watt delivered to the server costs you 1.6 watts at the meter: the extra 0.6 is cooling, power distribution and losses. The 2025 Uptime Institute Global Data Center Survey, drawing on more than 800 operators, puts the industry average PUE at 1.54, effectively flat for the sixth consecutive year, with facilities commissioned in the last five years averaging 1.48 and sites above 20 MW around 1.44. Uptime attributes the stagnation partly to "climate specific limitations to efficient cooling", which is exactly the caveat that matters here: a purpose-built hall in a cool climate can hit 1.3 or better, but an enterprise room in the Gulf will not. Unless your facility publishes a measured figure, model 1.5 to 1.7 for the GCC and expect the low end only in a modern purpose-built site (Uptime Institute).
For context on how good it can get with hyperscale engineering: Google reports a trailing twelve-month fleet PUE of 1.09 (Google). You will not match that with one rack. Do not let a vendor slide it into a spreadsheet.
The price per kWh, in the UAE. In Dubai, DEWA's slab tariff tops out at AED 0.380 per kWh above the consumption band, plus a fuel surcharge (AED 0.060 per kWh as of November 2026) and 5 per cent VAT, putting the upper band at roughly AED 0.462 per kWh, about $0.126 (DEWA slab tariff). A GPU cluster consumes far past the top slab every month, so that upper band is your planning rate. In Abu Dhabi, industrial users above 1 MW pay 27 fils per kWh off-peak and 36.6 fils during summer daytime peak hours (10:00 to 22:00, June to September), a time-of-use structure that rewards shifting batch workloads to nights (ArusGrid). Abu Dhabi's Department of Energy is also designing a dedicated data-centre tariff, with rates not yet published, so check before signing a long lease.
Worked power math for a single 10 kW IT node at PUE 1.6, Dubai:
facility draw = 10 kW x 1.6 = 16 kW
annual energy = 16 kW x 8,760 h = 140,160 kWh
annual cost = 140,160 x AED 0.462 = AED 64,750 (~$17,600)
three years = ~AED 194,000 (~$52,900)
If colocation is your model instead, budget $200 to $400 per kW per month for high-density hosting, so a 10 kW node costs $24,000 to $48,000 per year, typically with power bundled in (Corporate AI Consultants).
Cooling, fabric, storage and the rest of the plant
Beyond the interconnect costs covered under capex, three items deserve their own rows:
- Cooling beyond PUE. If you are building on-prem rather than colocating, a data-centre build-out including power distribution, cooling upgrades and rack space can exceed $50,000 per rack (Corporate AI Consultants). In Gulf ambient conditions, liquid cooling or rear-door heat exchangers stop being optional somewhere past 20 to 30 kW per rack.
- Storage and networking opex. For a single-node deployment, HostRunway's TCO model budgets $100 to $300 per month for storage and networking, plus $100 to $200 per month for inference software and observability (HostRunway). Model weights, checkpoints and eval traces grow faster than teams expect once fine-tuning starts.
- Redundancy. An inference cluster with a single node has no failure story. N+1 means either a second node or a cloud fallback path, and both cost money. Pick one deliberately rather than discovering the gap during an outage.
People: the line that usually decides the model
On-prem inference needs specialized operations: model serving, quantization, GPU cluster management, driver and firmware updates. Depending on existing capability, plan for one to three additional FTEs at $150,000 to $250,000 fully loaded per person per year in most markets (Corporate AI Consultants). Kentino's per-server estimate is 0.1 to 0.3 FTE per server, plus a redundant box or cloud fallback for downtime (Kentino).
The practical floor for one 8-GPU node with production traffic is roughly one FTE. In HostRunway's single-node model, a 0.25 FTE DevOps/MLOps allocation costs $3,000 to $5,000 per month against a total TCO of $6,930 to $10,790 per month, based on PUE 1.4, US electricity at $0.12 to $0.13 per kWh, and a senior engineer at $192,000 fully loaded (HostRunway). Even at the lowest staffing fraction, people are 40 to 50 per cent of the running bill. This is why the GPU quote feels so misleading in hindsight.
Software and support contracts sit alongside people: enterprise Linux, orchestration, monitoring and optionally a commercial serving platform add $20,000 to $100,000 per year depending on scale and vendor choices (Corporate AI Consultants).
Utilization: the single biggest swing factor
Here is the argument in one sentence: nearly all of your three-year cost is fixed, so cost per token is inversely proportional to the tokens you actually serve, and utilization is the only lever that moves the denominator.
Kentino's sensitivity analysis makes the shape explicit. At 60 per cent utilization a given build serves a certain token volume at a certain euro per Mtok; drop utilization to 30 per cent and every per-token figure doubles; push to 80 per cent and one reference build (8x L4 serving an 8B model) lands at €0.09 per million tokens, cheaper than any cloud API in its class. Below roughly 25 per cent utilization the math collapses, because TCO is dominated by capex you paid whether or not you used it (Kentino).
The brutal constraint is traffic shape. Traffic that only arrives during working hours, 40 of the week's 168 hours, keeps a GPU busy at most 24 per cent of the time before you even account for lunchtime dips (OpenRouter). A GCC enterprise with a 9-to-6 internal assistant and a 4:1 peak-to-average ratio is structurally near that 24 per cent ceiling. This is why the spare-capacity decision is a cost decision, not just an SRE decision:
- Serve everything on-prem, sized for peak: utilization lands around 15 to 25 per cent, and cost per token roughly triples versus a well-utilized cluster.
- Baseline on-prem, burst to API: size the cluster for the sustained floor (aiming at 60 to 80 per cent utilization) and let a hosted endpoint absorb peaks. You pay a premium per token only on the spiky tail.
The research side reinforces how little of the self-hosted cost is electricity. Measured on consumer Blackwell hardware, power-only cost runs $0.001 to $0.04 per million tokens depending on workload, with hardware amortization adding $0.01 to $0.05 per Mtok over two years; the rest of a real deployment's cost is everything around the card (arXiv 2601.09527). Electricity is a rounding error; people and idle silicon are the bill.
Worked example: an 8x H100 node in Dubai, three years
Scenario: a UAE enterprise runs an internal knowledge assistant and a document-processing pipeline on an open-weight 70B model. Data residency rules out hosted APIs for the production prompts. One 8x H100 SXM node, own data hall in Dubai.
Capex. GPUs at $30,000 each (mid-point of the $25,000 to $35,000 range): $240,000. Chassis, NVSwitch, NICs, storage, rack and install: $60,000. Total $300,000, amortized straight-line over three years.
Electricity. 10 kW IT draw, PUE 1.6 (an honest Gulf enterprise figure), DEWA upper band at AED 0.462 per kWh all-in: AED 64,750 (~$17,600) per year, $52,900 over three years.
People. 1.0 FTE platform/MLOps engineer at $150,000 fully loaded (the low end of the market range): $450,000 over three years.
Software and support. $30,000 per year (inside the $20,000 to $100,000 published range): $90,000.
Three-year TCO: $892,900.
| Line item | 3-year cost | Share |
|---|---|---|
| GPU capex (8x H100 SXM) | $240,000 | 27% |
| Chassis, fabric, storage, rack | $60,000 | 7% |
| Electricity (PUE 1.6, DEWA) | $52,900 | 6% |
| Ops (1.0 FTE) | $450,000 | 50% |
| Software and support | $90,000 | 10% |
| Total | $892,900 | 100% |
The GPUs are just over a quarter of the bill here; only at larger scale with shared ops do they approach the 30 to 40 per cent range. The engineer is half of it. Anyone quoting only the first two rows is not quoting the cluster.
Throughput and the utilization table. An 8x H100 node serving Llama 3.3 70B in FP8 sustains roughly 3,500 tokens per second (Kentino). Over three years that is up to 331 billion tokens at 100 per cent utilization; real deployments land far below.
| Utilization | Tokens served (3y) | Cost per Mtok |
|---|---|---|
| 24% (office-hours traffic) | 79.5 B | $11.23 |
| 30% | 99.3 B | $8.99 |
| 60% | 198.7 B | $4.49 |
| 80% | 264.9 B | $3.37 |
From 80 per cent down to 30 per cent utilization, cost per token more than doubles with no change to any other number. From 60 to 30, it doubles exactly, because 94 per cent of the spend is fixed.
What it competes against. At 60 per cent utilization the cluster delivers tokens at $4.49 per million. Against a hosted open-model API serving the same Llama 3.3 70B at roughly $0.88 per Mtok (Kentino), the API serves the same 198.7 billion tokens for about $175,000: the cluster loses by roughly 5x, and the residency requirement is the only thing that justifies it. Against a frontier API at a blended $9 per Mtok, the same volume costs $1.79 million: the cluster wins by 2x. So state the comparison honestly in your paper. The cluster competes with frontier and regulated workloads, not with the cheapest open-model endpoint.
One more nuance worth a row in your model: context length. Measured energy per token rises roughly 5.5x from 8k to 32k context, and doubling context approximately halves throughput (arXiv 2601.09527). A RAG-heavy workload at 32k context serves a fraction of the tokens per second of a chatbot workload on identical silicon, which pushes effective utilization and cost per token in the wrong direction simultaneously. Chunk retrieval aggressively.
What to do next
- Measure before you model. Two weeks of instrumented token counts, split by model tier and by hour of day. Without the peak-to-average ratio you cannot pick a utilization figure, and utilization decides the answer.
- Get a measured PUE, not a marketing PUE. Ask the facility for a trailing twelve-month figure. If none exists, use 1.5 to 1.7 for the Gulf. Never accept 1.1 in a spreadsheet for a single-rack deployment.
- Price electricity at your meter. In Dubai, model AED 0.462 per kWh (top slab plus November 2026 fuel surcharge plus VAT). In Abu Dhabi, model the time-of-use split, and consider shifting batch inference into off-peak hours.
- Run the formula at three utilizations: your measured average, 30 per cent and 60 per cent. If the 30 per cent number frightens you, the answer is not a smaller cluster but a hybrid: baseline on-prem at high utilization, burst overflow to a hosted endpoint.
- Budget the FTE explicitly. If the organization cannot staff it, the cluster will sit idle at 10 per cent utilization, which is the most expensive way to own GPUs.
- Stress the residency argument, not just the price argument. At GCC volumes the cluster often loses to cheap open-model APIs and wins against frontier APIs. The defensible case is usually sovereignty plus latency plus control, with cost per token as supporting evidence.
This is the model we run with clients before any hardware is ordered: platform assessment and design first, then the build, then running it, with our own team in Dubai doing the consulting, deployment and support (Azrty AI infrastructure). If you want a single control point for the hybrid pattern, a gateway in front of your own LLM servers and hosted providers keeps routing, budgets and access control in one place (FastLLM Proxy).
The single sentence to remember: the GPUs are the part of the cluster you negotiate once; utilization, PUE and headcount are the parts you pay for every day of the three years. Put all three in the model before the number goes to the board.
Originally published on Azrty.
Top comments (0)