Open or closed? The question shows up in every architecture review, and it usually gets settled with a benchmark chart and a price-per-token comparison. But it often gives the wrong answer, because the two options on the table are not the two you have.
There is a third sitting between them, and for most workloads it wins.
Why the binary is badly posed
The usual reasoning runs like this. An H100 rents for a few dollars an hour, while a frontier API charges a markup per token on that same hardware. So you run the weights yourself and keep the margin someone else is taking.
The catch is that "using an open model" and "operating an open model" got collapsed into one choice. So separate them, and the third option appears.
The three shapes
- Closed frontier API. Variable cost, zero operations, no access to the weights. Roughly $2.50 to $15 per million tokens (packet.ai, jul/2026).
- Hosted open-weight API. Variable cost, zero operations, and the same weights you would run. Roughly $0.07 to $0.90 per million tokens for Llama 4 and DeepSeek V4 class models (packet.ai, jul/2026).
- Self-hosted open weights. Fixed monthly cost, full control, and an operational surface you now own. Llama 4 70B on an H100 at batch=8 comes to about $0.18 per million output tokens (packet.ai, jul/2026).
Rows two and three run the same model, in the same order of magnitude. So the saving that justifies buying GPUs is not the one against the frontier API, it is the one against a provider already serving those weights, and there it mostly disappears.
The number that decides it is utilization
That $0.18 figure assumes the GPU runs at 85% utilization with vLLM continuous batching, against 30 to 40% for naive serving (packet.ai, jul/2026). Meanwhile, average GPU utilization across production Kubernetes fleets sits at 5%, and the best cluster in that dataset, a 136-node H200 deployment, reached 49% (Cast AI, State of Kubernetes Optimization Report 2026).
That gap is the whole argument. A rented GPU bills the same idle as saturated, so cost per token scales inversely with utilization. Drop from 85% to 5% and the same setup costs roughly seventeen times more per token.
Then add operations. Version upgrades, OOM debugging, monitoring and capacity planning run 10 to 20 hours a month, between $1,500 and $4,000 that never shows up in the GPU bill (packet.ai, jul/2026).
Which is why published break-even points disagree so much. Against managed frontier APIs, self-hosting on reserved capacity breaks even around 2 to 5 million tokens a day over 12 months (packet.ai, jul/2026). Against a hosted endpoint running the same weights, it moves out to 50 million or more (howaiworks, jul/2026).
And none of this is unique to inference. It is the same trap as provisioned capacity in a data platform, easier to see in Microsoft Fabric capacity units, where the meter runs on what you reserved and not on what you used.
Run it on your own numbers
HOURS_PER_MONTH = 730
def monthly_self_host(gpu_hourly, n_gpus, ops_multiplier):
"""Fixed. Billed by the hour, never by the token."""
return gpu_hourly * HOURS_PER_MONTH * n_gpus * ops_multiplier
def breakeven_tokens(gpu_hourly, n_gpus, ops_multiplier, price_per_mtok):
"""Monthly token volume where self-hosting matches the API bill."""
fixed = monthly_self_host(gpu_hourly, n_gpus, ops_multiplier)
return fixed / price_per_mtok * 1_000_000
def monthly_capacity(tokens_per_sec, n_gpus):
"""Ceiling. Use sustained throughput measured at your real batch size."""
return tokens_per_sec * n_gpus * 3600 * HOURS_PER_MONTH
Run breakeven_tokens twice, once against your frontier API price and once against a hosted endpoint serving the model you intend to run. Then check both against monthly_capacity, because a break-even you cannot reach is not a break-even.
- Below your volume and inside capacity. Self-hosting has a case, so load test the throughput figure before anyone reserves an instance.
- Above capacity. Reaching it means adding GPUs, which raises the fixed cost, which pushes break-even further out. At that price the lines never meet.
- Far above your volume, measured against the hosted endpoint. The open model wins and running it yourself does not.
What the math does not settle
Cost is loud, but compliance is louder. Data residency, contractual confidentiality, a no-training guarantee or a hard latency floor can close the question before the spreadsheet opens. That bites hardest in Europe, where routing inference through US platforms raises transfer questions under Chapter V of the GDPR (Lyceum Technology, ago/2026). If the weights have to stay inside your network, break-even is beside the point.
Although it cuts the other way too. Self-hosting to satisfy a requirement that a zero-retention agreement already covers buys permanent operational load for a guarantee you had on paper.
So, three options or one routing rule?
In practice it lands as a split. A small open model takes the high-volume, well-shaped work, so classification, extraction and routing, while the frontier API stays reserved for reasoning it cannot handle. Cost drops because the cheap path absorbs the traffic, and quality holds because the expensive path catches the hard cases.
Which turns open versus closed into a routing decision made per workload, rather than an architecture decision made once and defended for two years.
Top comments (2)
The utilization number is the whole post. Everyone benchmarks self-hosting against 85% GPU utilization because that's what the vendor's own case study assumes, then deploys at 5-10% because real traffic is spiky, not steady. We hit this exact wall building GTWY's routing layer: the "self-host is cheaper" math only holds if you can keep the box busy, and most teams can't without pooling traffic across multiple use cases they don't actually have yet. The routing-per-workload conclusion is right, but I'd add: build the routing logic before you buy the GPU, not after, because once you own the hardware the sunk cost quietly overrides the routing decision every time.
The operating-vs-using distinction is the one I wish more architecture reviews would force onto the table. "We run open models" means completely different things depending on whether someone is paging at 3am about KV-cache eviction or just calling a managed endpoint.
From running real workloads: the third option you describe is where almost all of my traffic landed. Dedicated capacity I own sits idle between bursts, and the per-token price of managed open-weight endpoints is now low enough that the math only flips back at sustained volume. What the article doesn't price in is cold-start latency and the operational tax of two inference paths — that cost showed up for me more often than the token cost.
The H100-margin argument as commonly stated also quietly assumes full utilization, which is exactly what most teams don't have. Good framing overall.