TL;DR — Per-token pricing hides the actual cost driver behind AI infrastructure: GPU capacity is a lumpy, reserved, fixed-cost asset, and the true marginal cost of a token depends entirely on how well that capacity is utilized. Low utilization silently inflates real cost per token far above what API price sheets imply. Teams that manage AI economics by watching token throughput instead of the utilization curve are optimizing the wrong number.
Every AI pricing page speaks the same language: dollars per million tokens. It's a clean abstraction, and it's almost entirely disconnected from how the underlying cost actually behaves. Tokens are the unit customers see. Utilization is the unit that determines whether serving those tokens makes or loses money. Confusing the two is the single most common mistake in how teams reason about AI infrastructure economics.
The Illusion of a Marginal Cost
Per-token pricing implies a marginal-cost model: each additional token costs roughly the same to produce, so you can price linearly and reason about margin per request. That model works for things like bandwidth or storage, where capacity is elastic and finely divisible. It breaks down for GPU-served inference, because GPU capacity isn't divisible in any way that matches token volume. You don't rent a sliver of a GPU per request. You reserve nodes, in advance, in fixed increments, often for hours or months at a time, based on a demand forecast that is wrong the moment you make it.
That means the real cost structure looks less like a metered utility and more like owning a fleet of trucks. The cost of the trucks is fixed once you've leased them. What varies is how full each truck is when it leaves the depot. A truck that leaves half-empty costs the same as one that's full — you just earned less revenue per mile driven. GPU serving fleets behave the same way. The token is the cargo. Utilization is the load factor. And most teams are reporting cost-per-token numbers that assume the truck is always full.
Utilization Is the Actual Cost Driver
Take a cluster provisioned for a target load, and compute the true cost per token as fixed infrastructure spend divided by tokens actually served. At high utilization, that number can look close to the advertised price. At the utilization levels most teams actually run — dominated by bursty traffic, uneven request lengths, and safety margin for peak load — the real cost per token can be several times higher than the sticker number implies. Nobody puts that number on a slide, because it depends entirely on operational discipline, not architecture.
This is why two teams running the identical model, on identical hardware, with identical published token prices, can have completely different unit economics. One runs at high sustained utilization because it aggregates diverse traffic across many customers and time zones. The other runs at low utilization because it serves a single internal application with spiky, predictable-only-in-retrospect demand. The model is the same. The economics are not even close.
Why LLM Serving Makes This Worse Than It Looks
Traditional web infrastructure has a decades-old playbook for utilization: autoscale horizontally, spin up commodity instances in seconds, spin them down when idle. LLM serving breaks that playbook in three specific ways.
Cold start is expensive and slow. Loading model weights onto a GPU, warming up kernels, and populating a KV cache is not a five-second operation. Autoscaling reaction time on web services is measured in seconds; on large model serving it's measured in minutes. By the time new capacity is online, the traffic spike that triggered it may already be over.
Requests are not uniform units of work. A token-based mental model assumes fungibility, but a long-context request and a short chat completion consume wildly different amounts of memory and compute per token. Provisioning for "requests per second" without accounting for this variance guarantees either wasted headroom or missed SLOs.
Batching creates a latency-utilization tradeoff that has no equivalent in stateless web serving. You can raise utilization by waiting to batch more requests together, but every millisecond of batching delay is a millisecond added to someone's response time. Utilization and latency compete directly for the same dial, and turning that dial is a business decision disguised as an infrastructure setting.
None of this shows up in a token-price comparison. It all shows up in the utilization curve, which almost nobody outside the infrastructure team ever looks at.
The Pricing Decisions This Actually Drives
Once utilization is understood as the real cost driver, several decisions that look like infrastructure problems turn out to be pricing and product problems in disguise.
Reserved versus on-demand capacity isn't just a procurement choice — it's a bet on how confidently you can predict and smooth your own demand curve. Reserved capacity is only cheap if you can keep it busy; otherwise it's a fixed loss dressed up as a discount. Spot or elastic capacity is only cheap if your workload can tolerate interruption and the slower provisioning latency, which for interactive inference is often not true.
Discounted pricing for off-peak or batch usage isn't a marketing gimmick, it's utilization arbitrage: it exists specifically to fill trucks that would otherwise leave empty. Any team offering flat per-token pricing regardless of time-of-day load is either subsidizing off-peak users with peak-hour margin, or quietly building in enough headroom that peak-hour margin barely exists.
And internal cost allocation — the perennial argument about which team "owns" the GPU bill — is almost always fighting over the wrong number. Charging teams by tokens consumed without accounting for whose traffic pattern is dragging fleet utilization down rewards spiky, inefficient usage and penalizes teams whose steady, predictable load is what makes the shared infrastructure viable in the first place.
What to Actually Monitor
If utilization is the real cost lever, it needs to be a first-class observability signal, not a quarterly finance exercise. That means tracking GPU-hours paid for versus GPU-hours productively serving traffic, at the same granularity you'd track latency or error rate. It means separating "tokens served" from "tokens served per dollar of reserved capacity," because the first number can grow while the second quietly collapses. It means treating batching window size, request admission policy, and autoscaling lag as economic levers with dollar values attached, not just performance-tuning knobs.
The token-based price sheet will keep being the number customers see, because it's simple and it's comparable across vendors. But it should never be the number an infrastructure team uses to reason about its own economics. The price of a token is a story you tell the market. The cost of a token is a story your utilization curve tells you, whether you're listening or not.
Top comments (0)