The Meter Is Always Running

For forty years, enterprise software rested on a comfortable assumption: that it was a fixed cost. You paid a license, amortized it over a depreciation schedule, and every unit of value you squeezed out afterward felt like it was approaching free. Seats, not usage. Capex, not cost of goods. A CFO could draw a straight line from spend to return and sleep at night.

AI breaks that line. Most organizations adopted it without noticing.

The thing that actually changed

The interesting shift in enterprise AI is not cloud versus on-premises. That is a deployment question, and deployment questions are the kind of thing infrastructure teams have solved a hundred times. The deeper shift is quieter: software stopped being a fixed cost and became a variable one.

A large language model does not bill like a license. It bills like a taxi. The meter runs on every token, upstream and downstream, and input and output are priced per million tokens at rates that differ by model and by direction. The fare depends on how long the conversation ran, how much reasoning it demanded, how many times a user hit retry. Two customers doing the "same" task can cost 10x different amounts because one of them pasted a novel into the prompt. There is no seat count that caps it. There is no version you buy once and freeze. The cost scales with success: the more people use the feature, the more it costs you to have built it.

That is a new financial object, and the mental model most companies applied to it, SaaS licensing or depreciable capital, is the wrong shape. You cannot amortize a variable cost. You can only manage it.

The on-prem mirage

The popular rebuttal goes like this: just run it yourself. Buy the hardware once, own it forever, and your marginal cost per query approaches zero. Convert the scary variable cost back into a familiar fixed one.

It is a seductive move, and it is mostly wrong.

The "own it forever" asset is not the GPU. It is the model, and the model is spoiling in real time. The cluster you justified this year runs a frontier model that will be mid-tier in twelve months. The hardware ages too: how long an AI accelerator stays economically useful is now an open argument on Wall Street, with investors questioning whether the six-year depreciation schedules on the books match a reality where a new generation lands every eighteen months. You did not buy a depreciable asset amortized over seven years. You bought a seat on a refresh treadmill, plus a standing bill for power, cooling, and the engineers who keep it fed.

And unlike the cloud meter, that silicon costs the same whether it runs at 90% load or 5%. Idle capacity is the norm, not the exception: it is common for GPU clusters to sit 70 to 80% idle between bursts of real work. You did not eliminate the variable cost. You traded a usage-based meter for a utilization risk, and for most enterprises, whose demand is spiky rather than steady, that is the worse trade.

Marginal-cost-to-zero is real, but only at high, sustained load on a model you are content to keep running. That describes a narrow band of workloads. It does not describe "our AI strategy."

Inference is cost of goods, not capex

Here is the reframe that survives contact with a spreadsheet: inference is cost of goods sold.

Not a license you buy. Not a machine you depreciate. It is a per-transaction input cost, like the cloud compute behind a web request or the fee on a card payment. Companies already know how to run a business where serving each customer costs money. They have gross-margin discipline, unit economics, dashboards that flag when a product line goes underwater. They simply never had to apply that discipline to software features before, because software features used to be free to serve once written.

File inference under cost of goods instead of "IT capex," and the right behaviors fall out on their own. You start asking the questions a margin-conscious operator always asks.

What that discipline looks like

Know your unit cost per feature. If you cannot say what a single run costs, you are blind on the one number that decides whether the feature is a business or a liability. Instrument tokens the way you already instrument latency.

Then route by need. Most requests do not warrant a frontier model; send the cheap, frequent, predictable work to smaller or local models and reserve the expensive reasoning for the calls that actually earn it. A good router is worth more than a bigger model.

Cache aggressively. Repeated system prompts, retrieved context, common answers: every hit is a fare you skip. Anthropic reports that prompt caching can cut input costs by up to 90% and latency along with it. Caching is not a nice-to-have optimization here. It is margin.

Put a ceiling on it. Variable cost without guardrails is how one looping agent turns into a five-figure surprise. Budget caps, rate limits, and per-tenant quotas are the circuit breaker. Build them before the invoice, not after.

And decide placement per workload, not per company. Some things belong on a local model you own; some belong on a metered API. "Cloud-first" and "on-prem-first" are both the wrong altitude. The unit is the workload, and the answer is a portfolio.

The reckoning is real. It is not the one on the thumbnail.

A reckoning is coming for a lot of AI initiatives, but it will not arrive as the dramatic collapse of anyone's cloud strategy. It will arrive as a quiet quarterly review where someone finally divides the AI line item by the number of customers it served and does not like the answer. The projects that survive that meeting will not be the ones that picked the right deployment location. They will be the ones that treated inference as what it is, a variable operating cost with no natural ceiling, and built the discipline to manage it from the first day.

The meter was always running. The only question is whether you were watching it.