I run a dedicated local inference host. A Mac mini M4 Pro, 14-core CPU, 20-core GPU, 64GB of unified memory, bought open-box for $3,349. It serves embeddings for a retrieval system, a fallback chat model, and a classifier for inbound notes. It has eight models resident, about 140GB on disk.
Over the past three weeks it has been busy 1.7% of the time.
That number comes from its own request log: the summed wall-clock duration of every generate, chat, and embed call, divided by the day. On the heaviest day in the window it hit 3.45%. On several days it did no inference at all. The daily request count looks healthy at roughly 1,440, until you notice that 1,440 is exactly one request per minute: an uptime monitor polling /api/version. The machine's most reliable customer is a health check.
I would still buy it again. But it broke my own cost model, and the way it broke is the part worth writing down.
The month-0 spreadsheet is not the month-12 bill
Almost every local-versus-cloud comparison, including the one I wrote last year, is a snapshot. You price the hardware, price the tokens, divide, and find a break-even volume. That math is fine on the day you run it. It quietly assumes three things that stop being true somewhere around month four.
The gap is not small. McKinsey's July 2026 survey found 93% of respondents exceeding their AI budgets while 62% had moved past experimentation into active deployment. McKinsey puts 20 to 30% of that spend within reach of better accounting. That is a survey of self-reported budgets, so treat the precise figure as directional. The direction is the interesting part: the overruns arrive after deployment, not during the pilot. These are the three line items that show up in that window.
Line item 1: the capacity you bought, not the capacity you used
Cloud inference bills you for work performed. Hardware bills you for work available. At 1.7% utilization those are different products wearing the same label.
Run my numbers forward. Roughly 0.4 busy hours a day is about 146 hours of actual inference a year. Amortize the $3,349 over a single year and each hour of real work costs $22.90. Stretch it across three years, which is fairer to the hardware, and it is still about $7.60 per busy hour. Electricity barely registers. The mini idles near 7W and peaks around 45W under load, so 8,600 idle hours plus 146 busy ones comes to roughly 67 kWh, about $23 a year at California rates.
That last figure is worth pausing on, because my earlier post led with cooling and power draw. At rack scale, with 700W GPUs, that emphasis is right. At the scale most teams actually start at, one box serving one team, power is a rounding error and utilization is the entire story. The line item that matters flipped when the deployment got smaller.
The honest version of the local-inference pitch is not "tokens are cheaper." It is "I am buying a fixed monthly cost and a latency floor, and I will pay it whether or not anyone sends a request."
Line item 2: the cloud price is a moving target and your hardware is not
You commit to hardware at a fixed price. The alternative you rejected keeps getting cheaper underneath you, and not evenly.
Epoch AI's analysis of inference pricing found that prices to reach a fixed performance level have fallen between 9x and 900x per year, depending on which capability you are buying. Matching GPT-4's performance on PhD-level science questions got about 40x cheaper per year. General-knowledge performance fell more slowly, in the 9x-to-40x band. Epoch adds a caveat most people quoting these curves skip: the fastest declines happened in the most recent period, so there is no guarantee they persist.
Two consequences for a 12-month decision. The break-even volume you computed in month 0 moves against local hardware every month you own it, because your capex is sunk while the rented alternative reprices. And the rate it moves depends on your workload, because the decline is task-dependent. A team doing hard reasoning has seen its cloud alternative collapse in price. A team doing bulk classification has seen a gentler slope, and their local box holds its case longer.
Nobody's model has a row for this. Build one: re-run the comparison quarterly with current API pricing rather than the pricing you started with.
Line item 3: you pay per token and you bank per accepted answer
The third omission is the denominator. Both sides of this comparison are usually priced per million tokens, which measures what the model emitted, not what a human kept.
I made this argument in isolation two weeks ago and it landed flat, because an argument without a table is just an assertion. Here is the table. A smaller local model that needs two attempts where a frontier API needs one has doubled its effective token cost, and it has spent your reviewer's attention twice. Reviewer attention is the expensive input here, not the tokens. Tracking where the token spend actually goes is the prerequisite; the acceptance rate is what turns that into a cost.
| Line item | Cloud API | Local hardware | How to measure it |
|---|---|---|---|
| Idle capacity | ~$0 when idle | Full cost regardless of use | Busy-seconds ÷ elapsed, from your server log |
| Price decay | Falls 9x–900x/yr, task-dependent | Fixed at purchase; capex is sunk | Re-price the API side quarterly |
| Rework | Per token, retries billed | Per token, retries billed in idle capacity too | Accepted outputs ÷ total generations |
Three rows, all measurable from data you already have. None of them appear in a per-token comparison.
Where this breaks
My 1.7% is a single-operator homelab, and it is the weakest possible case for owned hardware. A shared inference cluster behind a real queue, serving fifty engineers across time zones, lands far higher. Above roughly 40% sustained utilization the capacity argument inverts and hardware wins on cost alone. If your box is genuinely saturated, line item 1 is not your problem.
Two other cases where none of this decides anything. Data residency and compliance can make local inference the only lawful option, at which point the cost comparison is a formality. And batch or overnight work reshapes the math, because idle hours you can schedule into are not idle.
There is also a cost to my own advice. Measuring utilization honestly makes the case against a machine I own and use daily. I kept it, for reasons that never appear in a TCO model: no rate limits, no per-request thinking, models that stay put, and a place to run things I would not send to an API. Those are real. They are just not "cheaper," and calling them cheaper is how a $3,349 box turns into a line item somebody has to defend at renewal.
Go pull your own number first. If you run Ollama, your utilization is sitting in ~/.ollama/logs/server.log right now, and it takes one awk to find out whether you bought a workhorse or a very expensive uptime-monitor endpoint.
Top comments (0)