You cannot predict what an LLM call will cost before you make it

focxle — Mon, 10 Aug 2026 12:19:49 +0000

I spent a week building spending caps for AI agents on an assumption that turned out to be wrong, and the way it was wrong is more interesting than the feature.

The assumption is the obvious one. Before making a model call, estimate what it will cost. If that estimate breaks the budget, refuse the call. Every budget system works this way: check, then spend.

Then I measured the estimate against real calls, and it does not work. Not "needs tuning". Does not work, and cannot be made to.

The measurement

12 calls recorded through OpenRouter against openai/gpt-oss-20b:free, 9 of which had a comparable pre-call estimate. For each, I compared what the estimator predicted against what the vendor's own usage block reported afterwards.

	estimate ÷ actual
median	3.87x too high
worst overestimate	61x too high
worst underestimate	0.045x, or 22x too low

Then I replayed those 9 calls against a $0.005 per-call cap and asked how many the estimator would have refused, where the real cost turned out to be under the cap:

4 of 9 legitimate calls wrongly refused.

At a $0.05 cap, nothing was refused at all. Which sounds like the fix, until you look at which calls were underestimated.

Both directions is what kills it

If the estimate were consistently high, you would divide by a constant and move on. If it were consistently low, you would multiply.

It is neither. The two calls in my sample that did not declare max_tokens came in 18x and 22x too low.

That is the whole problem in one sentence. Tighten the cap and it refuses ordinary work. Loosen it until it stops doing that, and it waves through precisely the unbounded calls a cap exists to stop, because those are the ones it underestimates.

The estimator is most wrong exactly where being wrong is most expensive.

Why it is unfixable, not undertuned

Input tokens are knowable. You have the prompt, you can count it.

Output tokens are not knowable. That is not a gap in the estimator, it is a property of generation: the model decides how much to write while it is writing. "Summarise this" can return one line or four paragraphs, and the difference is 40x in cost.

No overhead constant fixes an unknown that varies 60x in one direction and 22x in the other. No rolling percentile fixes it either, because the p95 of a distribution with that spread is either useless as a limit or useless as a prediction.

People do reach for max_tokens here, and it does bound the worst case. But it bounds it at a ceiling nobody sets tightly, because setting it tightly truncates real answers. A ceiling of 4096 on a call that returns 200 tokens prices at 20x reality, which is the 61x row above.

What works instead

Enforce against spend that has already been reconciled.

After each call, the vendor tells you exactly how many tokens it used. Price that against a rate table and add it to a running total. When the total crosses the cap, stop.

Reconciliation error in the same measurement: 0. Not 0.1%. The vendor tells you the number, so there is nothing to estimate.

The tradeoff is real and worth stating plainly: you can overshoot by one call. The cap is checked before a call using spend up to that point, so the call that crosses the line still completes. You are trading exactness for the ability to work at all.

That is a good trade, because "one call of overshoot" is bounded and knowable, and "refuses 44% of legitimate work while missing the runaways" is neither.

The one place the estimate still earns its keep

A per-call ceiling.

If a single call could cost more than some absolute number, refuse it before making it. Here the estimate is fine, because:

max_tokens is declared in this case, so the ceiling is real rather than guessed
the threshold sits orders of magnitude away from typical, so a 3.87x error does not change the answer

A $50 per-call ceiling is not troubled by an estimator that is 4x off on a call that costs $0.02. The failure mode above only appears when the threshold is close enough to typical spend that a 4x error crosses it.

Caveats, before someone else supplies them

Nine comparable calls is a small sample. Two unbounded calls is a tiny one. It was all one model on one provider.

I would take this as directional rather than conclusive, and I would be genuinely interested if anyone has measured it at larger scale — I could not find anyone who had, which is partly why I am writing it down. The failure was consistent and bidirectional across every call I looked at, and the mechanism (output length being unknowable in advance) is not specific to a model or a provider.

The thing I built

focxle is what came out of it: a Python package that tells you what each of your AI agents cost across every vendor it touched, model calls and HTTP APIs together.

pip install focxle

import focxle
focxle.init()

It prints per agent, per vendor, when the process exits. It also shows what a cap would have stopped, using the reconciled approach above, so you can see whether you want one before turning one on.

Zero dependencies, no account, no proxy in front of your traffic, and no network calls at all on the free path. MIT, source at github.com/focxle/focxle-python.

Mostly though I wanted the negative result written down somewhere. The estimate-then-refuse design is the obvious one, it is what I would have built again, and it fails in a way you only see if you go and measure it.

DEV Community: focxle