Your request volume doubled this month. So you doubled your GPU spend. That is the most common scaling mistake in LLM inference, and it is also the most expensive one — because request volume tells you almost nothing about whether your routing is working.
This week the same question kept surfacing on X in different words: token prices have collapsed, so why are AI bills still painful? One widely shared thread argued the metric that matters is cost per successful outcome, not cost per token. Another pointed out the same model weights can cost 9x more at one provider than another. The underlying question in all of them is the same: when is it actually time to send more work to the big model — and when is that just burning money?
The answer is not "when requests go up." It is "when the task success rate says so."
Request volume is the wrong trigger
Request volume measures demand, not quality. It goes up for reasons that have nothing to do with your routing: a marketing push, a new customer, a retry storm from a flaky client, a prompt change that made outputs longer. None of those mean the small model is failing more often.
Scaling on volume has a second cost that is easy to miss: it buys capacity before you know what the capacity is for. If your small model is already completing 96% of chores correctly, doubling your large-model hours buys you almost nothing — you are paying frontier prices to re-do work that was already fine. And the price of that mistake is going up, not down: Lambda's published instance rates rise on November 16, 2026, with the H100 moving from $3.99 to $4.59 per GPU-hour and the A100 (80 GB) from $2.79 to $3.19. Every GPU-hour you buy on the wrong trigger costs more than it did last week.
Volume is a capacity signal. It answers "do I need more total compute?" It does not answer "does this task type need a bigger model?" Those are different questions, and answering the second one with the first one's data is how teams end up with a large-model bill and a small-model quality problem.
The right trigger: task success rate and cost per completed task
Track two numbers per task type — not per request, and not per token:
- Task success rate: the share of tasks that finish correctly without a redo. A task is a unit of real work — an extracted invoice, a classified ticket, a drafted summary — not one API call.
- Cost per completed task: total GPU cost divided by tasks that finished successfully, redos included.
Cost per request hides redos. Cost per completed task exposes them. The formula is simple:
effective cost = small-model cost + (redo rate × large-model cost) + final-answer cost
If a third of your extracted fields come back wrong and need a large-model redo, your "cheap" chores are no longer cheap — and no amount of request-volume data will tell you that. Only the success rate will.
The decision rule follows from the numbers, not from the traffic graph: scale a task type up to the large model when its success rate falls below the bar you set, and stays there across two measurement windows — not when request count crosses a round number. Conversely, if success rate is stable and cost per completed task is flat, a volume spike means you need more of the same capacity, not a bigger model.
The math, with prices checked today
These are public GPU listing prices I re-checked on November 6, 2026 — not a production benchmark, and not a promise about your throughput:
- RunPod Community Cloud: H100 SXM from $2.69 per GPU-hour, A100 SXM from $1.39; RTX 4090 listed at $0.89.
- Lambda (rates effective November 16, 2026): H100 at $4.59 per GPU-hour, A100 (80 GB) at $3.19.
Now the arithmetic on GPU-hours, because that is the unit you actually pay for when you self-host. Suppose a day's pipeline needs 10 GPU-hours: 8 hours of chores on a 4090, 2 hours of final answers on an H100:
- Chores: 8 × $0.89 = $7.12
- Final answers: 2 × $2.69 = $5.38
- Total: $12.50 per day for the work attempted.
Now apply the success rate. If 1,000 tasks are attempted and 970 complete successfully, cost per completed task is $12.50 / 970 = $0.0129. If the small model's success rate slips to 85% — 850 complete, 150 need a redo — the same $12.50 buys fewer completions ($0.0147 each), and the redos add large-model hours on top. A 12-point drop in success rate costs you more than a 12% rise in request volume ever would, and it is invisible in every dashboard that only counts requests.
That is the number to alert on. Set the bar per task type: a classification chore might need 98%, a first-pass summary might be fine at 90%. The bar is a product decision, not a universal constant. What matters is that it is written down before the volume spike arrives — because in the middle of a spike, every team "discovers" that the answer is a bigger model.
How to measure it without drowning in instrumentation
You do not need a full observability platform to start. You need three logs per task type:
- Outcome: completed, redone, or escalated — one row per task, not per request.
- Redo reason: wrong fields, missing citation, format failure, low confidence. The reason tells you whether the fix is a better prompt, a narrower task definition, or a bigger model. Most redo reasons are not "model too small."
- Cost: GPU-hours per task type, so cost per completed task is a division, not a guess.
Audit a random sample every week — this is the part readers of my last post kept asking about, and they were right to. One reader asked how big the sample needs to be before an audit hit rate means anything; the honest answer is that it depends on the category. Sample per task type, not across the whole pipeline, and audit 100% of the task types where an error is expensive. A 2% error rate on invoice extraction is a different business from a 2% error rate on first-pass summaries, and one blended number hides that.
Two practical guardrails:
- Two windows, not one. A single bad hour is noise — a deploy, a bad prompt, a weird input batch. Require the success rate to stay below the bar for two consecutive measurement windows (a day, a week, whatever fits your volume) before you move a task type up a tier.
- Fix the chore before you upgrade the model. If redos cluster on one input shape — long documents, a missing field, an ambiguous category — narrow the task or fix the prompt first. Upgrading the model to paper over a broken task definition is the most expensive way to learn the definition was broken.
We measure our own funnel the same way
This is not theory we exempt ourselves from. Our own GPU cost calculator page recorded 0 clicks and 0 impressions in Google Search Console over the last 28 days (October 9 to November 5, 2026) — it is not ranking yet, and we know that because we measured instead of assuming. We did not respond by buying promotion for it; we responded by checking what is actually being found (our whole site: 9 clicks, 350 impressions in the same window) and fixing the work in front of us first. The same discipline applies to routing: measure first, spend second.
The bottom line
Do not ask "how many requests did we get?" Ask "what share of tasks completed successfully, and what did each completed task cost?" Volume tells you when to buy more of what already works. Success rate tells you when what you have stopped working. Scale on the second number.
If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: https://qisuanai.com/gpu-cost-calculator/
Prices: RunPod and Lambda public pricing pages, checked November 6, 2026. X discussion referenced: cost-per-outcome framing in AI billing threads, week of November 3–6, 2026.
Top comments (9)
The two-measurement-windows rule is what keeps this from becoming another dashboard that panics on a retry storm — a volume spike from a marketing push says nothing about whether the small model is failing. The redo-rate math is the part teams feel in the bill before they see it in the metrics. How do you set the success bar per task type in practice — is 98% for invoice extraction a negotiated number with the business, or a starting guess you tune over time?
Good questions — the 98% needs a window and a task type to mean anything, so here is how I would pin it down.
Window: rolling 7 days per task type, not a single day. A daily window is too noisy — one bad batch moves the number by points — while a weekly window is long enough to see drift and short enough to act on. Keep the daily figure as an early warning, but gate decisions on the 7-day roll.
Success line per task type, yes. Extraction on structured fields (dates, amounts, ids) and on open-ended summaries carry different error costs: a wrong date silently misroutes a task, a loose summary just reads worse. So the line is set from the cost of being wrong, not from a round number.
And you are right that 98% is a starting guess for negotiation, not a law. The honest way to set it: take the measured cost of a wrong extraction — rework, a misrouted task, a retry — and pick the threshold where tightening the check costs more than the errors it prevents. If that lands at 96% for one task type and 99.5% for another, so be it.
On the canary: 3 days at 10% is the entry gate, not the whole test. The drift risk you flag is real — slow drift does not show in 3 days. So hold at 10% for a full week, and compare the canary cohort's extraction accuracy against the control cohort over the same window before scaling. If the two lines diverge, the canary did its job; if they track together for a week, scale with confidence.
The cost-of-being-wrong framing is the right anchor — a threshold set from the error's price beats a round number every time. The canary-versus-control comparison over the same week is the part most teams skip; they scale on the canary looking fine in isolation. How do you measure the cost of a wrong extraction in practice — rework hours, or do you price the misroutes directly?
Both, but priced directly — rework hours are the input, not the answer. For each wrong extraction we log three numbers on the same finding id: (1) re-run cost — tokens and dollars to re-extract, usually small, $0.02–$0.05 per finding in our setup; (2) rework — minutes a human spends fixing it, times a loaded hourly rate (we use $60/hr, so a 12-minute fix is $12); (3) downstream damage — what the wrong value actually did before it was caught, e.g. a misrouted task that ran on the wrong model and had to be re-run end to end, or a bad field that reached a customer-facing summary. In practice (2) dominates: the re-run is cents, the fix is dollars, and a single uncaught downstream error can be 10–50x the rework cost. So the threshold math is simple: a routing change is worth it when the expected wrong-extraction cost per 1,000 findings — misroute rate × (rework + downstream) — exceeds the cost of the stricter check. That is why we price per finding id rather than per request: two extractions can share one request and have wildly different blast radius.
Pricing per finding id rather than per request is the move that makes the whole threshold honest — two extractions sharing a request can have totally different blast radii, and a per-request number hides that. The 10–50x downstream multiplier is the part I'd want on a dashboard next to the success rate itself. Do you log the downstream damage as an estimate at detection time, or does it get revised once the full cost is known?
Both, in two steps. At detection we book an estimate — 25x, the midpoint of the 10–50x range, times that finding's rework cost — so the dashboard has a number. Once the downstream cost settles, we replace it with the actual and keep both columns, estimate vs. actual. The gap is the calibration signal: shrinking means the multiplier is getting truer; drifting means re-estimate it.
Per finding id, each finding carries its blast-radius tag — downstream tasks and customers affected — so two findings in one request are priced separately and the threshold is computed per finding, not per request. On the dashboard, success rate sits next to this month's estimated downstream loss and the actual; convergence means the estimate is trustworthy.
Keeping both columns is the part that makes this honest — the estimate-vs-actual gap as a calibration signal turns the multiplier from a guess into something that earns trust over time. Do you re-estimate the multiplier per finding type, or does one global number get tuned as the gap data comes in?
Per finding type — a single global multiplier hides exactly the skew you're trying to catch. I'd run one multiplier per finding type (auth bypass, data leak, prompt injection, and so on), each starting from the same prior — say 25x — and re-estimated monthly from that type's own estimate-vs-actual gap: new multiplier = median(actual / estimate) over the last 20 closed findings of that type, floored at 5x and capped at 100x so one outlier can't swing it.
The global number still has a job: it's the dashboard-level sanity check. If the weighted average of the per-type multipliers drifts more than about 2x from the global in a quarter, that's a signal the taxonomy itself is wrong — a finding type is too broad and is mixing cheap and expensive cases. The rule I'd keep: per-type numbers for pricing decisions, the global only for spotting taxonomy drift — never let the global override a per-type number on a live routing call.
The median over the last 20 with the 5x–100x floor and cap is the right guard — one outlier shouldn't reprice a whole finding type. The sharpest part is using the global drift as a taxonomy test rather than a pricing input: if the weighted average drifts 2x, the categories themselves are wrong, not the numbers. Do you have enough closed findings per type yet for the median to mean something, or is 20 a target only the common types will reach for a while?