Last week I argued you should scale your LLM routing on task success rate, not request volume. A reader, Alex Shev, asked the question that breaks that rule — politely, and correctly: what counts as "success" when the truth arrives late?
Plenty of tasks only reveal their real outcome days after the model answers. A human downstream corrects the extracted fields. A customer reopens the ticket. A refund lands. An invoice the model "completed" gets rejected by the finance system on Thursday. If your dashboard scores the task the moment the model answers, it is grading a provisional outcome and calling it final. Until the attribution window closes, every number on that dashboard is a draft.
The outcome is not the answer
A routing dashboard usually records one of two things and calls it success: the model returned something, or a cheap check passed. Both happen at answer time. Neither is the outcome.
Real outcomes arrive through channels your dashboard may never see:
- Downstream human correction. An operator fixes the fields the model extracted. The task looked complete at 10:04. It was actually wrong, and the correction is logged in a different system, or not logged at all.
- Customer action. The user reopens the ticket, rephrases the question, or churns. Silence after an answer is not confirmation — it is the absence of a signal, and it gets scored as success by default.
- Late system rejection. The draft summary passes your format check, then fails the compliance review two days later. The failure belongs to the task, but your dashboard already closed the book on it.
None of this is exotic. It is the normal shape of real work. And it means the success rate you alert on is systematically optimistic — not because anyone is cooking the numbers, but because the failures haven't arrived yet.
The math: what a 4-point lie costs
Take the same pipeline from my last post, so the numbers stay comparable. A day's work needs 10 GPU-hours: 8 hours of chores on an RTX 4090 at $0.89 per GPU-hour ($7.12) and 2 hours of final answers on an H100 at $2.69 ($5.38). Total: $12.50 per day.
Say 1,000 tasks run through it in a day. Scored at completion time, 970 succeed:
- Provisional cost per completed task: $12.50 / 970 = $0.0129.
Now let the 7-day attribution window close. Forty of those 970 "successes" come back — a corrected field here, a reopened ticket there. Final completions: 930.
- Final cost per completed task, before redos: $12.50 / 930 = $0.0134.
And the 40 late failures each need a large-model redo. Assume a redo takes about 2 minutes of H100 time — your number will differ, measure yours: 40 × 2/60 hours × $2.69 = $3.59 of redo cost. True daily cost: $16.09.
- True cost per completed task: $16.09 / 930 = $0.0173 — 34% above the number the dashboard showed on day one.
Four points of success rate, hidden by timing, turned into a third more cost per completed task. And the damage compounds: if you promoted a task type to a cheaper tier — or demoted one to a bigger model — on the provisional 97%, you made a routing decision on a number that was never true. The dashboard didn't lie on purpose. It just reported before the truth arrived.
The gap is itself a signal
Run two numbers in parallel and watch the gap between them:
- Provisional success rate — scored at answer time. Fast, useful for alerting, never a basis for a routing change.
- Final success rate — scored after the attribution window closes. Slow, and the only number a routing decision should rest on.
A narrow gap means your fast checks are honest. A wide gap means your routing is fast but fragile: it looks good on the day and quietly bleeds redos all week. That gap is the most useful number on the dashboard, and most dashboards don't show it at all.
Two guardrails from the discussion under my last post, both earned the hard way:
- A window that hasn't closed is not a window. The two-window rule — require the success rate to stay below the bar for two consecutive windows before moving a task type up a tier — only works if both windows are closed. A window still inside its attribution period is a partial count. Count it and you are back to deciding on drafts.
- Don't react to single flips. When a task type has a near-zero failure baseline, one late failure doubles the failure rate. That is noise wearing a costume. We run a 30-closed-findings floor before trusting a routing change, and we ask how many flips — not just what percentage — moved before acting. The percentage is the headline; the flip count is the evidence.
How long is the window?
There is no universal answer, and anyone selling you one is guessing. Ours is 7 days per task type — and I will be explicit that this is a starting guess, not a measured constant. The way to calibrate it is a replay: re-score the same tasks at 3, 7, and 14 days and see where the numbers stop moving. If the 7-day and 14-day scores agree, 7 days is long enough for that task type. If they don't, lengthen it. The window is a property of the task type — an invoice extraction that gets corrected same-day needs a shorter window than a customer ticket that reopens a week later. Fix it per task type, write it down, and don't let the dashboard quietly use "until now" as the window.
And timestamp every success event. "Succeeded" without a time is unscoreable: you cannot tell whether it succeeded before or after the window closed, and you cannot replay it. A success event with no timestamp is a rumor.
We measure our own funnel the same way
This is not theory we exempt ourselves from. Our own numbers have the same delayed-truth shape:
- Google Search Console reports with a lag of a couple of days, so the dashboard we read this morning is already a draft. In the latest 28-day window (October 13 to November 9, 2026), our whole site recorded 6 clicks and 248 impressions — and our GPU cost calculator page recorded zero impressions, meaning it still isn't ranking at all. That zero is a fact about the window, not a verdict on the page; the window hasn't finished telling us yet.
- The last post itself kept changing after it was "done." Its comment count went from 27 on the morning of November 9 to 42 by the afternoon of November 10 — a 55% increase in engagement, days after publishing, driven by readers arriving late with real questions. The provisional engagement number understated the final one by more than half. If we had judged the post on day one, we would have judged it wrong.
Measure first, spend second — and measure again after the window closes.
The bottom line
Do not ask "what is our success rate?" Ask "what is our success rate after the attribution window closes, and how far did it move from the provisional number?" Volume tells you when to buy more of what works. Success rate tells you when what you have stopped working. The attribution window tells you whether either number was real yet.
If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: https://qisuanai.com/gpu-cost-calculator/
Prices: RunPod Community Cloud list — RTX 4090 $0.34, H100 SXM $2.69 per GPU-hour; RTX 4090 Secure Cloud $0.89 (the rate used in the worked example, carried over from the previous post for comparability). Lambda H100 $3.99 now, $4.59 from November 16, 2026. All re-checked November 10, 2026; community-cloud availability varies by the hour. The worked example is arithmetic on GPU-hours, not a production benchmark — your redo times and correction rates will differ, and they are the numbers that decide your actual bill. The 7-day window is our starting guess, stated as such; calibrate yours by replay. Thread data: discussion under "Don't Scale Your LLM Routing on Request Volume" (November 6, 2026), including Alex Shev's delayed-outcome question and the 30-findings floor, two-window rule, and flip-count discussion with Agentic Affairs.
Top comments (0)