Stop Picking a Model. Start Routing Workloads.
Procurement wants one model. Your architecture wants two.
That tension is where most enterprise AI programs quietly lose money. Someone runs a bake-off, picks a winner on aggregate benchmark scores, signs an enterprise agreement, and then routes every workload — spreadsheet reconciliation, CI/CD debugging, contract analysis, browser automation — through the same endpoint. The bill arrives in Q3 and nobody can explain it.
Here's the thing the benchmark leaderboard can't tell you: "best model" is not a property of a model. It's a property of a model plus a workload plus a tool surface plus an acceptance criterion.
Consider two requests hitting the same architect in the same week. Finance wants an agent that opens a sales spreadsheet, reconciles the numbers, builds a chart, updates a deck, and saves the document. Engineering wants an agent that investigates a recurring inventory-service bug, traces dependencies, proposes a fix, and verifies that the fix doesn't break a neighboring service.
Those look similar on a roadmap. They fail in completely different ways. The spreadsheet workflow fails when the agent writes to the wrong file. The debugging workflow fails when a plausible-looking patch hides the actual defect. In neither case is "produced a fast answer" the definition of done.
You're not hiring the smartest colleague. You're assigning responsibilities, tools, and acceptance criteria to two different jobs.
The reflex vs. the fix
Most teams run this loop:
REFLEX: pick a model → give it every task → optimize the bill later
FIX: define the job → test the model + tools → verify the outcome
The reflex isn't stupid. It's operationally cheap — one vendor, one key, one SDK, one set of rate limits. It only becomes expensive when the failure modes diverge, which is exactly what happens the moment your agents start acting instead of answering.
The fix requires you to score each workload on four decisions before any model gets assigned.
The four-decision routing test
1. Execution — can it complete the required actions across your tools?
Not "can it write code that would work." Can it drive your actual tool surface: your CRM API, your shell, your browser, your filesystem? This is an agentic capability question, and it's mostly orthogonal to reasoning quality.
2. Building — can it understand the problem and produce a correct, maintainable artifact?
This is the analytical axis. Does the output survive code review? Does the contract comparison hold up when a lawyer reads it? An artifact that's wrong in a subtle way is worse than no artifact, because it consumes review cycles and then fails in production anyway.
3. Time — does it meet the deadline for an accepted result?
Note the word accepted. Time-to-first-token is a demo metric. Time-to-accepted-outcome includes retries, tool errors, and the human who has to look at it. A model that answers in four seconds and gets rejected twice is slower than one that takes forty seconds and lands.
4. Cost — what does that accepted result cost, fully loaded?
cost_per_accepted_outcome =
(input_tokens_uncached × rate_in)
+ (input_tokens_cached × rate_cached)
+ (cache_write_tokens × rate_cache_write)
+ (output_tokens × rate_out)
+ (retry_count × full_attempt_cost)
+ (human_review_minutes × loaded_hourly_rate / 60)
÷ acceptance_rate
That last division is the one everyone forgets. At a 70% acceptance rate, your real cost per outcome is 1.43× your per-attempt cost. Token price is maybe a third of the equation.
Reading benchmarks as capability signals, not rankings
The useful move isn't "which model is #1." It's "what does this specific test actually measure, and which of my workloads does that map to?"
Working from the Artificial Analysis September 15 snapshot at maximum reasoning effort (with Fable's default fallback enabled — meaning another model may service some requests, so treat these as directional, not matched):
Astra reads as an operator. It leads on AutomationBench-AA, which measures completing multi-step business workflows across SaaS apps through APIs while respecting business rules. It also posts stronger Terminal-Bench 4.0 results — command-line engineering and ops work. OpenAI reports it ahead of GPT-5.6 Sol and Claude Opus 5 on OSWorld 2.0's offline partial-credit comparison, though that published setup doesn't establish a matched head-to-head against Fable 5.1.
Map that to: CRM record updates, support escalation processing, employee onboarding flows, expense-vs-budget checks, failed build triage, dependency resolution, CI/CD pipeline debugging, supplier invoice retrieval, portal form filling, frontend QA.
Fable reads as an analytical builder. It leads on SciCode — scientific programming specifically, not software development broadly — and on AA-LCR v1.1, long-document reasoning.
Map that to: engineering simulations, numerical work, research-oriented code where domain reasoning and code generation have to interlock, supplier contract comparison, RFP analysis, financial report review, conflicting-policy detection, regulatory requirement mapping.
Then the economics split in an interesting way: Astra shows lower cost per task in the cited evaluation, while Fable is cheaper on cached input. Those favor different shapes of work.
Cached-input pricing is a workload-shape question, not a discount. If the same large context — a policy corpus, a codebase, a contract set, a documentation tree — gets reused across many calls, cache-friendly pricing dominates the bill. If every call carries a fresh context, it's irrelevant. Check your reuse ratio before you let a pricing page pick your model.
The pattern that beats picking one: plan with the builder, execute with the operator
Once you stop treating model choice as a procurement decision, an obvious architecture falls out. Split the workflow at the plan/act boundary.
┌──────────────────────────────┐
task + context → │ PLANNER (analytical model) │
│ • decompose the goal │
│ • read the long context │
│ • emit typed action plan │
└───────────────┬──────────────┘
│ structured plan
│ (JSON, validated)
┌───────────────▼──────────────┐
│ VALIDATOR (deterministic) │
│ • schema check │
│ • policy / scope check │
│ • blast-radius estimate │
└───────────────┬──────────────┘
reject ◄──┤──► approve
│
┌───────────────▼──────────────┐
│ EXECUTOR (operator model) │
│ • drive tools step by step │
│ • no re-planning authority │
│ • emit per-step receipts │
└───────────────┬──────────────┘
│
┌───────────────▼──────────────┐
│ VERIFIER │
│ • did the artifact change? │
│ • does it pass acceptance? │
└──────────────────────────────┘
pass → done | fail → replan
The critical design constraint: the executor does not get replanning authority. If it hits an unexpected state, it fails the step and returns control to the planner. That one rule is what makes the split worth the added latency — it keeps an operator-tuned model from improvising its way into your production filesystem.
Here's the handoff contract, which matters more than either model choice:
{
"plan_id": "recon-2026-09-19-001",
"goal": "Reconcile Q3 regional sales against the GL extract",
"acceptance": {
"type": "assertion",
"check": "variance_report.total_unexplained < 0.005 * gl_total"
},
"steps": [
{
"id": 1,
"tool": "sheets.read",
"args": { "file_id": "1AbC...", "range": "Q3!A1:M400" },
"writes": false
},
{
"id": 2,
"tool": "sheets.write",
"args": { "file_id": "1AbC...", "range": "Variance!A1" },
"writes": true,
"blast_radius": "single_sheet",
"requires_approval": true
}
],
"on_unexpected_state": "halt_and_return"
}
Every step declares whether it writes. Every write declares a blast radius. The validator can now enforce policy before a token of execution is spent — and your autonomy limits become config, not vibes.
Autonomy is a dial, not a switch
The four decisions give you capability. This gives you containment:
| Tier | Model class | Write access | Human gate |
|---|---|---|---|
| T0 | Small / cheap | None (read-only) | None |
| T1 | Operator | Scratch + sandbox | Post-hoc audit |
| T2 | Operator | Scoped production writes | Pre-approve plan |
| T3 | Builder + Operator | Broad production writes | Approve plan + verify artifact |
Most workloads that get routed to a frontier model belong at T0 or T1. Classification, extraction, routing, summarization of a single document, formatting — these don't need frontier reasoning, and paying for it is the quiet majority of overspend in most agent platforms I've seen described.
You do not need a frontier model at every step. You need the right amount of intelligence at each step.
What to actually do on Monday
- List your top five agentic workloads. For each, write the acceptance criterion as an assertion, not a sentence.
- Score each on the four decisions. Execution and Building especially — most teams have never separated them.
- Compute cost per accepted outcome, including retries and review minutes. Divide by acceptance rate.
- Find the workloads with high context reuse. Those are your cache-pricing candidates.
- Pick one workflow and split it at the plan/act boundary with a typed handoff contract. Measure acceptance rate before and after.
Key takeaways
- "Best model" isn't a model property. It's model × workload × tool surface × acceptance criterion. Leaderboards can't encode the last three.
- Score four decisions, not one score: Execution (can it drive your tools?), Building (is the artifact correct and maintainable?), Time (to accepted result, not first token), Cost (per accepted outcome, fully loaded).
- Benchmarks are capability signals. Read what a test measures — SaaS workflow completion, terminal ops, scientific programming, long-document reasoning — and map it to a workload, not a ranking.
- Cached-input pricing only matters if your context is reused. Measure your reuse ratio before letting a pricing page make an architecture decision.
- Split plan from execute, and deny the executor replanning authority. A typed, write-annotated handoff contract turns autonomy limits into enforceable config.
- Most steps don't need a frontier model. Tier your autonomy and your model class together. ## Your turn
Here's what I keep going back and forth on: the planner/executor split adds a network hop, a serialization boundary, and a second vendor relationship. It only pays off if the acceptance rate improves enough to cover that.
So — has anyone actually measured it? If you've run a single-model agent and a plan/execute split on the same workload, what happened to your cost per accepted outcome? And if you standardized on one model anyway, what made that the right call?
I'd rather see one real number than ten more benchmark charts.
Top comments (1)
The cost-per-accepted-outcome formula with acceptance_rate in the denominator is the part that changes how you think about model choice entirely. At 70% acceptance, you're paying 1.43× per attempt before you factor in retry costs and human review minutes. The teams spending the most per outcome are often using a fast, cheap model with a 50% acceptance rate rather than a slower model with 90%.
The executor-has-no-replanning-authority rule is the one I'd highlight most. That single constraint is what keeps an operator-class model from improvising its way through unexpected state into something irreversible. If it can't figure out the next step, it fails and returns control rather than getting creative. That's the right failure mode.
The typed handoff contract with writes: true/false and blast_radius on every step is also the right pattern — it makes your autonomy limits enforceable at validation time rather than discoverable in production. "Requires approval before execution" becomes a config field, not a manual check.
To your question: yes, the split only pays off if acceptance rate improvement covers the extra hop. The right measurement is to run both modes on the same workload for a fixed number of attempts and compare accepted outcomes / total cost, not accuracy scores.