The coding model I want in production is the one that gets an accepted patch through validation at the lowest total cost. That includes failed attempts, escalation, tool execution, and reviewer time. A cheap response that creates another debugging session is not a cheap result.
GPT-5.6 and Claude both offer enough tiers to make routing worth evaluating. Luna and Haiku cover lightweight work; Terra and Sonnet are general coding candidates; Sol, Opus, and Fable belong in harder-task evaluations. Those are starting points, not equivalent capability classes. The specific model names, benchmark results, and prices below are the source article’s reported figures; verify current provider documentation before using them in a budget or deployment.
Start With an Accepted Task
I would track this metric before optimizing token spend:
Cost per successful task =
(primary model cost + retry cost + fallback cost
+ tool cost + human review cost) / successful tasks
The denominator matters. Expected API spend per submitted task is not automatically cost per successful task: some tasks can still fail after escalation. Record first-pass success and final success separately, then account for everything spent on unsuccessful attempts too.
For each task, I would log model and effort level, input and output usage, cache reads and writes, tool calls, retries, escalation, end-to-end latency, and human review minutes. Model-and-effort combinations are the unit of comparison. Higher reasoning effort only earns its additional spend when it improves acceptance or reduces downstream correction.
Build a Shortlist, Not a Benchmark Leaderboard
The comparison attributed to OpenAI’s GPT-5.6 evaluation reports these results:
| Model | Artificial Analysis Coding Agent Index v1.1 | SWE-Bench Pro |
|---|---|---|
| GPT-5.6 Sol | 80 | 64.60% |
| GPT-5.6 Terra | 77.4 | 63.40% |
| GPT-5.6 Luna | 74.6 | 62.70% |
| Claude Fable 5 | 77.2 | 80.00% |
| Claude Opus 4.8 | 72.5 | 69.20% |
Sol leads the Coding Agent Index in this table; Fable leads SWE-Bench Pro. The source also reports different results across DeepSWE and Terminal-Bench 2.1. I would use that disagreement to broaden an eval, not declare an overall winner. Harnesses, available tools, reasoning settings, and execution environments all affect coding-agent performance.
My initial lightweight candidates would be Luna and Haiku 4.5 for classification, routing, and simple explanations. For repository Q&A, tests, reviews, and scoped fixes, I would compare Terra and Sonnet 5, escalating to Sol or Opus 4.8. Multi-file refactors justify testing Sol or higher-effort Sonnet first, with Opus or Fable as additional candidates. For architecture migrations and security-sensitive changes, stronger models do not remove the need for human review.
Put the Rates in One Place
These are the source’s reported prices in dollars per million tokens. GPT-5.6 figures are for Standard short-context requests; long-context, Batch, Flex, and Priority processing have separate rates. Check the applicable service tier in the OpenAI pricing documentation.
| Model / period | Input | Cached read | Cache write | Output |
|---|---|---|---|---|
| GPT-5.6 Luna | $1.00 | $0.10 | $1.25 | $6.00 |
| GPT-5.6 Terra | $2.50 | $0.25 | $3.13 | $15.00 |
| GPT-5.6 Sol | $5.00 | $0.50 | $6.25 | $30.00 |
| Haiku 4.5 | $1.00 | $0.10 | $1.25 / $2.00 | $5.00 |
| Sonnet 5, through Aug. 31, 2026 | $2.00 | $0.20 | $2.50 / $4.00 | $10.00 |
| Sonnet 5, from Sept. 1, 2026 | $3.00 | $0.30 | $3.75 / $6.00 | $15.00 |
| Opus 4.8 | $5.00 | $0.50 | $6.25 / $10.00 | $25.00 |
| Fable 5 | $10.00 | $1.00 | $12.50 / $20.00 | $50.00 |
Claude’s two write prices correspond to five-minute and one-hour caches. Sonnet’s introductory $2 input / $10 output pricing runs through August 31, 2026; the reported standard $3 / $15 rates begin September 1.
On headline rates, introductory Sonnet is cheaper than Terra. Haiku matches Luna’s input price and charges $5 rather than $6 for output. Opus matches Sol’s $5 input price and charges $25 rather than $30 for output. After Sonnet’s pricing transition, Terra has cheaper input at $2.50 versus $3, with both charging $15 for output. None of those comparisons includes reliability or actual token consumption.
Token Counts Are Part of the Price
I would not apply one provider’s token estimate to another provider’s bill. The source attributes to Anthropic a newer tokenizer in Sonnet 5, Fable 5, and newer Opus models that can produce approximately 30% more tokens for the same text, depending on workload. Log returned usage rather than assuming identical token counts.
There is also a model-selection trap: the source states that the generic gpt-5.6 alias maps to Sol. Where Terra or Luna passes the eval, select that tier explicitly instead of allowing an alias to determine flagship usage.
Work Through the Fallback Arithmetic
Assume each attempt consumes 80,000 input tokens and 10,000 output tokens, without caching. There is one primary attempt and a stronger fallback when it fails. This is a pricing illustration, not measured model performance; tokenization, tool use, effort, and actual success rates can change the result.
| Route | Primary attempt | Fallback attempt | Expected API spend with 25% fallback |
|---|---|---|---|
| Terra → Sol | 0.08 × $2.50 + 0.01 × $15 = $0.35 |
0.08 × $5 + 0.01 × $30 = $0.70 |
$0.35 + 0.25 × $0.70 = $0.525, about $0.53 |
| Introductory Sonnet → Opus | 0.08 × $2 + 0.01 × $10 = $0.26 |
0.08 × $5 + 0.01 × $25 = $0.65 |
$0.26 + 0.25 × $0.65 = $0.4225, about $0.42 |
| Standard Sonnet → Opus | 0.08 × $3 + 0.01 × $15 = $0.39 |
$0.65 | $0.39 + 0.25 × $0.65 = $0.5525 |
At equal fallback rates, introductory Sonnet wins this example; Terra becomes slightly cheaper after the reported September 1 change. But reduce Terra’s fallback rate to 10% and its expected spend becomes $0.35 + 0.10 × $0.70 = $0.42. That is slightly below introductory Sonnet’s unrounded $0.4225 and below standard Sonnet’s $0.5525. Rounding both introductory results to cents hides the first difference.
I would not call any row a production winner yet. These figures exclude tools and review, assume identical usage on primary and fallback attempts, and do not establish whether the fallback succeeds.
Measure Cache Reuse Alongside Escalation
The source describes OpenAI matching prompt prefixes through implicit caching, with GPT-5.6 additionally supporting explicit breakpoints and prompt_cache_key. It gives cache writes a 1.25× normal-input rate and reads the discounted cached-input rate.
Claude caching is enabled through cache_control, using a request-level automatic breakpoint or explicit content-block breakpoints. Its default lifetime is five minutes, with an optional, more expensive one-hour write. Reads cost 0.1× the base input rate. These implementation differences matter when repository instructions, tool definitions, coding standards, or project context recur across calls.
For the Terra example, serving 40,000 of the 80,000 input tokens from cache changes a subsequent request to $0.10 regular input + $0.01 cached input + $0.15 output = $0.26, down from $0.35. Writing that 40,000-token prefix instead gives $0.375 for the request using the stated 1.25× multiplier. The displayed $3.13/MTok write price is rounded; the multiplier corresponds to $3.125/MTok.
Caching pays through reuse, not necessarily on the first request. I would measure writes, reads, retries, and fallback together. A lower input bill cannot rescue a route that repeatedly produces unusable patches, and context-window capacity alone says little about effective task cost.
Run a Small Repository Eval Before Building a Clever Router
Around 30 representative tasks is a practical first pass: 10 bug fixes, 10 implementation or test-generation tasks, 5 refactors, and 5 reviews. Compare Terra, Sol, Sonnet 5, and Opus 4.8 where they fit the workload. Add Luna and Haiku for lightweight subtasks, and Fable as a higher-capability reference for difficult work.
Keep acceptance criteria identical: passing tests, successful builds, lint and type checks, resolution of the requested issue, and the amount of human correction required. Parser or AST checks, pytest, npm test, and isolated patch execution provide useful automatic validation. They make cheaper-first routing testable; they do not prove every change correct.
Segment results by task class. A model that is economical for reviews may be a poor default for bug fixes. Track first-pass and final success, total API spend, retries, fallback frequency, cache-hit rate, latency, and review time. Otherwise, an aggregate average can conceal exactly the workload distinction the router needs.
Keep the First Router Rule-Based
My starting policy would be: classify the task, choose the lowest-cost route that passes the relevant eval, validate, then escalate on failure. A candidate sequence is Luna or Haiku → Terra or Sonnet → Sol or Opus → Fable or human review. That is an evaluation hypothesis, not a requirement to traverse every tier.
High fallback rates suggest strengthening the initial route. Premium calls that rarely improve acceptance suggest reducing escalation. Extra effort without better outcomes suggests lowering effort. Repeated context dominating spend suggests improving cache reuse. For security-sensitive or architectural work, I would require human review rather than treating automatic checks as sufficient.
A unified interface can reduce integration work when comparing providers: CometAPI exposes supported models through an OpenAI-compatible Chat Completions interface, allowing model selection through the model parameter. That simplifies switching, but it does not replace provider-specific usage accounting or quality evaluation.
The deployment decision I care about is specific: which route delivers accepted changes for this task class, at an acceptable latency, with the lowest combined API, execution, and review cost? Public benchmarks identify candidates. Production telemetry decides which ones stay.
Originally published at cometapi.com
Top comments (0)