There is no benchmark result that settles this choice for every coding agent.
The published Terminal-Bench 2.1 numbers are close: Claude Opus 5 scores 89% at Max effort, GPT-5.6 Sol scores 88.8% in the reported single-agent run and 91.9% in Ultra, and Gemini 3.7 Flash scores 85.8%. Those results are useful for narrowing the field, but they are not a clean ranking. The reasoning effort, harness configuration, and Ultra's multi-agent setup differ.
For a fixed request containing 20,000 input tokens and 2,000 output tokens, the listed prices are USD 0.018 for Gemini 3.7 Flash, USD 0.120 for Claude Opus 5, and USD 0.128 for GPT-5.6 Sol at its short-context rate.
My production metric would be cost per accepted patch, measured against the same repository tasks, tools, tests, retry policy, and sandbox.
The Short Version
| Model | Published coding result | Listed price | Model ID | Main integration routes |
|---|---|---|---|---|
| Claude Opus 5 | Terminal-Bench 2.1: 89% at Max effort | $4/M input, $20/M output; $0.120 for the fixed scenario | claude-opus-5 |
Chat Completions, Anthropic-compatible Messages |
| GPT-5.6 Sol | Terminal-Bench 2.1: 88.8%; 91.9% Ultra | $4/M input and $24/M output up to 272K input tokens; $8/M input and $36/M output above that; $0.128 for the fixed scenario | gpt-5.6-sol |
Chat Completions, OpenAI Responses |
| Gemini 3.7 Flash | Terminal-Bench 2.1: 85.8% | $0.60/M input, $3/M output; $0.018 for the fixed scenario | gemini-3.7-flash |
Chat Completions, Gemini-native generation |
As of August 24, 2026, these are the listed rates. Prices and supported routes can change, so I would recheck the individual model pages and text API documentation before deployment.
The fixed-request calculation follows the provider pricing guide:
20,000 input tokens + 2,000 output tokens
Claude Opus 5:
20,000 * $4/M + 2,000 * $20/M = $0.120
GPT-5.6 Sol:
20,000 * $4/M + 2,000 * $24/M = $0.128
Gemini 3.7 Flash:
20,000 * $0.60/M + 2,000 * $3/M = $0.018
That is token cost only. Retries, tool calls, rejected patches, and long-context pricing can change the economics substantially.
Model-Specific Constraints
Claude Opus 5
Claude Opus 5 is available as claude-opus-5.
- Input: text, image, and PDF
- Context window: 1M tokens
- Routes: Chat Completions and Anthropic-compatible Messages
- Price: $4/M input and $20/M output
- Fixed scenario: $0.120
I would use Messages when the agent depends on Claude-specific controls. Chat Completions is more useful for a portable harness that switches between providers.
The 89% Terminal-Bench 2.1 result was measured at Max effort. That makes it a strong signal, but the higher output-token price needs to translate into a higher accepted-patch rate.
GPT-5.6 Sol
GPT-5.6 Sol uses the model ID gpt-5.6-sol.
- Input: text and image
- Routes: Chat Completions and OpenAI Responses
- Price up to 272K input tokens: $4/M input and $24/M output
- Price above 272K input tokens: $8/M input and $36/M output
- Fixed scenario at the short-context rate: $0.128
Responses is the natural route for OpenAI-native agent features. Chat Completions is the better fit for a common comparison harness.
The reported Terminal-Bench results need careful interpretation. The single-agent result is 88.8%. The 91.9% Ultra result uses a multi-agent setup, so I would not compare it directly with a single-agent run.
Gemini 3.7 Flash
Gemini 3.7 Flash uses the model ID gemini-3.7-flash.
- Input: text, image, video, audio, and PDF
- Context window: 1,048,576 tokens
- Routes: Chat Completions and Gemini-native generation
- Price: $0.60/M input and $3/M output
- Fixed scenario: $0.018
Gemini-native generation makes sense when Google-specific features matter. Chat Completions is more convenient for a shared harness.
The large context window and multimodal input expand the possible test surface, especially for design-to-code workflows. They do not prove that repository navigation or stale-context handling will be good.
What the Other Benchmark Says
Terminal-Bench is not enough to characterize a coding agent. The published DeepSWE v1.1 figures provide another signal:
| Evidence | Claude Opus 5 | GPT-5.6 Sol | Gemini 3.7 Flash |
|---|---|---|---|
| Terminal-Bench 2.1 | 89% at Max effort | 88.8%; 91.9% Ultra | 85.8% |
| DeepSWE v1.1 | 73.7% | 72.7% | 65.3% |
| Context window | 1M tokens | 1,050,000 tokens | 1,048,576 tokens |
| 20K input + 2K output | USD 0.120 | USD 0.128 at short-context rate | USD 0.018 |
DeepSWE v1.1 measures long-horizon software engineering in real codebases. I would only compare those numbers when the scaffold and evaluation setup match.
Context capacity is also not retrieval quality. A model can accept a million-token repository and still fail symbol discovery, cross-file consistency, or stale-context handling.
Build a Repository-Level Evaluation
A useful comparison requires more than sending the same prompt three times. Pin the repository, expose the same tools, use the same dependency versions and test commands, and enforce the same executable acceptance checks.
I use four task classes because each reveals a different failure mode:
| Task | What the agent must do | Acceptance condition |
|---|---|---|
| Repair a failing CI build | Read the failure log, trace a dependency or type error through the manifest, source, and tests, then run the affected suite. | CI is green without disabling checks or introducing regressions. |
| Complete an SDK migration | Update imports, configuration, types, and tests across multiple packages while preserving the public interface. | The full suite passes with no deprecated calls or interface breaks. |
| Patch a security finding | Follow the vulnerable call path, make the smallest safe change, add a regression test, and explain the risk boundary. | The exploit test fails, the regression test passes, and unrelated behavior is unchanged. |
| Implement a UI from a reference | Use a screenshot or design-system input, reuse existing components, and update visual or interaction tests. | Functional tests and visual thresholds pass without adding a duplicated component layer. |
Keep these variables fixed:
- Dependency versions
- Tool schemas
- Token limits
- Retry policy
- Execution sandbox
- Test commands
- Fallback behavior
Disable fallback during the model comparison. Otherwise, a successful patch may be attributed to the wrong model.
The primary result should be accepted-task rate. After that, report:
- Tool-call success
- Regression count
- p50 and p95 end-to-end latency
- Total token spend
- Cost per accepted patch
Store the repository commit hash, raw responses, tool traces, usage records, and environment details. Without those artifacts, the result will be difficult to reproduce.
Choosing the Primary Model
I would choose Claude Opus 5 when the agent needs Messages-native controls or when its Max-effort performance remains strong on multi-file repository tasks. Its published benchmark result is slightly ahead of GPT-5.6 Sol's single-agent result, but the output price needs to be justified by fewer retries or more accepted patches.
I would choose GPT-5.6 Sol when the agent is built around Responses or when difficult terminal and long-horizon tasks justify the premium. Track the 272K input-token threshold in cost estimates, and keep the Ultra result separate from single-agent measurements.
I would choose Gemini 3.7 Flash when raw token cost, multimodal input, or a 1,048,576-token context window matters. It is the cheapest in the fixed request scenario, but rejected patches and retries can remove that advantage quickly.
One Harness, Multiple Providers
The common path for these three models can use one OpenAI-compatible base URL, one key, and Chat Completions, switching only the model ID:
https://api.cometapi.com/v1
claude-opus-5
gpt-5.6-sol
gemini-3.7-flash
A unified multi-model API such as CometAPI is useful here when the goal is to keep authentication, request plumbing, retries, and usage accounting in one harness.
That does not eliminate provider-specific work. Messages, Responses, and Gemini-native generation expose different semantics and features. If production uses one of those native routes, benchmark that exact route rather than assuming the common path is equivalent.
| Model | Common route | Native route | Integration decision |
|---|---|---|---|
| Claude Opus 5 | Chat Completions | Anthropic-compatible Messages | Use Chat for common tests; Messages for Claude-specific behavior. |
| GPT-5.6 Sol | Chat Completions | OpenAI Responses | Use the endpoint that production orchestration actually calls. |
| Gemini 3.7 Flash | Chat Completions | Gemini-native generation | Use Chat for common tests; native generation for Google-specific behavior. |
Cost and Fallback Policy
Raw token price is a shortlisting metric. The production metric is:
cost per accepted patch =
total spend across attempts, tool calls, retries, and rejected patches
/
tasks that pass the acceptance suite
After selecting a primary model, evaluate fallback separately. Retryable failures include rate limits and HTTP 500, 503, 504, and 524 responses. Log which model ultimately served every request.
Invalid requests and authentication failures, including HTTP 400 and 401, should be fixed rather than rerouted.
Practical Answers
Is Claude Opus 5 better than GPT-5.6 Sol?
Their published Terminal-Bench 2.1 results are close: 89% for Claude Opus 5 at Max effort, versus 88.8% for GPT-5.6 Sol in the cited single-agent run and 91.9% in Ultra's parallel-agent setup.
Use Claude when Messages-native controls matter. Use GPT when Responses-native orchestration matters. For quality, rerun the same repository tasks because the published settings are not identical.
Is Gemini 3.7 Flash production-ready for coding agents?
It can be, if it passes the repository's acceptance gate. Its published results are 85.8% on Terminal-Bench 2.1 and 65.3% on DeepSWE v1.1, with the lowest raw token price in this comparison.
Measure accepted-patch rate, regression count, tool-call validity, latency, and retry rate before deploying it.
Which model has the best price-to-performance ratio?
There is no universal answer. Start with Gemini 3.7 Flash if raw token cost is the dominant constraint, then calculate total spend per accepted patch.
Claude Opus 5 or GPT-5.6 Sol can still be cheaper per successful task if they require fewer retries or produce fewer rejected changes.
Which is better for large repositories?
Claude Opus 5 lists a 1M-token context window, while Gemini 3.7 Flash lists 1,048,576 tokens. GPT-5.6 Sol lists 1,050,000 tokens.
Those capacities do not establish repository performance. Test symbol discovery, cross-file consistency, stale-context handling, and full-suite pass rate on the same pinned codebase.
Is GPT-5.6 Sol more expensive than Gemini 3.7 Flash?
For the fixed 20K input and 2K output scenario, GPT-5.6 Sol costs USD 0.128 at its short-context rate, compared with USD 0.018 for Gemini 3.7 Flash. GPT-5.6 Sol also moves to a higher rate above 272K input tokens.
The relevant comparison remains cost per accepted patch, not token price alone.
Can one coding-agent infrastructure switch among all three?
Yes, for the common Chat Completions path. The harness can retain one client and key while changing the model ID. Claude Messages, OpenAI Responses, and Gemini-native features require route-specific request handling.
That distinction should be reflected in the evaluation: compare the common route for portability, and benchmark native routes separately when they are part of the production design.
Originally published at cometapi.com
Top comments (0)