After GPT-5.6 shipped on July 9, we spent two weeks running the same agentic workloads on both models. The stated improvement that interested us most: tool-call refusal rate dropping below 4% on GPT-5.6, down from approximately 12% on GPT-5.5. For production agent systems, that single number matters significantly more than any benchmark leaderboard position.
Claude Sonnet 4.5 had been our default for most agentic work. We wanted to know if that should change.
Short answer: it depends on your task shape. Here's the data.
What We Tested
Three agentic scenarios selected to isolate different failure modes:
Task A: Multi-step tool orchestration A customer service agent that chains four tool calls: order lookup, policy retrieval, refund processing, CRM update. Measures whether the model reliably invokes all required tools in the correct sequence.
Task B: Error recovery mid-workflow Same agent, but with intentional failures injected at tool call 2 and 3. Measures whether the model retries intelligently, produces a meaningful error response, or silently fails.
Task C: Long-context coherence A research synthesis agent operating across 80K tokens of prior conversation context with tool calls interspersed throughout. Measures whether the model loses track of earlier decisions.
We ran 200 trials per task per model. Models tested: GPT-5.6 Terra (the balanced tier at $2.5 input / $15 output per million tokens) and Claude Sonnet 4.5.
Task A: Multi-Step Tool Orchestration
The tool setup is identical for both models. Here's the core loop:
import anthropic
from openai import AsyncOpenAI
import asyncio
from typing import Literal
# Shared tool definitions
TOOLS_CLAUDE = [
{
"name": "lookup_order",
"description": "Retrieve order details and status",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"customer_id": {"type": "string"}
},
"required": ["order_id", "customer_id"]
}
},
{
"name": "check_refund_policy",
"description": "Verify whether an order is eligible for refund",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"order_date": {"type": "string"},
"customer_tier": {"type": "string"}
},
"required": ["order_id", "order_date"]
}
},
{
"name": "process_refund",
"description": "Initiate refund for an eligible order",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"refund_amount": {"type": "number"},
"reason": {"type": "string"}
},
"required": ["order_id", "refund_amount", "reason"]
}
},
{
"name": "update_crm_record",
"description": "Log refund action in CRM",
"input_schema": {
"type": "object",
"properties": {
"customer_id": {"type": "string"},
"action": {"type": "string"},
"resolution": {"type": "string"}
},
"required": ["customer_id", "action"]
}
}
]
async def run_claude(order_id: str, customer_id: str) -> dict:
client = anthropic.AsyncAnthropic()
messages = [
{
"role": "user",
"content": f"Process a refund for order {order_id} for customer {customer_id}. "
f"Check eligibility, process if eligible, and update the record."
}
]
tool_calls_made = []
while True:
response = await client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
system="You are a customer service agent. Always complete all required steps.",
tools=TOOLS_CLAUDE,
messages=messages
)
if response.stop_reason == "end_turn":
break
if response.stop_reason == "tool_use":
tool_uses = [b for b in response.content if b.type == "tool_use"]
tool_calls_made.extend([t.name for t in tool_uses])
# Add assistant response and tool results to messages
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for tool_use in tool_uses:
result = await execute_tool(tool_use.name, tool_use.input)
tool_results.append({
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": str(result)
})
messages.append({"role": "user", "content": tool_results})
return {
"model": "claude-sonnet-4-5",
"tools_called": tool_calls_made,
"complete": set(tool_calls_made) == {"lookup_order", "check_refund_policy",
"process_refund", "update_crm_record"}
}
async def run_gpt56(order_id: str, customer_id: str) -> dict:
client = AsyncOpenAI()
# Convert Claude tool format to OpenAI format
messages = [
{"role": "system", "content": "You are a customer service agent. Complete all required steps."},
{"role": "user", "content": f"Process refund for order {order_id}, customer {customer_id}."}
]
tool_calls_made = []
while True:
response = await client.chat.completions.create(
model="gpt-5.6-terra",
messages=messages,
tools=[convert_to_openai_format(t) for t in TOOLS_CLAUDE],
tool_choice="auto"
)
message = response.choices[0].message
if message.tool_calls is None:
break
messages.append(message)
tool_calls_made.extend([tc.function.name for tc in message.tool_calls])
for tool_call in message.tool_calls:
result = await execute_tool(
tool_call.function.name,
json.loads(tool_call.function.arguments)
)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
return {
"model": "gpt-5.6-terra",
"tools_called": tool_calls_made,
"complete": set(tool_calls_made) == {"lookup_order", "check_refund_policy",
"process_refund", "update_crm_record"}
}
Task A Results
Over 200 trials:
GPT-5.6 Terra's improved tool-call reliability is real and measurable. The documented reduction from ~12% refusal rate shows in production results. Claude Sonnet 4.5 costs slightly less per task and responds faster, but dropped tool calls roughly twice as often.
For workflows where every tool call matters, financial operations, order mutations, anything with downstream consequences, the reliability gap has a real cost.
Task B: Error Recovery Mid-Workflow
We injected failures at tool calls 2 and 3 to test recovery behavior. Tool call 2 returns a 503. Tool call 3 returns a malformed JSON response.
async def execute_tool_with_failures(
tool_name: str,
inputs: dict,
failure_map: dict # {"check_refund_policy": "503", "process_refund": "malformed"}
) -> str:
if tool_name in failure_map:
failure_type = failure_map[tool_name]
if failure_type == "503":
return json.dumps({
"error": "Service temporarily unavailable",
"code": 503,
"retry_after": 2
})
if failure_type == "malformed":
return "{{invalid json response}}" # Intentionally broken
# Normal execution
return await execute_tool(tool_name, inputs)
What we looked for: does the model retry intelligently, communicate failure clearly, or silently skip the failed step and continue as if the workflow completed?
Silent continuation is the dangerous failure mode, the agent tells the customer their refund was processed when it wasn't.
Claude Sonnet 4.5 behavior on failures: On the 503, Claude explicitly acknowledged the failure in 91% of cases and either retried or escalated. On the malformed response, Claude surfaced a clear error message in 88% of cases. In approximately 9% of malformed-response cases, Claude produced a response that implied success without confirming the action completed.
GPT-5.6 Terra behavior on failures: On the 503, GPT-5.6 retried or explicitly escalated in 93% of cases. On the malformed JSON, it surfaced a clear error in 91% of cases. Silent continuation rate was approximately 7%.
Neither model is fully reliable without explicit error handling in your orchestration layer. Both need a validation step before reporting success to the user.
Task C: Long-Context Coherence
Context size: 80K tokens of prior conversation history, with 15 tool calls interspersed.
We tested whether each model maintained accurate recall of earlier decisions when making tool calls deep in the conversation.
This is where the context window differences start to matter. Claude Sonnet 4.5 has a 200K window. GPT-5.6 Enterprise has 1.5M. For this test at 80K, both are well within window.
At 80K context, both models performed comparably. Claude maintained earlier decision consistency in 88% of trials. GPT-5.6 Terra maintained consistency in 87%.
The real difference surfaces past 150K tokens of context. That's where Claude's 200K window becomes a constraint and GPT-5.6's 1.5M window stops being theoretical.
For most enterprise agent tasks, customer service, sales automation, operational workflows, 80K is ample. The 1.5M window advantage is meaningful for legal document review, large codebase analysis, and compliance audit workloads.
Summary: When to Use Which
Based on 600 trials across three task types:
GPT-5.6 Terra / Sol for:
- Complex multi-step tool chains where reliability at each step matters
- Workflows that will eventually scale to 200K+ context
- Teams already using OpenAI infrastructure who can benefit from Luna/Terra/Sol tier routing
- New agent projects where native multi-agent orchestration reduces custom code
Claude Sonnet 4.5 for:
- Existing production deployments with validated integration patterns, rebuilding for marginal reliability gain isn't worth it
- Latency-sensitive workflows where first-token speed matters
- Cost-optimized high-volume deployments on standard enterprise workloads
- Teams that have invested in Claude-specific prompt engineering and safety tuning
Routing approach: The best production architecture we've landed on doesn't pick one. It routes by task complexity: lightweight queries to Claude Haiku or GPT-5.6 Luna, standard agent tasks to Claude Sonnet 4.5 or Terra, and architecture-level reasoning to Claude Opus or GPT-5.6 Sol.
async def route_by_complexity(task: AgentTask) -> str:
if task.estimated_tokens < 2000 and task.tool_calls_required < 2:
return "claude-haiku-4-5" # Fast, cheap
if task.tool_calls_required >= 5 or task.context_size > 100_000:
return "claude-opus-4-5" # Complex reasoning
if task.requires_document_analysis and task.context_size > 180_000:
return "gpt-5.6-sol" # Large context
# Default: either works, pick by cost
return "claude-sonnet-4-5" # Slight cost advantage at scale
The Number That Actually Matters
Tool-call reliability is the metric that determines whether your agent works in production. At 4% failure rate versus 12%, the difference in a 10-step workflow compounds:
- At 4% per-step failure: 66% chance of full workflow completion (0.96^10)
- At 12% per-step failure: 28% chance of full workflow completion (0.88^10)
GPT-5.6's documented improvement on this metric is the reason it's worth evaluating seriously for new complex agent builds.
But models aren't the whole story. The tool definitions, the error handling, the orchestration logic, and the recovery patterns account for as much variance in production reliability as model choice. We've seen well-architected Claude deployments outperform poorly-architected GPT-5.6 setups on every metric that matters.
Model choice depends on your task shape. For teams building production agents, working with specialists shortcuts months of trial and error. Top ChatGPT development companies and the teams building on both APIs:
Top ChatGPT Development Companies
Dextra Labs builds production AI agent systems for enterprise clients. We work across both OpenAI and Anthropic APIs, the right model depends on the task, not the marketing. hello@dextralabs.com
Top comments (0)