OpenAI's GPT-5.6 Sol scored 38.3% on ARC-AGI-3 with custom API settings, beating Opus 5's 30.2%, but scored 7.8% in the official harness, exposing benchmark parity issues.
OpenAI's GPT-5.6 Sol scored 38.3% on ARC-AGI-3 with custom API settings, beating Anthropic's Opus 5 at 30.2%. But in the official test harness, the model landed at 7.8%, sparking a debate over benchmark fairness.
Key facts
- GPT-5.6 Sol: 38.3% with custom API, 7.8% in official harness.
- Opus 5: 30.2% on ARC-AGI-3.
- ARC Prize co-founder François Chollet acknowledged parity issue.
- Official harness used older API that lacked OpenAI's features.
- OpenAI used Retained Reasoning and Compaction settings.
OpenAI is claiming a win on ARC-AGI-3, the logic benchmark that Anthropic's Claude Opus 5 dominated just last week. According to The Decoder, GPT-5.6 Sol achieved 38.3 percent using OpenAI's Responses API with two custom settings: "Retained Reasoning," which preserves chain-of-thought between steps, and "Compaction," which summarizes old context instead of truncating it. That beats Opus 5's 30.2 percent, but the score comes with a major caveat: in the official ARC-AGI-3 harness, which uses a standardized setup without provider-specific features, GPT-5.6 Sol scored just 7.8 percent.
The gap highlights a structural tension in how frontier models are evaluated. ARC-AGI-3 is designed to measure pure model reasoning, not the infrastructure around it. The official harness strips away API-level optimizations to ensure apples-to-apples comparisons. OpenAI argues that "benchmarks never measure just the model but also the technical setup around it," a point that ARC Prize co-founder François Chollet partially conceded. In a statement, Chollet distinguished between "harnesses custom-made to solve the benchmark" (disallowed) and general-purpose API settings "that were not developed for ARC-AGI-3 and that are available to all API users" (allowed). He acknowledged that different providers using different settings "does create a potential parity issue," but considers it acceptable "as long as the settings and the cost are clearly reported."
The API asymmetry problem
The real story is not which model is smarter, but that the benchmark infrastructure itself is no longer provider-neutral. The official ARC-AGI-3 harness reportedly used an older "OpenAI-style completions API" that lacked features like retained reasoning—features that the Claude API already offered. That means Anthropic's Opus 5 may have benefited from similar API-level advantages that OpenAI is now exploiting, but that OpenAI was effectively penalized for not having them earlier. Chollet noted "a lot of back and forth with OpenAI about how to best test their models, especially with regard to compaction," suggesting the benchmark organizers are aware of the asymmetry but haven't standardized the API layer.
This matters beyond ARC-AGI-3. As models increasingly rely on chain-of-thought, context management, and agentic loops, any benchmark that doesn't account for these infrastructure differences risks measuring API quality as much as model capability. OpenAI's 38.3 percent score is a number, but it's also a signal that the era of simple, single-harness evaluations is ending.
What the scores actually mean
OpenAI's 7.8 percent in the official harness is a reminder that GPT-5.6 Sol's raw reasoning—without the API crutch—is still far below Opus 5. But the 38.3 percent shows what the model can do when the infrastructure is aligned. For practitioners, the takeaway is that benchmark scores are increasingly a joint function of model and deployment stack. The ARC Prize's response—that settings and cost must be transparent—is a step toward comparability, but it doesn't solve the parity issue for future evaluations.
This is also part of a broader trend. OpenAI recently claimed GPT-5.6 Sol scored 72.7% on DeepSWE, another benchmark where API-level features like compaction matter. As previously reported, the company is pushing the frontier of what its API can do, not just what its model can do.
What to watch
Watch for the ARC Prize's next standardized harness version, which may mandate API-level settings to ensure parity. Also track whether OpenAI publishes GPT-5.6 Sol scores on ARC-AGI-3 with the official harness after a future API update that includes retained reasoning by default.
Source: the-decoder.com
[Updated 31 Jul via openai_blog]
The same efficiency push behind GPT-5.6 Sol's ARC-AGI-3 performance is now driving major price cuts. Starting July 30, OpenAI slashes GPT-5.6 Luna prices by 80% (to $0.20/M input, $1.20/M output) and Terra by 20%, crediting Sol for optimizing inference kernels and load balancing, reducing serving costs by 20%. [per OpenAI] Luna is now cheaper than Google's Gemini 3.1 Flash-Lite and a fifth of Anthropic's Claude Haiku 4.5 input price, reshaping the low-cost model landscape. This suggests the API-level advantages in benchmarks are part of a broader infrastructure efficiency strategy.
Originally published on gentic.news
Top comments (0)