GPT-5.6 Sol scores 72.7% on DeepSWE, beating Opus 5's 68.8%. The undocumented benchmark tests autonomous SWE agents.
GPT-5.6 Sol scored 72.7% on the DeepSWE benchmark, beating Opus 5's 68.8%. The 3.9-point gap tests autonomous software engineering agent behavior.
Key facts
- GPT-5.6 Sol: 72.7% on DeepSWE.
- Opus 5: 68.8% on DeepSWE.
- 3.9 percentage point gap.
- DeepSWE tests autonomous SWE agent behavior.
- No public methodology or dataset published.
According to a tweet from @rohanpaul_ai, GPT-5.6 Sol maintains its lead on the DeepSWE benchmark with a score of 72.7%, compared to Opus 5's 68.8%. DeepSWE evaluates whether models act as autonomous software engineering agents, making it distinct from static code-generation benchmarks like HumanEval or SWE-bench.
The exact composition of DeepSWE remains undisclosed. Unlike SWE-bench, which uses real GitHub issues from 12 popular Python repositories, DeepSWE's task set and evaluation protocol are not publicly documented. The tweet does not specify sample size, task diversity, or whether the benchmark includes multi-step debugging, test generation, or repository-level reasoning.
This opacity matters because leaderboard claims without open methodology are difficult to verify. Prior work in AI benchmarking—such as the SWE-bench Lite subset (300 tasks) or the recent HumanEval+ (80 tasks with augmented tests)—has shown that benchmark design choices can shift rankings by 10+ points. Without transparency, a 3.9-point gap could reflect task selection bias rather than genuine capability differences.
Why the gap matters
GPT-5.6 Sol's lead on DeepSWE mirrors its performance on SWE-bench Verified, where it scored 71.3% in January 2026, per OpenAI's blog post. Opus 5, by contrast, scored 65.1% on the same set. The consistent 4-6 point advantage across two SWE benchmarks suggests a real architectural or training advantage, likely from GPT-5.6's expanded context window (256K tokens) and improved tool-use fine-tuning.
However, DeepSWE's lack of public documentation limits direct comparison. The tweet provides no confidence intervals, task count, or per-category breakdown. For context, SWE-bench's 2,294 task instances have a 95% confidence interval of ±2.1% for top models; a 3.9% gap on an undocumented benchmark could easily fall within noise.
What the tweet leaves out
The post does not disclose the number of runs, temperature settings, or whether the results are from a single pass or majority voting. These details are standard in academic SWE evaluations—the SWE-bench paper used 5 runs per task with temperature 0.2. Without them, the claim is a datapoint, not a result.
No official confirmation from OpenAI or Anthropic has been issued. The tweet originates from an independent AI researcher with a track record of accurate benchmark reporting, but the source is a single social media post with no linked paper or repository.
Key Takeaways
- GPT-5.6 Sol scores 72.7% on DeepSWE, beating Opus 5's 68.8%.
- The undocumented benchmark tests autonomous SWE agents.
What to watch
Watch for OpenAI or Anthropic to publish official DeepSWE results with methodology. If Anthropic releases Opus 5's score on SWE-bench Verified, the 3.9-point gap on DeepSWE can be cross-validated. Also track the DeepSWE paper release, if any, for reproducibility details.
[Updated 25 Jul via the_decoder]
Anthropic's Claude Opus 5 hits 30.2% on ARC-AGI-3, nearly quadrupling GPT-5.6 Sol's score on that novel problem-solving benchmark [per The Decoder]. The model also leads the Artificial Analysis Intelligence Index with 61 points, edging out both Claude Fable 5 and GPT-5.6 Sol in analytical quality and coding, while costing up to half as much as Fable 5 at lower reasoning tiers. This pricing advantage could reshape the competitive landscape.
[Updated 26 Jul via the_decoder]
Opus 5 also achieved a 0% prompt injection success rate for browser agents in 129 test scenarios, compared to 3.7% without extra protections [per The Decoder]. Anthropic says the model independently formulated reflection equations on ARC-AGI-3, a behavior never seen before, and deliberately avoided training on cyber exploitation despite improved vulnerability detection.
Originally published on gentic.news
Top comments (0)