Real-SWE ran frontier models against licensed, private enterprise codebases (billing, tax, multi-service work) and one number jumped out at me: rollout duration barely moves resolution.
71.4% of rollouts that finished in under 10 minutes FAILED. 73.4% of rollouts that ran 10 minutes or longer also FAILED. Pass rate sits flat at 27-29% either way. Extending runtime from minutes to long rollouts shifts the outcome by about two percentage points, which is noise.
The leader, Fable 5.1 on Claude Code, only lands 38.8% resolution. GPT-6 Astra on Codex CLI gets 33.8%. The top model still fails roughly six out of ten private enterprise tasks.
This is the part vendor demos skip. The easy stuff gets solved fast, so on a short rollout you see high apparent throughput. But the tasks that matter, the ones buried in real payroll and tax and integration code, hit a structural wall. The agent doesn't run out of compute on those. It runs out of understanding, or context, or the harness doesn't give it the right entry point. Another ten minutes of looped retries doesn't fix any of that.
The other thing Real-SWE does right is treat each score as model+harness, not model alone. Fable 5.1 is only 38.8% paired with Claude Code's scaffold. That's a harness result on private code, not a statement about the model in a vacuum. So many leaderboards still publish model names with no harness pinned, and then people compare them across totally different scaffolding and draw nonsense conclusions.
Takeaway for anyone buying an agent: when a vendor shows you a pass rate, ask which slice that number came from. A model that looks great because it clears the fast, shallow tasks is hiding the exact set you actually need it to solve.
Benchmark source: withspecific.com/benchmarks/real-swe
Top comments (1)
The harness pinning point is the one that frustrates me most about these leaderboards. I've run the same model through two different scaffolds on our internal evals and gotten a 15-point gap on tasks that should be equivalent. The flat curve from 10 to 60 minutes also matches exactly what I see in data pipeline generation work: past a certain complexity threshold the agent just loops on plausible-looking wrong attempts, and more time just means more of those.