If you want the highest first-try success rate on real enterprise code, Fable 5.1 running inside Claude Code wins: it resolved 38.8% of tasks on the Real-SWE benchmark at an estimated $6.96 per rollout, the most expensive setup measured (Specific Labs). For cost-sensitive teams, Gemini 3.8 Flash in Gemini CLI is the value pick at 31.2% for an estimated $2.50 per rollout, and GPT-6 Astra in Codex CLI splits the difference at 33.8% for $4.67 (Specific Labs). One honest caveat: the confidence intervals in that top cluster overlap, with Fable at roughly [32%, 45%], Astra at [27%, 40%] and Gemini at [25%, 38%], so read first versus third as directionally consistent rather than settled.
TL;DR
- Fable 5.1 via Claude Code leads Real-SWE at 38.8% resolution, followed by GPT-6 Astra via Codex CLI at 33.8% and Gemini 3.8 Flash via Gemini CLI at 31.2% (Specific Labs).
- Estimated cost per rollout ranges from $2.50 for Gemini 3.8 Flash to $6.96 for Fable 5.1 (Specific Labs).
- Every score is a model plus harness combination, not a model in isolation, which is why the same CLI produced 33.8% with GPT-6 Astra and 16.2% with GPT-5.6 Sol (Specific Labs).
- The tasks are hard: six of the ten sit below 15% resolution, and one, the analytics stream reducer, was never solved by any tested combination (Specific Labs).
- Letting agents run longer did not help. Rollouts under ten minutes failed 71.4% of the time against 73.4% for runs of ten minutes or more (Superpower Daily).
- Missed requirements, not broken syntax, is the dominant failure mode across the field (Specific Labs).
- Last verified: 2026-09-14.
Which LLM is best for coding on real enterprise code?
On the evidence available today, Fable 5.1 paired with Claude Code. Real-SWE, published in September 2026 by Specific Labs, evaluates frontier models on private production codebases rather than public repositories, with tasks licensed from real companies covering billing, tax, customer migration and infrastructure work (Specific Labs). Across eight model-and-harness configurations and ten tasks, the benchmark scored 640 rollouts, reporting resolution rate as pass@1 averaged over eight independent runs per task with 95% confidence intervals (Specific Labs).
The full leaderboard, with estimated cost per rollout:
| Rank | Model and harness | Resolution rate | Est. cost per rollout |
|---|---|---|---|
| 1 | Fable 5.1 via Claude Code | 38.8% | $6.96 |
| 2 | GPT-6 Astra via Codex CLI | 33.8% | $4.67 |
| 3 | Gemini 3.8 Flash via Gemini CLI | 31.2% | $2.50 |
| 4 | GLM 5.3 via Claude Code | 28.8% | $5.12 |
| =5 | Grok 4.6 via Grok Build | 23.8% | $3.44 |
| =5 | Muse Spark 1.3 via Muse Code | 23.8% | $2.74 |
| 7 | Kimi K3 via Kimi Code | 18.8% | $3.90 |
| 8 | GPT-5.6 Sol via Codex CLI | 16.2% | $2.65 |
All figures from Specific Labs. Usage data was incomplete for Grok 4.6 and Kimi K3, so their real costs may be higher. The cost spread is corroborated independently by LavX News.
If you are comparing the two flagship contenders directly, our earlier breakdown of GPT-6 Astra against Claude Fable 5.1 covers pricing and behaviour outside benchmark conditions.
Why is the benchmark leader also the most expensive?
Because Fable 5.1 writes far more. On the tax jurisdiction task it produced a mean of 78,000 output tokens per rollout, 95,000 on API token metering, 86,000 on the linearizable scan and 88,000 on the analytics stream reducer. GPT-6 Astra worked in a band of 13,000 to 32,000 output tokens per task (Specific Labs). That difference in verbosity, not a difference in list pricing, is the mechanism behind the gap between $6.96 and $2.50 per rollout (Specific Labs).
For engineering managers the practical read is straightforward: you are buying a few extra percentage points of first-try success with a lot of extra tokens. If a human reviews every agent patch anyway, the cheaper setup with a second attempt may land in the same place for less money. Our cost-per-task comparison of Kimi K3 and Claude Fable 5 works through that arithmetic in more detail.
Why do the same models score so differently across CLIs?
Because Real-SWE scores combinations, and the page says so explicitly: results describe a model together with its harness, not a model on its own (Specific Labs). The clearest illustration is within a single CLI. On Codex CLI, GPT-6 Astra reached 33.8% while GPT-5.6 Sol reached 16.2%. Inside Claude Code, Fable 5.1 reached 38.8% and GLM 5.3 reached 28.8% (Specific Labs).
We see the same sensitivity in our own testing. Across three trials each on an identical seven-constraint article-planning task, Gemini 3.8 Flash (High) and Claude Opus 4.6 (Thinking) both scored 17 of 17 on machine-checked constraint adherence. Median wall time was 23 seconds for Gemini against 67 seconds for Opus. That is a small sample, n=6, measured on 2026-09-14 with our own harness via Antigravity CLI and machine scored, so treat it as a signal rather than a result. The point it supports is the one Real-SWE makes: pairing and effort settings move cost and speed even when output quality ties.
If you are choosing between agent shells rather than models, our Codex CLI versus Claude Code comparison covers the harness side.
How hard are the Real-SWE tasks, and why does that matter?
Hard enough that averages hide most of the story. Six of the ten tasks scored below 15% resolution (Winzheng). The multi-region sweep task reached 67.2% resolution while the tax jurisdiction task reached 3.1%, and the analytics stream reducer produced no successful rollouts at all among the tested combinations (Specific Labs).
Scope explains part of that. Reference solutions edit a median of 11 files, against 6 in FrontierCode and DeepSWE according to the published comparison linked from the benchmark, with a median instruction length of 1,742 characters (Specific Labs). Tasks were run in isolated sandboxes in Harbor format, with verifiers injected at grading time and drawn from or inspired by the codebase's existing test suites.
The failure taxonomy is the useful part for anyone deploying agents. Missed requirements is the most common failure category across the field: 67.2% of Grok 4.6's failed runs, 41 of 61, and 53.8% of Kimi K3's, 35 of 65, alongside 38.6% for GLM 5.3 and 36.7% for Fable 5.1 (Specific Labs). These agents are not mostly failing on compilation. They are failing to do everything the ticket asked. Longer runs do not fix that either, given the near-identical failure rates for short and long rollouts (Superpower Daily).
What should you actually do with this?
Pick by constraint. If review time is your bottleneck and the work touches billing or migrations, run Fable 5.1 in Claude Code and accept the token bill. If you are running agents at volume across a large backlog, Gemini 3.8 Flash in Gemini CLI gives you most of the capability for a fraction of the spend. Then attack the dominant failure mode directly: write tickets with explicit, enumerated acceptance criteria, because missed requirements is what breaks these runs, not code that will not compile.
For wider context, see our best LLM for coding cluster page, the speed-tier comparison if latency matters more than depth, and our local runtime comparison if your code cannot leave your network.
FAQ
Q: Which LLM is best for coding on private enterprise codebases?
A: Fable 5.1 running in Claude Code, which resolved 38.8% of Real-SWE tasks, ahead of GPT-6 Astra in Codex CLI at 33.8% and Gemini 3.8 Flash in Gemini CLI at 31.2% (Specific Labs). The confidence intervals in that group overlap, so the ordering is directional.
Q: Is the most accurate coding model worth the extra cost?
A: Not automatically. Fable 5.1 is estimated at $6.96 per rollout against $2.50 for Gemini 3.8 Flash (Specific Labs), and the gap comes from output volume, with Fable producing 71,000 to 95,000 output tokens on several tasks versus 13,000 to 32,000 for GPT-6 Astra (Specific Labs).
Q: Does giving a coding agent more time improve its results?
A: No, on this evidence. Rollouts shorter than ten minutes failed 71.4% of the time and rollouts of ten minutes or longer failed 73.4% (Superpower Daily).
Corrections
No corrections to date. Spot an error in the figures above? Send it in and we will update the article with a dated note.
Top comments (0)