8 Frontier Models, 10 Real Enterprise Tickets: The Benchmark That Humbled the Leaderboards

If you believe the vendor announcements, coding agents are a solved problem. OpenAI describes GPT-6 Astra as its most capable model yet. Every lab ships a leaderboard where the latest release clears every bar.

Then a benchmark called Real-SWE landed on Hacker News this week and pulled up 271 points and 151 comments in a day, mostly engineers saying some version of "finally, numbers that match what I see." The setup is simple and brutal. The team at Specific Labs took 10 real tasks from private, real-world enterprise codebases. Not toy repos, not SWE-bench puzzles that half the internet has already memorized. Real tickets that real engineers worked on, some for weeks. They gave 8 frontier models 8 attempts each and graded against the actual merged PRs.

The best model, Claude Fable 5.1, completed 38.8% of them. OpenAI's GPT-6 Astra: 33.8%. GPT-5.6 Sol, an OpenAI flagship from earlier this year: dead last at 16.2%. And one task, an analytics stream reducer, went 0 for 64 attempts across all eight models.

Full disclosure before we go further: I have not run this benchmark myself. Everything below comes from the Real-SWE results page, the benchmark authors' comments on the Hacker News thread, and the practitioners in that discussion who have built private versions of the same test on their own codebases. But the failure data in this benchmark is more useful than any leaderboard I have seen this year, because it does not just say who won. It says how everyone lost.

What Real-SWE actually measures

Most coding benchmarks share one flaw: the tasks are public, the repos are public, and the solutions have leaked into training data. Real-SWE attacks that from three directions.

• Private codebases. The 10 tasks come from enterprise codebases that have never been public. The models cannot have memorized them.
• Hidden specifications. The prompt gives the model the ticket, but grading also checks against unwritten requirements the human engineer had to discover. Passing tests is not enough. The benchmark reveals the hidden spec only when grading.
• Weeks-scale scope. The benchmark author confirmed on Hacker News that these are "ambitious tasks that real engineers worked on for weeks," not one-session fixes. Tasks carry names like Customer identity migration, Billing schedule migration, Tax jurisdiction, and Linearizable scan.
• Fair-ish harness rules. Each model ran on its native provider harness at high reasoning, 8 rollouts per task, with per-rollout cost estimates published alongside scores.

That last bullet matters more than it looks. The headline resolution rate and the cost per rollout tell two different stories, and when you put them side by side, the ranking shuffles.

The scoreboard, with price tags

All figures are medians from the Real-SWE results page.

• Claude Fable 5.1: 38.8%, about $6.96 per rollout
• GPT-6 Astra: 33.8%, about $4.67 per rollout
• Gemini 3.8 Flash: 31.2%, about $2.50 per rollout
• GLM 5.3: 28.8%, about $5.12 per rollout
• Grok 4.6: 23.8%, about $3.44 per rollout
• Muse Spark 1.3: 23.8%, about $2.74 per rollout
• Kimi K3: 18.8%, about $3.90 per rollout
• GPT-5.6 Sol: 16.2%, about $2.65 per rollout

Look at the gap between first and third place. Fable 5.1 outscored Gemini 3.8 Flash by 7.6 percentage points while costing 2.8 times more per rollout. If your agent loop runs a model against dozens of tickets a week, a mid-tier model with a slightly lower hit rate can be dramatically cheaper per solved task. Solving a task with Gemini costs roughly $8 of compute on average. Solving one with Fable costs about $18. Whether Fable's extra quality is worth that depends entirely on how expensive a failed or half-finished ticket is in your workflow.

One more surprise: GLM 5.3, an open-weights model, took fourth place ahead of Grok, Muse Spark, Kimi, and GPT-5.6 Sol. The benchmark author's explanation on Hacker News: "the tasks on the benchmark are long horizon SWE tasks, where GLM does surprisingly well." Several commenters pushed back because the ranking contradicts their daily driver experience, which is fair. But that contradiction is itself the finding, and it leads straight to the most interesting part of the data.

Six of ten tasks have a resolution rate below 15%

The aggregate score hides the shape of the difficulty. Broken out per task, the benchmark stops looking like one leaderboard and starts looking like a map of what agents cannot do.

• API keys and environments was the most solvable task at 65.6% overall. Three models passed all 8 rollouts. This is well-defined plumbing work: rotate keys, wire up environment config, keep tests green. Agents are genuinely good at it.
• Tax jurisdiction collapsed to 3.1% overall. The single best result was 1 out of 8. A model has to understand tax rules that live in business logic, regulatory documents, and institutional knowledge, not in the code.
• Analytics stream reducer went 0 for 64. Not one rollout from any of the 8 frontier models produced a correct implementation.

The pattern across the hard tasks is consistent: the less a problem is fully specified inside the codebase, the worse every model does. Tax jurisdiction requires knowing things no repo contains. The stream reducer requires holding a distributed-systems design in your head while changing it. The benchmark author was candid that lower success rates happened because "we gave models ambitious tasks that real engineers worked on for weeks."

One Hacker News commenter distilled it in a line that got wide agreement: "The closer your project is to CRUD, the higher the chance of success."

Missed requirements are the number one killer

This is the section worth screenshotting. Real-SWE classified every failed rollout into failure categories, and the distribution is wildly different per model. Two agents with the same score can fail in opposite ways, which changes which one you should hire for which job.

• Missed requirements is the most common failure overall. Grok 4.6 failed this way on 67.2% of its failed runs. Kimi K3 on 53.8%. These models wrote code that worked, but not the code that was asked for.
• Unverified assumptions dominates for OpenAI models. GPT-5.6 Sol failed this way on 43.3% of failed runs, GPT-6 Astra on 34.0%. The model guesses a constraint instead of checking it, then builds on the guess.
• Integration errors are Gemini 3.8 Flash's signature failure: 49.1% of its failed runs broke something in how the change connected to the surrounding system. Muse Spark 1.3 shows 41.0%.
• Regressions, breaking previously working behavior, were rare for everyone but rarest for the open-weights models. GLM 5.3, Grok 4.6, and Kimi K3 all recorded 0% regression failures. Gemini was the worst at 10.9%.

Read those together and a practical selection guide falls out. If your tickets are well-specified and the risk is breaking existing behavior, the open models look better than their ranking suggests, because they fail safely by touching less. If your problem is requirement discovery, a model that assumes instead of verifies is dangerous no matter how impressive its demos are. Astra topped the leaderboard by volume but a third of its failures came from building on unchecked guesses. The commenters who run mixed setups said the same thing from experience: one practitioner uses a "watcher" model whose only job is to catch the implementing model's unverified assumptions, and reported that reversing the roles made both worse.

The Hacker News thread stress-tested it

No benchmark survives 151 comments unscathed, and the objections are worth knowing before you cite these numbers.

• Privacy skeptics asked whether sending private codebases to model APIs really keeps them private. The author confirmed the codebases stayed private, but the deeper point stands for your own experiments: whatever you send to a hosted API should be considered shared with the provider.
• Ranking dissenters pointed out that GLM beating Sol contradicts both their experience and other leaderboards. True, and unresolved. This is 10 tasks, not 500. Treat it as one data point with unusually good methodology, not ground truth.
• Practitioners validated the number that matters. Multiple people independently said the 20 to 35% range matches what they see on their own codebases. One wrote that the ~30% figure "tracks with my experience. I thought I was going insane for expecting too much from the models." Another, who built a private version of this exact test, moved from vibes to data and reported that open models were "very worthwhile to layer in as reviewers."

That last commenter's setup is the actionable part of the whole thread, because it means you do not have to wait for benchmark maintainers to test models on your reality.

Build your own private eval in an afternoon

Here is the playbook, assembled from the HN discussion. It needs no new tooling, just git history and discipline. Disclosure: this is the workflow described by practitioners in the thread, presented as their method, not something I have run myself.

• Pick 3 to 5 past tickets that landed as clean PRs. Favor ones with a clear starting commit and a meaningful test suite. This is the slowest step; ticket-to-PR history is rarely tidy in real repos.
• Rewind the repo to the commit just before each change. The agent works from the same starting point the human engineer had.
• Sandbox it with everything the agent needs at that commit: dependencies, credentials for test systems, a network policy that matches your real constraints. Narrow its ability to wander outside the project.
• Use the original ticket as the prompt. No polishing, no hints you did not write at the time. The whole point is measuring how the model handles the spec the way it actually existed.
• Grade against hidden specs, not just tests. Diff the agent's work against the accepted PR and check the requirements that never made it into the ticket. That is where missed requirements and unverified assumptions will surface.
• Run each model several times. Outputs vary at temperature 1. One practitioner's advice: averages matter less than ten vertical samples, which reveal a model's tendencies, like whether it overengineers or whether it stops short.
• Expect the exercise to improve your tickets. The recurring report from people who did this: writing eval tasks made them better at writing real tickets, because they started noticing which requirements lived only in their head.

Even a 5-task, single-model comparison on your own codebase tells you more than any public leaderboard, because it is measured against your domain, your conventions, and your definition of done.

The takeaway I keep coming back to

The vendors' story is that coding is nearly solved. Real-SWE's story is that frontier agents solve about a third of genuinely hard enterprise tickets, nobody solves the ones that need knowledge outside the repo, and the models fail in characteristically different ways that no single ranking captures.

That second story is more useful. It tells you where agents earn their keep today: well-specified, codebase-contained work, with a human owning requirement discovery. It tells you what to watch when you pick a model: not the average score, but the failure mode. And it tells you the cheapest path to certainty for your own stack is a small private eval, not someone else's leaderboard.

If you want the full data, the benchmark with all rollouts and costs is public at withspecific.com/benchmarks/real-swe, and the Hacker News discussion is worth reading in full.

I write about AI engineering, benchmarks, and backend development every week. Subscribe, it is free, and it means the next deep dive lands in your feed instead of the algorithm's mercy.

Have you measured any coding agent against your own codebase rather than a public benchmark? What surprised you? I am collecting stories for a follow-up piece on private eval setups.