Yesterday Artificial Analysis shipped version 2.0 of its Coding Agent Index, and almost every agent on the board got worse on paper. Of the 21 configurations measured on both versions, 20 moved down. The mean index score fell from 54.8 to 47.4. No model regressed and no harness regressed. The test changed.
That distinction matters if you pick agents off leaderboards, or if you benchmark your own. Here is what actually moved, and how I read the new board.
What changed on October 23
The index now averages three benchmarks with equal weight, over 290 tasks total:
- FrontierCode v1.1 (from Cognition): 100 software engineering tasks on real open-source repositories. New in v2.0, replacing DeepSWE v1.1.
- Terminal-Bench 4.0 (Laude Institute): 66 terminal tasks. Carried over unchanged from v1.5.
- SWE-Atlas-QnA-AA (Scale AI's tasks, Artificial Analysis grading): 124 repository Q&A tasks. Same tasks, reworked grading.
Each task runs three times and the attempts are averaged. The full construction is on their methodology page.
FrontierCode grades like a maintainer
This swap is where the score drop comes from. The same 21 configurations that averaged 65.6 percent on DeepSWE v1.1 average 45.1 percent on FrontierCode v1.1.
The grading explains the gap. FrontierCode scores each attempt against a rubric of blocking and non-blocking checks. Fail a blocking check and the attempt scores zero, even if the code works. Pass all the blocking checks and you land between 70 and 100 percent, depending on the non-blocking ones. The benchmark's own framing: changes to real open-source repositories, graded on whether a maintainer would merge them.
The 100 tasks come out of a 150-task set that is not publicly released, which keeps the tasks out of search results and training data. That is the point of a private split.
The row is a variant, not a model
From the methodology page: each public row is an agent variant, not a model. Settings that change behavior, including reasoning settings, get their own rows.
At the top of the new board, Claude Code with Claude Opus 5.5 (max) leads at 61.7, ahead of Claude Code with Claude Sonnet 5.5 (max) at 59.3 and Muse Code with Muse Wonder Full (max) at 58.9. Claude Code with Claude Fable 5.1 (max with fallback) sits at 57.6. Same harness, three models, three scores. Two rows can share a model and sit several points apart on settings alone.
This is the part I keep pointing at when people ask whether agent configuration matters. The index cannot report a model's score without first fixing a harness and its settings. The model is half the measurement and the harness is the other half, and only one of those halves is under your control.
The judge got stricter too
SWE-Atlas-QnA kept its 124 tasks, but Artificial Analysis reworded 127 of the 1,306 rubric criteria and switched the judge to GPT-6 Luna (xhigh) with five votes per criterion. The dataset and judge configuration are published on Hugging Face.
The effect: across the 21 shared configurations, the Q&A mean fell 1.7 points, from 58.8 to 57.1 percent, with 14 of 21 moving down. Individual rows swung by up to 7 points. Muse Code with Muse Spark Open (xhigh) fell 7.0. Codex with GPT-6.1 Sol (max) gained 2.1.
When a judge revision moves rows by more than the gap between adjacent products, judge noise is part of the reading. A 2-point difference on a leaderboard is not a verdict.
The cost frontier survived
Four configurations sit on the cost frontier, per the launch announcement:
| Configuration | Index score | Cost per task |
|---|---|---|
| Codex with DeepSeek V4.1 Flash (max) | 47.3 | $0.39 |
| Codex with GPT-6.1 Sol (max) | 54.1 | $1.50 |
| Codex with GPT-6 Astra (max) | 54.7 | $7.04 |
| Claude Code with Claude Opus 5.5 (max) | 61.7 | $11.42 |
Those are pay-per-token API costs, not subscription prices. The leading row also averages about an hour of agent wall time per task, 3,661 seconds in the leaderboard's efficiency data. Cheapest and best are 14.4 index points and about $11 apart. Both ends of that line are rational picks, at different task volumes.
One methodology detail worth knowing before you compare these numbers against your own bills: cached input is priced separately from uncached input where the provider supports it, and cache-write charges are included. Prefix caching moves agent bills in ways a flat per-task figure hides, which I broke down with numbers here.
Cheating is scanned, not assumed
Agents get internet access on FrontierCode. A scan of each trajectory zeroes any attempt that fetched the upstream solution, meaning the source pull request, its merge commit, or a mirrored copy. Artificial Analysis runs a modified version of Cognition's rule-based scanner for this.
Terminal-Bench 4.0 adds an agent judge, Claude Code with Claude Sonnet 5, run through Harbor. It reads the full trajectory and flags the classics: editing the test files that grade the task, writing to the verifier's reward file, or fetching a reference solution from outside.
Two rules in the methodology deserve credit. Installing packages and reading library documentation count as normal work. And a search for the solution that turns up nothing is not a violation, as long as the agent works the answer out afterward. The line is whether the network supplied the answer, not whether the agent connected.
How to read the next leaderboard
What I check now, in order:
- The index version. v1.5 and v2.0 scores are not comparable. The 54.8 to 47.4 mean drop is what a yardstick change looks like. Treat any screenshot without a version stamp as noise.
- What the row actually is. A variant, with harness, model and settings fixed. "Claude Opus 5.5 scores 61.7" is a shorter sentence than the truth.
- The attempts behind it. Three attempts, averaged. A single benchmark run on your own repo is weather, not climate.
- The blocked rate. Safety refusals that block an attempt score zero, so a 2 percent blocked rate costs at most 2 points. Worth checking before you blame the model.
And if you run your own comparisons, pin the setup the way you pin dependencies:
# eval-setup.md, committed next to the results
benchmark: Coding Agent Index v2.0 (FrontierCode v1.1 / Terminal-Bench 4.0 / SWE-Atlas-QnA-AA)
agent: <name + version>
model: <name + reasoning setting>
attempts: 3, mean reported, date of run recorded
I keep making this argument for versioning agent config files, and benchmark versions deserve the same discipline. An eval you cannot reproduce is a number you cannot defend.
What I take from the reset
The drop from 54.8 to 47.4 is not bad news. FrontierCode's maintainer standard is closer to the PRs I actually review than a diff-applies-and-tests-pass check is. If the industry's headline number gets harder and more honest, that helps everyone choosing tools.
The packing is the other takeaway. On FrontierCode, 15 of the 24 configurations on the index score between 43 and 49 percent. The field is tight in raw capability. Which means the harness, the settings and the instructions you wrap around a model are not a rounding error in the outcome. They are a measurable part of it, and a leaderboard made of agent variants keeps proving it.
Top comments (0)