Best Model for Agentic Coding: October 2026 Benchmarks

There's no single best model for agentic coding this month, at least on our two benchmarks. I spent the weekend reading the code behind the boards, and the scores matter less to me now than they did on Saturday. The models at the top weren't cleverer in any way I could see. Mostly they checked who was asking before showing an issue, kept reading a list until Jira said it was the last page, and when they wrote "next" they did it. If you pay for long agent runs, I'd look there before the price.

Paying per token, I'd run GPT-6.1 Sol. It's 3rd on our Jira app benchmark, at $1.19 a build. On a tight budget, MiMo V2.6 Flash came 7th on the same benchmark for 31 cents a build. Scores run from 0 to 1.

Both benchmarks have a model build a real app through our fork of goose, the open-source coding agent, with nobody in the loop. Yes, a fork. We changed a lot of it for our own use. A scorer then runs the app. No model grades anything. Gauntlet 7.2 asks for a payments console with a 3D view of the payments, with 150 model calls and one run per model. Forge 2.0 hands the model v1 of a Jira app built on Atlassian's Forge platform and asks it to ship v2, with no internet and 300 calls. Each cloud model gets two independent builds there, and the board ranks the median.

The Forge 2.0 app is a scope ledger for Jira sprints. It writes one row for each issue added to or removed from a sprint. Each row records which path recorded it first, a Jira event or the hourly reconcile job that re-reads Jira's history for missed events. The upgrade has to migrate v1's rows, and they must keep the label v1 gave them. The scorer installs each build on the same three Jira test sites, each seeded with different data, and checks them at seven points along the way. A critical defect, such as showing a person an issue they may not see, multiplies the score by 0.6.

These are goose runs with nobody watching. If a model ends a turn without calling a tool, goose stops, and so does Claude Code by default (a Stop hook can make it push on). It's what ended both of Haiku's builds. Each judgement here rests on one or two builds on our setup.

Which model I'd use for agentic coding, by situation

1. Paying per token for long builds, GPT-6.1 Sol. Its builds took 20 and 25 minutes at $1.19 a build. MiMo V2.6 Flash is within 0.1 of it on both boards for a quarter of the bill, and two builds can't split them. I pay the difference because Sol earned more of the tests in both builds (0.95 and 0.94 of them, MiMo 0.88 and 0.93) and passed every webhook check twice.
2. On a Claude subscription through Claude Code, Claude Opus 5.5, if your plan's limits stretch to it. It has the best Forge 2.0 median of the models we called directly, with Sol within 0.1. Each build ran over two hours and billed about $21 at per-token prices, and we haven't measured what one build does to a plan's limits.
3. For the top Gauntlet score, GPT-6.1 Sol Pro, from one run, the only run the scorer didn't cap. It has no Forge 2.0 run yet. It's the same model as Sol run in OpenAI's pro reasoning mode, at the same price per token.
4. On a tight budget, MiMo V2.6 Flash. It's open weights (MIT), but at 309B parameters it isn't a laptop model.
5. Claude Sonnet 5.5 is level with Sol on quality at almost five times the bill. I'd still take Sol.
6. Not for big unattended builds, on our two builds each: Claude Haiku 5.5, whose builds ended on a text turn, Grok 4.7 at $26.68 a build for a 0.3248 median, or GLM-5.3 Prime at $49.83 for 0.4302.
7. Not a local model yet. The untrained Qwen3.8 27B base came 23rd of 31 on a MacBook, and the fine-tuned 27B hasn't run Forge 2.0.

The two boards on 20 October 2026

This closes off our runs from 2 to 19 October: 44 models, 43 entries on Gauntlet, 31 on Forge 2.0. I read both boards from our public dataset on Saturday 17 October, on Sunday 18 October at 19:34 EEST after DeepSeek's update came in, and again on Tuesday 20 October before this went out. Six models landed after the Sunday read: Pareto 26.10, Mistral Large 4, Nemotron 3.5 Super VL, Glimpse, Beam 1.0 and Muse Spark Open. Their places below are current, but I didn't read their builds for this piece, so none of the moments uses them. Each side is its own ranking, so read down a column. The last row skips down to Claude Haiku 5.5.

rank	Gauntlet 7.2 (one run each)	score	rank	Forge 2.0 (two builds each)	median	range
1	GPT-6.1 Sol Pro	0.9574	1	Jev Router (a router)	0.7460	0.6398 to 0.8523
2	Qwen3.8 Max Prime	0.8364	2	Claude Opus 5.5	0.6745	0.6196 to 0.7294
3	Claude Sonnet 5.5	0.7984	3	GPT-6.1 Sol	0.5933	0.5371 to 0.6495
4	GPT-6.1 Sol	0.7978	4	Qwen3.8 Max Prime	0.5765	0.5015 to 0.6515
5	Pareto 26.10	0.7971	5	Claude Sonnet 5.5	0.5692	0.5485 to 0.5899
6	Pareto 26.10 Preview	0.7957	6	Gemini 4 Argon	0.5434	0.2943 to 0.7926
7	Claude Opus 5.5	0.7926	7	MiMo V2.6 Flash	0.5283	0.5028 to 0.5538
8	MiMo V2.6 Flash	0.7909	8	Ember 1	0.5011	0.4710 to 0.5312
9	Muse Wonder	0.7638	9	DeepSeek V4.1 Flash 1017	0.4520	0.2422 to 0.6618
10	Gemini 4 Argon	0.7078	10	GLM-5.3 Prime	0.4302	0.4046 to 0.4557
11	Jev Router	0.6975	11	Qwen3.8 Flash	0.4077	0.3164 to 0.4990
12	DeepSeek V4.1 Flash 1017	0.6957	12	Gemini 3.8 Flash	0.3979	0.2999 to 0.4958
19	Claude Haiku 5.5	0.6610	29	Claude Haiku 5.5	0.0621	0.0610 to 0.0632

Ember 1 is Fireworks' model built on Kimi K3, and Jev Router is TypeSafe's router. OpenRouter's records showed 117 of Jev's 118 calls going to GPT-6.1 Sol (the Jev piece has the trace). We threw out nine invalid runs, and none of their scores are used here.

Treat a Gauntlet gap under a few hundredths as a tie. Forge 2.0 splits models into comparison groups by goose build. Opus, Sol, MiMo and the September DeepSeek share one. Max Prime, Sonnet, Haiku and most others share another. Jev Router and Gemini 4 Argon ran a newer goose release. DeepSeek's update shares a group only with Mistral Large 4. Qwen3.8 Flash and Muse Wonder each have one of their own, and Glimpse, Muse Spark Open and Beam 1.0 share another. The board advises comparing within a group, and the local 27B is in none, so a rank across groups is my reading.

Gauntlet caps a run's score for each band of conditions it misses, so places 3 to 8 sit under one cap, 0.0075 apart, a tie on one run each. Qwen3.8 Max Prime, second, wrote in its closing note that "every restart resumes, events stay contiguous". The scorer killed its event relay halfway through a write, and the relay never picked back up.

All five moments below are from Forge 2.0, where I could open both builds of each model.

Moment 1: "I'll get to them next", and who kept going

Claude Haiku 5.5 is the surprise of the month, and its own words explain the score. Build 1 ended after 16 of its 300 calls with this, and no tool call after it. (A call is one model response, which can ask for several tools.)

I've written the v2 ledger layer and the numbers module. The sync path, migration,
and resolvers are still to write, and I'm at about 18% of the context window.
I'll get to them next.

Build 2 lasted 37 calls and ended the same way: "Given how much remains, I'll stop expanding the design here and finish the work in order, testing each part as I go." Build 1's manifest pointed at an admin page file it never wrote, so Forge's own lint failed, and the only time it had run lint was at about call 4, before it had written a single file. It never ran its own code.

Opus 5.5 talks about its next step too. At message 331 of its first build it wrote "Non-admins are refused. Next, a low-share run: does the backfill defer and resume correctly?", and then made the tool call that runs it. Of its 40 text messages, only the final report ends a turn. And 53 of its 137 tool requests ran lint, invoked a function, fired a scheduled job or opened the page in a browser. GPT-6.1 Sol barely talks: three text messages in 47 calls. It writes files from inside shell commands, often running the checks in the same command, and ran 27 test, lint or run commands in that build.

The evidence is the tail, the call counts and the lint failure. Why Haiku stopped is my guess. Anthropic's own docs make it a fair one. Its Haiku 5.5 prompting guide says that "with a long coding-agent system prompt at low effort, it sometimes stops early and hands the task back to the user", and that moving from low to medium "roughly halved early stopping". We ran it at medium, so by Anthropic's own number it still stops early, about half as often as at low. The same guide offers a short piece of system-prompt text for this case. Goose's prompt doesn't carry it. The guide also says that at medium Haiku "sometimes reports a code change as done without running a check". The Opus 5.5 guide says its progress updates sometimes end the turn with text and no tool call, and warns that "an unattended agent loop that treats such a turn as the end of the task stops running there". Goose is that kind of loop.

The opposite failure burns the whole call budget. Grok 4.7's worse build and Hy4 Preview's used up all 300 calls. Grok tried rg 44 times on a machine without it and got "command not found" 42 times. Four of its replies hit the output limit repeating the same line or two, one of them 292,726 characters of "The field values are present, so I'll run the full suite once more." That build ran at our medium pin, one level below xAI's default, and billed 1,452 reasoning tokens over the whole build. I can't tell the pin from the model there. Hy4 read the same starter file 45 times. Its prompt also shrank by more than half 15 times in that build, which may be why, and we haven't traced it.

Moment 2: the hidden issue that leaked anyway (BILL-306)

Our app shows a sprint's scope changes to anyone who opens the sprint, minus the issues that person may not browse. v1 already did this right: it asks Jira, as the person, which issues they can see. Two v2 builds leaked anyway, by different routes. Each took the 0.6 critical.

Qwen3.8 Omni Flash's worse build (0.1659) runs a request as the person only when it's passed who: 'user'. Its admin check passes it:

.request({ path: route`/rest/api/3/mypermissions?permissions=ADMINISTER`, who: 'user', method: 'GET' })

Its visibility lookup, under v1's comment, doesn't:

// Which of these issues can the invoking person browse, and under which key? Read as them: bulkfetch
// leaves out issues the person may not see (§3). One request per 100 ids.
export async function visibleIssues(client, issueIds) {
  ...
    const page = await client.json({
      path: route`/rest/api/3/issue/bulkfetch`,
      method: 'POST',
      body: { issueIdsOrKeys: ids.slice(i, i + 100), fields: ['summary'] },
    });

So all 62 of those lookups ran as the app, which can see everything, and my guess is a refactor that moved identity into a client object and missed one call, with the comment carried along. Our scorer plants this exact bug in a test copy of the app. Omni lost the same three checks the planted copy loses. Its other build asked as the person.

Grok 4.7 asked as the person both times. Its worse build (0.1815) then sent this back to the sprint view:

return { sprint, ...totalsView(totals), hiddenCount: live.length - rows.length, changes: rows, members };

members is the sprint's full issue list, each row with its issueKey. In the recorded response BILL-306 is missing from changes and sitting at members[25]. Grok's better build doesn't return members, and passed.

Opus, Sol and Jev Router kept v1's call as the person and send no member list, and all three passed the leak check with 18 of 18 hidden counts right in the builds I read. Each guards its admin page on the server by asking Jira for the caller's ADMINISTER permission. Opus answers "Only Jira administrators (ADMINISTER) can use this page."

Moment 3: moving the v1 rows

Gemini 4 Argon scored 0.7926 on its first Forge 2.0 build, the second best build on the board, and 0.2943 on its second. Both ran on the same goose build with the same settings and seeds. I traced what changed.

An upgrade has to copy v1's ledger rows into the new store (59 on the site I traced) and keep what v1 recorded about each one. In build 2 the admin panel's progress counter builds the migration list itself:

export async function ensureMigrationCatalog(cfg) {
  let mig = await kvs.get(MIGRATION_KEY);
  if (mig && Array.isArray(mig.v1Keys)) return mig;
  ...
  mig = {
    total: v1Keys.length,
    migrated: 0,
    complete: v1Keys.length === 0,
    v1Keys,
  };
  await kvs.set(MIGRATION_KEY, mig);

On the test site whose trace we kept, the first thing anyone did was open the admin page. The app's config didn't exist yet, so this wrote {"total": 0, "migrated": 0, "complete": true}, and nothing wrote that key again. The migration returned early for the rest of the run, and the hourly reconcile rebuilt the same changes from Jira's history and labelled them "reconcile". So 53 of the 59 rows had the wrong source label at each of the seven checkpoints, 53 times 7, the 371 corrupted rows the scorer reported. The rest of each row was right. The build's own comment, at line 710 of the same file, promises "preserving v1's source".

Build 1 has the same "an empty list means done" rule. But its panel only reads: it returns "Migrated 0 of 0 v1 rows" without writing anything, so the first list came from the scheduled sync, with all 59 rows. Build 2 did test its migration. It wrote a script that compared each row's source with v1's, and its reasoning noted "achieving 88/88" (its dev site had 88 rows). It always ran the sync before opening the admin page, so the scorer's order never happened there. My reading is that it comes down to which of two designs the model happened to write first. It took a second build and the scorer's order to expose the bad one.

The best answer to this moment came from the one model we ran on a laptop. The local Qwen3.8 27B base repairs exactly this case:

// Migration adoption: the v2 app first recorded the change (an event or reconcile that arrived before
// the migration reached it), but the row's kind, issue, author and source must stay as v1 recorded
// them. estimate/estimateField/boardId/deleted/deployedEnvs are kept from the v2 write (v1 had none).
export async function adoptV1Row(key, v1Row, enriched) {

and then overwrites source: v1Row.source on a row that got there first. Opus prevents it instead: its one row writer, used by every path, looks up v1's record before it creates a row ("a change v1 recorded keeps v1's record"). Both kept 59 of 59 rows intact. GLM-5.3 Prime's worse build broke the same field another way. Its hourly job reconciles before it migrates, so for the first two scored hours 82 rows on one of the three sites read "reconcile". Its other build migrates first. It has no critical. The same critical build answered all five sprint actions exactly on all three sites, which neither Sol build did on any.

Moment 4: the webhook that said no and still wrote

Our app takes a signed webhook from CI saying an issue was deployed. An unsigned or tampered call must get a 401 and change nothing, and one event sent twice at once must apply once. Three builds answered 401 correctly and still wrote something on each bad call, and in each of them the write sits in a wrapper around a signature check that works. In the local 27B base's second build, dose is the app's hourly budget of Jira calls, and persist saves it, signed request or not:

export async function ciWebTrigger(event) {
  const dose = await makeDose();
  try {
    const out = await handleCi(event, dose); // handleCi wraps the dose into a dosed Jira policy
    await persist(dose);
    return out;

DeepSeek V4.1 Flash, the September release that sits 14th, saves its counter the same way in a finally block, and GLM-5.3 FlashX writes its counter before it has even read the secret. DeepSeek's 17 October update made no write on a bad call in either build. On 13 October I wrote that the September build's critical was wrong, or at least far too expensive, since it's only a counter write after a correct 401. The scorer kept it, because the rule counts any write, and Opus, Sol and Jev show further down that the write isn't needed.

The same local build also checked whether it had seen an event, applied it, and only then wrote it down. Two copies arriving together both passed the check and both applied. GLM FlashX made the same choice and said so in a comment: "Effect first (idempotent: environments are a set), then the once-only gate."

The local base's other build got that order right and lost the deployments anyway. It claimed each event first, Jira answered its next read with a 429, and the error was swallowed while the webhook still answered "accepted". A retry from CI then counts as a duplicate, and the deployment is gone for good, on two of the three test sites. DeepSeek's build, counter writes and all, does the step missing here. When the effect fails it deletes its claim: "Never leave a half-applied event reserved: a retry must be able to record it."

Opus, Sol and Jev got it all right. They read the secret first and compare the signature in constant time before writing anything. Then they claim the event with a Forge storage write that fails if the key already exists (FAIL_IF_EXISTS), so only one copy gets in. Opus deletes its claim if queuing fails. Sol and Jev store the event as applied: false, a to-do item that a worker flips once the deployment is recorded. Jev's build says why:

// The acceptance record IS the durable outbox; replays can safely repair a lost enqueue, never reapply a deployment.
try { await kvs.set(key, { body, applied: false }, { keyPolicy: 'FAIL_IF_EXISTS' }); }

Sol's build writes the same applied:false record with nearly the same code. Whether Jev's routing beat Sol's own pair, or the newer goose build and luck did, two builds can't say. I wouldn't put a router in front of an agent because of it.

Moment 5: what the money bought

The two Prime models billed $36 to $52 a Forge build, MiMo V2.6 Flash $0.31. I expected the expensive ones to be doing more of something, and they weren't:

build	calls	bill	test or run commands	context per call	tests earned	score
GLM-5.3 Prime, better build	295	$47.76	99	272,000	0.895	0.4557
Qwen3.8 Max Prime, better build	191	$35.98	66	327,000	0.967	0.6515
MiMo V2.6 Flash, better build	188	$0.31	90	308,000	0.881	0.5538
Claude Opus 5.5, better build	133	$20.71	53	177,000	0.963	0.7294
GPT-6.1 Sol, better build	47	$1.15	27	87,000	0.950	0.6495

The two Primes sit in another comparison group. Treat the score gaps here as my reading. The Primes and MiMo, priced so differently, did the same kind of work at the same scale: 190 to 300 calls, about 300,000 tokens of context on each, dozens of test runs. They weren't re-reading files either, at most eight repeat reads per build. Both Primes cached over 99% of the prompt. A cached token cost $0.50 per million on Max Prime, $0.56 on GLM Prime and $0.0028 on MiMo. Cache reads were 86% of Max Prime's bill and 93% to 94% of GLM Prime's, and output was under $3 of either.

Did the money buy score? Not much. Max Prime's better build earned 0.967 of the tests, the most of the five, and still finished below Opus on reliability, while GLM Prime earned less of the tests than Sol, who made 47 calls for $1.15. Near the end Max Prime caught a bug in its own checker ("My check script had a member-collision bug (multiple members per issue). Re-validating properly:"), and MiMo did the same in its second build ("RACE: widget still loading! Need to wait for metric OR needs-config. My check was too eager."), for about a hundredth of the bill.

Divide the average bill per build by the median and each 0.1 of Forge 2.0 score costs $0.06 with MiMo, $0.07 with DeepSeek's 17 October update (in another group), $0.20 with Sol, $3.15 with Opus and $11.58 with GLM-5.3 Prime. That update arrived while I was writing: 0.6957 on Gauntlet, 12th, and Forge 2.0 builds of 0.6618 and 0.2422 for about $0.31 each. I haven't read those builds yet. The chart has the 29 models with a bill for both builds. Gemini 4 Argon ran on Google's own API, and the local 27B has no API bill.

Reading to the end, and where the score gap really comes from

Even Opus missed a paged read. Its first build checks a board's active sprints, to notice when one closes, with one request:

const page = await jira(route`/rest/agile/1.0/board/${bid}/sprint?state=active&maxResults=50`, {}, {

There's no loop. The test sites serve small pages, so the check missed sprints, though the app pages that endpoint properly elsewhere. On its worst site this build walked 50 of 108 multi-page reads to the end. No other build scoring 0.5 or more did worse. Jev Router's best build sends all its board and sprint lists through one helper that stops only when Jira says isLast, and walked 83 of 83. My guess is that two hand-written loops are where a long build drifts, and one shared helper would have stopped it.

Most of the score gap on Forge 2.0 is in a reliability factor, and paging is only one check in it. A build's score is the tests it earned times that factor. The factor drops with the worst test in each group, on the worst of the three sites. The 16 builds that scored 0.5 or more earned a median 94% of the tests and kept a factor of 0.64, while the 24 builds between 0.2 and 0.5 earned 83% and kept 0.44, and five of them also took a critical defect. None of the top 16 did.

What the model cards say, and what they don't

Most of what you'd want to blame a behaviour on isn't in the vendors' cards.

model	architecture, per its own card	our "medium" against the vendor default	price per million, in / out / cached
GPT-6.1 Sol, Sol Pro	not published	the default	$2 / $10 / $0.10
Claude Opus 5.5	not published	the default	$4 / $20 / $0.20
Claude Sonnet 5.5	not published	one below (high)	$2 / $10 / $0.10
Claude Haiku 5.5	not published	the default; Anthropic suggests high for longer agent tasks	$0.10 / $0.50 / $0.01, five times past 100k
Gemini 4 Argon	sparse mixture of experts, no counts	the default	$2 / $10 until 30 November
Grok 4.7	"a new, larger base model"	one below (high)	$2 / $6 / $0.50
MiMo V2.6 Flash	309B, 15B active; 39 of 48 layers see a 128-token window	no effort levels	$0.14 / $0.28 / $0.0028
Qwen3.8 Max Prime	fast tier of Qwen3.8 Max; open sibling 2.4T, 95B active	not published; the open sibling defaults to xhigh	$4 / $12 / $0.50
GLM-5.3 Prime	256 experts plus 1 shared; total not stated	not a level (low, high, max)	$2.80 / $8.80 / $0.56
Qwen3.8 27B, our local base	dense 27B; 48 of 64 layers linear attention	one below (xhigh; there's no high)	local, 8-bit
Jev Router	router, pool not published	picks its own effort	the routed model's

Sources, all read on 18 October: OpenAI's Sol page, Anthropic's effort docs, Google's Argon card, xAI's Grok 4.7 post and reasoning docs, the Hugging Face cards for MiMo, Max's open sibling, GLM-5.3 and Qwen3.8 27B, Alibaba on Prime and OpenRouter on Jev. OpenRouter lists the closed models' quantisation as "unknown".

The Prime bills are a pricing story more than an architecture one. Alibaba's Prime mode is a fast tier, "1.5 to 2 times" the tokens per second of the standard API. It's billed per token like the standard API, at about twice the price in Alibaba's own table, and OpenRouter lists Max Prime at exactly twice Qwen3.8 Max. MiMo's card puts 39 of its 48 layers on a 128-token window, which should make a long cached context cheap to hold. That may be why hosts can sell its cache reads 180 to 200 times cheaper. The card doesn't say so.

GLM-5.3's open card says it runs any effort other than low or high as max, and Alibaba hasn't published how its Prime maps our "medium". Output was under $2 of each GLM Prime bill, so the reasoning tokens weren't what cost money. Whether a higher effort made it take more calls, one setting can't tell us.

Google says Gemini 4 Argon ignores temperature and top_p, so we couldn't have pinned sampling between its two builds. I doubt it would have made two 300-call builds alike anyway. Sol's builds billed about 14,000 reasoning tokens each, Opus's first 595,000, and OpenAI publishes nothing about Sol's output length.

What it says about the models

1. Claude Opus 5.5: careful and slow. A build took over two hours and about $21.
2. GPT-6.1 Sol: terse and fast. It often runs its checks in the same shell command as the edit. Like MiMo it answered few sprint actions exactly, none of five on its worst site in either build.
3. Jev Router: the best build on the board, written like Sol's.
4. MiMo V2.6 Flash: works like the expensive models for $0.31 a build. Once in each build it told a person the app was out of Jira calls, as Sol did in one build and Opus in neither.
5. Qwen3.8 Max Prime: the most tests earned, then lost on reliability.
6. GLM-5.3 Prime: the most expensive builds on the board. One answered five of five sprint actions exactly and broke one site's migration for two hours.
7. Gemini 4 Argon: the second best build and a 0.29, from a progress read that also writes.
8. Claude Haiku 5.5: it planned out loud and stopped, in both builds, and goose ends a run on a turn like that. On the smaller Forge 1.0 it scored 0.9661.
9. Grok 4.7 and Qwen3.8 Omni Flash: each leaked hidden issues in one build and not the other, Omni by asking as the app, Grok by returning an extra list.
10. The local Qwen3.8 27B base: the best migration code I read, and a webhook bug in each build that would stop me shipping it. Its builds took 400 and 521 minutes on a MacBook with an M3 Max and 64 GB.

Our local base is the plain 8-bit Qwen3.8 27B we fine-tuned for Forge in September, published as Mihai-LeanZero/Qwen3.8-27B-Atlassian-Q8-base-mlx. It ran on our own MLX engine through a development build of the goose app. The board shows its pair as one local entry.

Where I was wrong, and where our harness was

On 12 October I wrote that if I were picking a cheap model to build something by itself, I'd pick Haiku 5.5. I even quoted Anthropic's line in that piece. Sonnet 5.5 and Opus 5.5 "remain better choices for complex agentic coding tasks", and Haiku is "best suited to more narrowly scoped tasks". I gave the pick anyway. For Forge 1.0-sized apps I'd still say it. On Forge 2.0 it's 0.0621.

On 8 October I called Qwen3.8 Flash the Forge pick on a budget, with the docs in its context. Forge 2.0 gives the model no docs. Flash billed at least $1.68 a build for a 0.4077 median and MiMo $0.31 for 0.5283, in a different comparison group, so on this board my budget pick is MiMo.

Our own mistakes are a longer list.

Our scorer crashed on a call-heavy app, and the crash hid penalties. On 18 October the Forge probe threw a stack overflow on one test site for the local 27B's second build. 39 checks went unavailable, and the build showed an unpublishable 0.2736. After the fix the re-score is 0.1128, because the blanked checks were the webhook ones, and they held two criticals.

Our secret probe is still wrong. On a build from the earlier local pair it took the admin's account ID for the CI secret and charged a critical. Neither build on the board now tripped it, and it stays wrong until a scorer release fixes it.

The board kept moving under this article. When the first local pair, 0.1457 and 0.1342, was retracted on 18 October, our research sheet carried its webhook line ("valid: status 401") over to the new pair. So I first had the 0.2576 build down as rejecting validly signed webhooks. Its code had accepted them and lost them to the swallowed 429. Then DeepSeek's update landed on Forge 2.0 while I was writing, and on Gauntlet half an hour later. Six more models arrived before it went out, and Pareto 26.10 pushed MiMo from 7th to 8th on Gauntlet.

"Medium" isn't one setting. We pin each Forge 2.0 build at medium effort to keep the setup fixed. For Sol, Opus, Haiku and both Geminis that's the vendor default. Anthropic's effort docs say to start Haiku at medium for agentic coding and to use high for longer agent tasks, and ours is both. Sonnet and Grok ran a level below their defaults, and Qwen3.8 27B one below on a scale that skips high. GLM-5.3 and the Kimi K3 under Ember 1 don't have a medium, and MiMo has no effort parameter on OpenRouter at all. The Qwen card warns that lower effort "can also lead to insufficient analysis, more failures, and repeated retries".

And goose ends a headless build on a turn with no tool call, while Anthropic documents that its models sometimes end a turn with a progress update. Haiku's builds died at that point, and so did one build of the cloud Qwen3.8 27B, after 18 calls ("Let me look at the ModuleSchema definition"). A Stop hook, a nudge or Anthropic's own prompt text might have changed Haiku's score.

Liquid's LFM2.5 24B-A2B scored 0.0187 on Gauntlet in one run on our own engine, after a plain-text tool call our new parser didn't convert, so that's no verdict on LFM yet.

Opus and Sol for Forge, my 5 October call, held up. MiMo before DeepSeek on a budget, from 13 October, is open again: the update costs the same $0.31 a build, and in another group its median sits 0.08 below MiMo's. I haven't read its builds.

What to check before you trust a model with an unattended build

1. Read the last message of a stopped run. If it describes the next step without taking it, your harness decides whether the work continues.
2. Find each read that decides who sees what, and check which identity it runs as, whatever the comment above it says.
3. Open the admin page, or whatever reads state, before the first background job has run. One build on our board froze its migration that way.
4. Send an unsigned request to each webhook and diff your storage afterwards.
5. Look for list reads without a cursor loop.
6. Run two builds. Gemini 4 Argon's pair was 0.50 apart on the same setup, and DeepSeek's update 0.42.
7. Price the cached token as well as the headline one. Builds of about 190 calls cost $0.31 on MiMo and $36 on Max Prime.

What we test next

From Monday 19 October we run a weekly new-model review through both benchmarks, locally when a model fits on our Macs and through OpenRouter when it doesn't. I most want a Forge 2.0 pair for GPT-6.1 Sol Pro and the fine-tuned Atlassian 27B against its own base. After that, a Jev Router rematch beside a fresh Sol pair on one goose build.

Each run page has the video and screenshots of its build, so check what was built before trusting a score here. You may read the same code and blame our harness instead, and on Haiku you might be right. Our goose fork isn't production ready (we use it because we know what's in it), so every run page is public and you can check us. Do you leave long builds running with nobody watching, in goose, Claude Code or your own agent? Has one ever told you it would do the rest next and then stopped? Tell us on our Discord or at [e-mail], and tell us too if a number here is wrong.

---

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.

Comments

[Pierre-Laurent Medori]

So the gap wasn't intelligence. The top models just did the boring checks: who's asking? Is this really the last page? Did I actually do the "next" I said I would? We've seen the same thing with the agents calling our MCP server in production.

That's also why switching models runs a test suite you never wrote. When we moved our app builder to a new model, it started sending back patches where the prompt asked for whole files. Our validator only checked the format, so it let them through.

One question on Forge 2.0: did the permission defects show up more on the hourly reconcile path than on the event-driven one? A background job has nobody in front of it, and that's exactly where I'd expect "who's asking" to get lost.

[Mihai Perdum]

Not in the way you'd expect, and I can't split it cleanly by path. The leak check only reads what a person is shown (the sprint view, the sprint action's responses, Rovo and LLM output), so it never asks which writer put a row there. Of the 64 Forge 2.0 builds in our store, two took that critical, Qwen3.8 Omni Flash's worse build and Grok 4.7's worse one, and both leaked through the sprint view someone opens. Omni ran the visibility lookup as the app instead of as the person, and Grok sent back the full member list next to the filtered changes. Neither involved the reconcile job.

The background side is graded separately. A check called r_as_app looks at whether the hourly job's Jira calls run as the app, since nobody's there to ask as. 62 of the 64 builds had every scheduled call as the app, and the other two couldn't be measured because the probe recorded no call lineage. So your instinct points at the right place to look, but the two defects the scorer found were read-time mistakes with a person in front of them.