I ran DeepSeek V4.1 Flash through our Forge 2.0 coding benchmark twice and it scored 0.2893 and 0.3790. Its median is 0.3342. Claude Opus 5.5 got 0.6745 on the same task, and MiMo V2.6 Flash, which cost about a quarter as much per build, got 0.5283.
DeepSeek's own table has this model at 74.2 on DeepSWE v1.1 against 74.0 for Claude Opus 5 (The Next Web reported it). Several of the benchmark pages I found for DeepSeek V4.1 Flash repeat that table. This is an independent run, on one hard task, and it says something different.
It also says something about us. The lower of the two scores carries a 40% penalty for a security defect, and when I opened the app's code to write it up, what the app had done was save its own quota counter. I think 0.2893 is too low. I give both numbers below and you can pick.
And the day before these runs, the same model had 0.7979 on our board, a hair under Opus. That one was my mistake too, in the other direction, and it's the last section.
DeepSeek V4.1 Flash benchmark scores, with the six other models on Forge 2.0
Forge 2.0 hands the model a working Atlassian Forge app (version 1 of a sprint scope ledger for Jira), installs it on a simulated Jira site with about 1,000 issues, and asks for version 2. There's no internet, nothing to install and no reference solution to find. It gets 300 model requests. Then the harness upgrades the site to whatever the model wrote and grades it by running it. No model takes part in the grading.
This is the board as I read it on Tuesday 13 October 2026, early afternoon EEST. The live Forge board is the current word.
| model | median | build 1 | build 2 | cohort | cost, two builds |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.6745 | 0.7294 | 0.6196 | 1 | $42.50 |
| GPT-6.1 Sol | 0.5933 | 0.6495 | 0.5371 | 1 | $2.38 |
| Claude Sonnet 5.5 | 0.5692 | 0.5485 | 0.5899 | 2 | $11.26 |
| MiMo V2.6 Flash | 0.5283 | 0.5538 | 0.5028 | 1 | $0.62 |
| Qwen3.8 Omni Flash | 0.3381 | 0.5104 | 0.1659 | 2 | $6.14 |
| DeepSeek V4.1 Flash | 0.3342 | 0.2893 | 0.3790 | 1 | $2.39 |
| GPT-6 Luna | 0.2389 | 0.2197 | 0.2580 | 2 | $1.50 |
Each model built the app twice from scratch, and the board ranks by the median. The cohort column is what the board calls a comparison group. Both cohorts ran the same task, the same scorer and the same three sites. They differ in one recorded thing, the engine file: cohort 1 ran on the pre-release copy of desktop app 3.0.110 that the task was repaired on, cohort 2 on the released 3.0.110. The board says to compare inside a cohort. DeepSeek, MiMo, Opus and Sol are all in cohort 1.
Qwen3.8 Omni Flash went up on the board this morning. Its median is 0.004 above DeepSeek's, it's in the other cohort, and its two builds are 0.3445 apart. Its build 2 carries a 0.6 critical multiplier too, in a different defect class, and I haven't read that build's code. I don't read any of that as Qwen being ahead of DeepSeek.
What I found, in four lines
- On tests earned DeepSeek is 8 points behind Opus (0.8658 and 0.8686 against 0.9625 and 0.9408). The scoring rule turns that into a 34-point gap.
- Build 1's 0.6 multiplier came from one storage write per rejected request, and that write was the app's quota counter. No unsigned request changed any ledger data.
- Seven of the twenty test tiers were perfect in both builds, including the admin panel, the Forge LLM hardening and the boot budget.
- DeepSeek billed the same as GPT-6.1 Sol ($2.39 and $2.38 for two builds) because it needed about five times as many requests.
A group of tests with a zero in it costs 10% of everything
A Forge 2.0 score is three numbers multiplied: tests earned, critical defects, reliability.
Tests earned is the ordinary weighted sum. The 105 checks are sorted into 20 weighted tiers for that sum, and into 19 groups for reliability. Critical defects is 1.0 unless the scorer observes one of five named defect classes (a duplicate side effect, a write without a valid signature, and so on), and each class observed multiplies by 0.6. Reliability is where most models lose their score: each of the 19 groups multiplies the whole score by 1 minus 0.10 times the shortfall of its worst test. So a group whose worst test scores zero costs a tenth of everything, however many other tests in that group passed.
DeepSeek's build 1 page shows it in one line: 0.8658 tests earned × 0.6000 critical defects × 0.5568 reliability = 0.2893.
DeepSeek V4.1 Flash vs Claude Opus 5.5: 0.33 against 0.67
The critical defect in build 1 was a quota counter
Requirement R6 is a web trigger that a CI system calls to mark issues as deployed. Atlassian's docs say web triggers "have no built-in authentication. As such, anyone can use the URL", so the task contract asks for an HMAC-SHA256 signature compared with crypto.timingSafeEqual, then a timestamp within 300 seconds of now. For a request with no signature, a wrong one, a changed body or a stale timestamp, the contract's table says: answer 401, effect none.
The scorer sends five invalid requests (bad signature, unsigned, tampered, and two with a stale timestamp). Its line for DeepSeek's first build reads: "side effects without a valid signature: bad-signature (1), unsigned (1), tampered (1) (+2)". That's a critical defect, and the score was multiplied by 0.6.
So I went to read how an unsigned request got through. It didn't. DeepSeek's handler checks the signature first, with timingSafeEqual, then the timestamp, and returns before writing anything. All five invalid requests got a 401 (the status-code test in that group scored 10 of 10), and the harness recorded zero ledger writes for each of them. The one write it did record per request comes from the wrapper around the handler:
export async function ciWebTrigger(request) {
const budget = new Budget({ limit: Number.MAX_SAFE_INTEGER });
await budget.load();
try {
return await ciTrigger(request, { ...CI_POLICY, budget });
} catch (e) {
return { outputKey: 'unauthorized' };
} finally {
await budget.save();
}
}
budget.save() stores the app's own count of Jira rate-limit points spent this hour, under a key like quota:<hour>. It runs on every call, rejected or accepted. (I've trimmed a comment and one logging line from the snippet.)
Is that a write without a valid signature? Literally, yes, and the contract does say none. But our scorer isn't consistent about it. The neighbouring test, for a replayed event, also has "none" in the contract, and it passed this same app with the note "bookkeeping writes 1". One test forgives the counter and the other calls it a critical security defect. There's an argument for the difference: a replay carries a valid signature and these five don't. And the save writes back the count it read a moment earlier, so a request that arrives while a background job is updating the counter can undo that update. I still think the critical is wrong, or at least far too expensive for what happened. Anyone with the URL can make this app save a counter. Nobody without the secret can make it change a ledger row.
The published score stays 0.2893, because the task and scorer are frozen and I'm not going to edit one result by hand. Without the multiplier, build 1 is 0.8658 × 0.5568 = 0.4821, and DeepSeek's median would be 0.4306. That's 0.64 of Opus where the board says half. It's still below MiMo's 0.5283. It would put DeepSeek above Qwen3.8 Omni Flash (0.3381) and back in fifth place, and it still would if I dropped Qwen's 0.6 as well (0.3934). Build 2 made no write at all on the five invalid requests and passed the test.
It reacted correctly to 6 of 56 rate-limit responses
This one is real. In build 1 the simulated site answered 56 requests with a 429 and a reason header, and DeepSeek's app reacted correctly to 6 of them. It also had 4 of 972 person-facing requests refused for quota.
Atlassian set enforcement of Jira's points-based limits to begin on 2 March 2026, and on Tier 1 an app "shares a single 65,000 point hourly quota across all tenants". An app that ignores the reason on a 429 burns that quota for all of its customers. Build 1's dosing tier scored 0.504, its lowest. Build 2 scored 0.75 there.
Build 2 failed six groups outright
DeepSeek's second app had no critical defect and still came out at 0.3790, because its reliability was 0.4363. Six groups had a worst test at zero:
- Re-estimated sprints: 0 of 6 showed the right numbers after their estimates changed.
- Deleted issues: 0 of 3 ledger rows were kept as history. The app erased them.
- The native JQL function: 0 of 24 initial-membership cases correct.
- Future sprints: the widget didn't show "not started" for a sprint that hasn't started.
- The Rovo action: 0 of 5 active sprints answered with exact numbers.
- Quota: 6 of 975 person-facing requests were refused.
Multiply 0.90 six times and you get 0.53, before the partial losses. Its dashboard charts were off too: the worst bar sat 74.80 pixels from where a linear scale puts it, and 2 of 8 widget views had exact numbers. Build 1 had three groups with a zero (the live widget update, the Rovo action, the quota refusals).
Opus has zeros too. Both of its builds lost two ledger rows when a sprint closed, and its first build answered 3 of 5 active sprints exactly through the Rovo action. But Opus didn't have a single person-facing request refused for quota, and its dashboard widget updated live. Opus lost one or two groups per build. DeepSeek lost three and six.
Seven tiers were perfect in both builds
Most of what DeepSeek wrote worked. In both builds, with nothing lost on any of the three sites, it scored 1.00 on lint and bundles, storage, visual checks, the virtual-time invocation limits (R3), the UI Kit admin panel (R5), Forge LLM hardening (R8) and the Custom UI boot budget (R9).
It also used its budget. DeepSeek made 261 and 245 of its 300 model requests and worked for 99 and 38 minutes. Opus made 133 and 115.
DeepSeek V4.1 Flash vs MiMo V2.6 Flash: the cheaper model scored higher
I expected DeepSeek to win this one. It lost both builds. MiMo V2.6 Flash lists on OpenRouter at $0.14 in and $0.28 out per million tokens. DeepSeek's own endpoint for DeepSeek V4.1 Flash lists $0.30 and $1.20 in its peak hours and half that off-peak (OpenRouter's API, 13 October). MiMo's two builds scored 0.5538 and 0.5028, DeepSeek's 0.2893 and 0.3790.
| DeepSeek V4.1 Flash | MiMo V2.6 Flash | |
|---|---|---|
| median score | 0.3342 | 0.5283 |
| tests earned, b1 / b2 | 0.8658 / 0.8686 | 0.8807 / 0.9278 |
| reliability, b1 / b2 | 0.5568 / 0.4363 | 0.6288 / 0.5420 |
| critical defects | one (build 1, disputed above) | none |
| groups with a zero, b1 / b2 | 3 / 6 | 3 / 2 |
| model requests, b1 / b2 | 261 / 245 | 188 / 195 |
| cost, b1 / b2 | $1.63 / $0.76 | $0.31 / $0.31 |
On tests earned they're close in build 1. MiMo is ahead because it lost less to the multipliers and never tripped a critical. Take my disputed critical away and build 1 of DeepSeek (0.4821) is still under both MiMo builds. MiMo's weak spot is the same as DeepSeek's: in build 1 it answered 6 of 61 rate-limit responses by their reason.
It isn't only this task. On Gauntlet 7.2, our other benchmark (a payments app, one run per model), DeepSeek V4.1 Flash scored 0.591, 18th of 34. MiMo V2.6 Flash scored 0.7909, 0.0017 behind Opus 5.5.
One more thing about MiMo's number. Vals AI reported that in two-thirds of the coding environments Xiaomi released for MiMo v2.6, the answer can still be read in the task's Git history, and the model finds it. Forge 2.0 ships no reference v2 and no network to look for one.
DeepSeek V4.1 Flash price and cost per task: $1.63 and $0.76
| model | list price in / out, per M tokens | build 1 | build 2 | minutes, b1 / b2 |
|---|---|---|---|---|
| Claude Opus 5.5 | $4.00 / $20.00 | $20.71 | $21.79 | 134 / 125 |
| Claude Sonnet 5.5 | $2.00 / $10.00 | $6.26 | $5.00 | 74 / 59 |
| DeepSeek V4.1 Flash | $0.30 / $1.20 | $1.63 | $0.76 | 99 / 38 |
| GPT-6.1 Sol | $2.00 / $10.00 | $1.15 | $1.23 | 20 / 26 |
| GPT-6 Luna | $0.10 / $0.50 | $0.88 | $0.62 | 42 / 40 |
| MiMo V2.6 Flash | $0.14 / $0.28 | $0.31 | $0.31 | 31 / 30 |
| Qwen3.8 Omni Flash | $0.15 / $0.47 | $1.52 | $4.61 | 91 / 147 |
Costs are from the campaign's billing ledger, to the cent. Fourteen builds came to $66.79, and Opus is $42.50 of it. DeepSeek's list price here is its own endpoint's peak-hour rate. OpenRouter sent our two DeepSeek builds to other hosts, mostly InferenceNet, which lists $0.077 in and $0.55 out.
DeepSeek V4.1 Flash vs GPT-6.1 Sol: the same bill, 0.26 apart
Sol's output tokens list at about eight times DeepSeek's peak rate and about eighteen times what InferenceNet lists, and the two models billed the same. Sol finished in 47 and 48 requests. DeepSeek needed 261 and 245, and an agent resends the conversation on each one, so build 1 pushed 101 million prompt tokens through the API. 96.3% of those were cache reads, which worked out at between one and two cents per million, or the bill would have been a lot more than two dollars.
So per token DeepSeek V4.1 Flash is cheap, and per finished app it cost what GPT-6.1 Sol cost, for a median 0.26 lower (0.16 lower if you drop the critical). Why did build 1 cost twice build 2? It ran 99 minutes against 38 and sent 3.76 million uncached prompt tokens against 0.32 million. Why the cache missed more, I don't know. I haven't traced it.
Which one I'd use
Two builds don't separate Opus 5.5 from GPT-6.1 Sol on this board. Their medians are 0.6745 and 0.5933 and their ranges overlap. Sol cost about $1.19 a build and Opus about $21. For an unattended agent building a Forge app I'd start with Sol, and read the diff whichever one I used.
If budget is the constraint, MiMo V2.6 Flash before DeepSeek V4.1 Flash. It scored higher on both of our benchmarks, and on Forge 2.0 it cost about a quarter as much.
I'd use DeepSeek V4.1 Flash for a first draft I'm going to review, and I'd look at two things before the rest: what the code does with a 429, and what it does when data changes after the fact (a re-estimate, a deleted issue). Those are where its zeros were. Its signature check, for what it's worth, was fine both times.
What could be wrong with these numbers
Two builds per model is a small sample. DeepSeek's two differ by 0.0897 and Opus's by 0.1098, so I'd treat any gap under about 0.1 as unresolved. Qwen3.8 Omni Flash's two differ by 0.3445, the widest pair on the board and as wide as the gap between DeepSeek and Opus. So 0.1 is the least I'd call unresolved, and one model's two builds can land much further apart. DeepSeek against Opus is 0.34 apart with no overlap (0.24 without the critical), and DeepSeek's best build is under MiMo's worst. I'm comfortable with those two orderings and not much else. A study of 60,000 agent trajectories found single-run estimates on SWE-Bench-Verified moving by 2.2 to 6.0 points depending on which run you pick. Ours move more, because our rule multiplies.
Each build is graded on three differently seeded Jira sites, and each test keeps its worst site. A model is told the scoring site differs from the one it develops on. It isn't told there are three, or that the worst one counts.
Reasoning settings weren't identical. Opus, Sol, Sonnet and Luna were asked for medium effort. DeepSeek, MiMo and Qwen3.8 Omni Flash were run with reasoning enabled at the provider's default, because that's what their OpenRouter routes accepted. Each model's literal request is on the board. A different effort level could move DeepSeek, and I haven't tested by how much.
The disputed critical isn't our scorer's only problem. Nine of the fourteen builds scored zero on "person-facing requests are never refused for quota", sometimes for a single request out of 992, and the board now carries a note that the frozen contract overstates what the workload guarantees. DeepSeek lost that group in both builds, and so did seven other builds, though neither of Opus's. I don't know yet how many of those nine zeros are the models' fault and how many are ours.
Everything ran inside our own agent, Goose Swarm (a fork of goose). I don't know how DeepSeek V4.1 Flash would do on this task in another harness.
And it's one task, a Forge one. I don't know how much of the gap is DeepSeek knowing less Forge than Opus does.
The 0.7979 I took down
Forge 2.0 first published on the evening of Saturday 10 October with desktop app 3.0.109. Six models ran overnight, one build each. By 06:46 on Sunday the board read MiMo 0.8461, Sol 0.8111, Opus 0.8100, DeepSeek 0.7979, Sonnet 0.7698, Luna 0.6250.
I paused the queue at 08:09. At 08:48 the six runs were marked invalidated, 10.5 hours after the first one had published.
An audit of our own scorer, run that morning, found four kinds of problem.
Some checks passed when the thing they tested never happened. If an app's admin actions all failed, the check that a non-admin can't write was marked not applicable and cost no points. If valid CI events had no effect, the unsigned-write check was skipped the same way.
Our migration check compared identity and time and didn't look at the migrated values. Another check, on the LLM cache, could pass when both clicks made zero calls and the app only displayed an error. And the workload meant to stress the hourly quota used 228 points of a 1,680-point background share, so the dosing requirement was barely exercised. Three whole requirement families gave full marks to all six models.
On top of that, each model had been graded once, on a different random set of sites from the others. So the ranking mixed model quality with site luck.
Repairing it took the rest of Sunday. The repaired task has a tenth requirement family (the native JQL function), a quota workload that uses about 1,595 of the 1,680 points, 105 checks where there had been 99, and three common sites for everybody. No money was spent on model runs until it was frozen. Then every model ran twice, and desktop app 3.0.110 made two independent builds the default for a Forge run.
I didn't re-score the old numbers. They stay up, marked, with their recordings.
I asked for Forge 2.0 on the evening of 9 October, because the top five of Forge 1.0 sat between 0.945 and 0.977 and couldn't be told apart. The first result published about 27 hours later. That was too fast. The six withdrawn runs cost $23.71, and for half a day the board showed a wrong ranking.
And now the repaired scorer has a penalty I'd argue with, found by reading one app's code for this article. So open the run page before you trust a number from us. Both DeepSeek builds have one, with every test line and a recording: build 1 and build 2.
The method behind Gauntlet and Forge 1.0 is in Claude Sonnet 5.5 vs Opus 5.5, the small-model pairing is in Claude Haiku 5.5 vs GPT-6 Luna, and the Forge 1.0 results for the Qwen family are in Qwen3.8 Flash vs Max Prime.
Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.
Top comments (0)