Mistral Large 4 coding benchmark: the money bug was ours, and DeepSeek V4.1 Flash still won on Forge

Key takeaways

• Mistral Large 4 scored 0.5462 on our Gauntlet 7.2 after a critical check for wrong money multiplied its score by 0.6. Its money was right: our check compared the 12 amounts on screen with the oldest 50 payments, and Mistral's table opens on the newest. Joined by row id, 24 of 24 cells pass. By my arithmetic it would have scored about 0.70, 11th of 44 instead of 28th on the 21 October board.
• The same audit found a second flaw in that check: when the visible rows hold no yen or no dinar amount, the yen or dinar test scores 0 and the run's score is multiplied by 0.92. Four more ranked runs paid it with all visible amounts right. Neither fix is shipped yet.
• On Forge 2.0 its two builds scored 0.1086 and 0.3304. Build 1 throttled itself. It booked 51 rate-limit points for each list call that really cost about 2, so it believed it had spent 1,567 of its 1,680-point hourly share when the site had charged 715. It paused and never ran the v1 migration. Neither build's admin page ever displayed.
• Mistral's first app code arrived 27 to 55 minutes into each of its last five runs, against 2 to 5 minutes for DeepSeek V4.1 Flash 1017. Its two Forge builds billed at least $10.95 at launch-sale prices for a median score of 0.2195, against at least $0.62 for DeepSeek's median of 0.452.
• Our own setup hurt it in three places: our harness gave it a 262,144-token window on a 1M-context model (DeepSeek got 655,360 and never compacted), the OpenRouter host pin we set for Mistral sent the harness's summary calls to Mistral instead of a small Gemini model, and one of its four Gauntlet attempts ended because the harness put an image-only user message between two tool results, which Mistral refuses with a 400.

Mistral Large 4 scored 0.5462 on our Gauntlet benchmark, 28th of 44. About a third of what it lost came from a check of ours that read its table the wrong way round. All 12 amounts it put on screen were correct. I went in expecting to write about a slow, expensive model, and on our Forge benchmark that part holds up.

Update, 6 November 2026: Mistral has since released the open weights (29 October), moved the model from Public Preview to general availability and set its list price at $0.70 per million input tokens and $2.10 output. The runs and bills below are from 9 to 19 October, and the ranks are from our boards on 21 October. The November section near the end says what changed and how it compares with DeepSeek V4.1 Flash today.

If you pay for a coding agent, this model is pitched at your budget. Mistral released Large 4 on 6 October as a Public Preview, 1.05 trillion parameters with 52 billion active and a 1M-token context, at half its list price for the first two weeks. Their launch post puts it ahead of DeepSeek V4 Pro on their own Coding Agent Index (49.8%). That kind of claim moves a budget, so we ran it through both of our agentic coding benchmarks. The Gauntlet has an agent build a payments console with two services and a 3D view inside 150 model calls, then scores it on 99 checks. Forge 2.0 asks for a v2 upgrade of a working Jira Forge app inside 300 calls and scores it on 105 checks against three seeded Jira sites.

Getting a Gauntlet score took four attempts. The first, on 9 October, ended after 49 calls on OpenRouter's "Upstream idle timeout exceeded". On 18 October the second was ended by Mistral's servers. Early on 19 October the third was ended by a bug in our agent harness (our own fork of an open-source coding agent), and the fourth finished that afternoon. Its Forge 2.0 pair scored 0.1086 and 0.3304 for at least $10.95. DeepSeek V4.1 Flash 1017 did better on the same Forge task. It's the 17 October snapshot I wrote about in my DeepSeek piece, and it scored 0.2422 and 0.6618 for at least $0.62 in about a quarter of the time.

On the Gauntlet the low score is mostly our scorer's fault. On Forge the code losses were Mistral's own. Our harness also gave it a smaller context window than DeepSeek. The build trees show where each loss came from. This is one scored Gauntlet run and one Forge pair, in our setup, on a model whose vendor says it can change silently while it's in preview.

Mistral Large 4 benchmark results: Gauntlet 0.5462, Forge 2.0 median 0.2195

Our chart from the run pages, verdict files and session logs. Both Forge 2.0 pairs sit in the same comparison group on our board. The Gauntlet has no comparison groups: these two runs were on different Macs, most likely one harness release apart, with context windows of 262,144 and 655,360 tokens, and both asked for medium reasoning effort.

	Mistral Large 4	DeepSeek V4.1 Flash 1017
Gauntlet 7.2	0.5462, 144 min, $4.93	0.6957, 23 min, $0.21
Forge 2.0 build 1	0.1086, 118 min, at least $5.40	0.2422, 28 min, $0.30
Forge 2.0 build 2	0.3304, 106 min, $5.55	0.6618, 30 min, at least $0.32
Unscored Gauntlet attempts	9 Oct: idle timeout after 49 calls, at least $1.43. 18 Oct: Mistral 503 after 30 calls, at least $1.18. 19 Oct: our 400 after 97 calls, $3.04	none

I checked these on the live boards on 21 October; if a run page says something else now, trust the run page. All six Mistral runs together cost at least $21.53, with three of the bills only lower bounds.

Six things we found in the Mistral Large 4 builds

1. The x0.6 "wrong money" critical on the Gauntlet is our bug. Our money check compares the amounts on screen with the oldest 50 payments, and Mistral's table opens on the newest.
2. Mistral's money formatter works on integer digits with a per-currency decimals table, the same approach as DeepSeek's.
3. Forge build 1 scored below build 2, and by my reading most of that gap came from a throttle it wrote itself: it booked 25 to 34 times the real cost of a list call and paused its own scheduled work each hour, before its v1 migration could run.
4. Neither Forge build ever displayed its admin page, and neither ran a browser self-test.
5. The 18 October attempt opened with one 37,853-word thought that ran 18 minutes, hit a 65,536-token output limit and was thrown away.
6. Mistral's API refused a message order that three other providers' endpoints had answered, through OpenRouter, in our runs two weeks earlier.

Mistral Large 4 on our Gauntlet: 0.5462, about 0.70 without a check of ours that misread its table

The run page as published, where a single red tag carries the whole multiplier.

Most Gauntlet checks just add to a weighted average, but a handful are critical, and when one of those fails it multiplies the whole score down, by as much as x0.6. Wrong money in a payments console is one of those. The money check is called b_money_rendered. A browser probe reads the amount cell of the first 12 rows of the payments table. Then the scorer checks that the digits match a stored amount and that the decimals match the currency, because amounts are stored as integers in the currency's smallest unit (cents, or fils for the Kuwaiti dinar). A correct cell shows two decimals for EUR and USD, none for yen and three for dinars. Mistral's run scored 0.00 on it, with the detail line "0.00 of 12 cells exponent-correct; JPY/KWD traps 0.00", and the multiplier took the whole run from 0.9104 down to 0.5462.

What Mistral rendered

These are the 12 cells the probe read, verbatim: KWD 111.000, ¥594, ¥199, $4.47, ¥809, €224.00, KWD 82.400, €33.10, KWD 2.420, ¥1,060, $1,840.00, KWD 72.400. Our probe also records each row's id. Looked up in the fixture (the stored payment data for this run), pay_12287 is stored as KWD 111000, so 111.000 dinars is right, and pay_12286 is JPY 594, which needs no decimals. All 12 are right. The probe's 12 extra rows are right too.

Its formatter in web/app.js (lines 7 to 25, with the source's \u escapes shown as characters) never touches a float:

const CUR = {
  EUR: { sym: '€', exp: 2 },
  USD: { sym: '$', exp: 2 },
  JPY: { sym: '¥', exp: 0 },
  KWD: { sym: 'KWD ', exp: 3 },
};
...
const s = String(a).padStart(exp + 1, '0');
const int = s.slice(0, s.length - exp) || '0';
const frac = exp > 0 ? s.slice(s.length - exp) : '';

It planned this before writing it. At 10:58 our time (EEST), 27 minutes in, it checked the contract's example and two real payments in its thinking: "129900 EUR → €1,299.00 ✓; JPY 8564 → ¥8,564 ✓; KWD 8263000 → KWD 8,263.000 ✓". Our scorer's own screenshot of the inspector shows that dinar payment as KWD 8,263.000, and the inspector check (q_payment_context) passed.

Taken by our scorer's browser during the run. Three decimals for dinars, none for yen, as the contract asks.

Why the check said zero

Here is the scorer, score_sb7.py around line 973:

first_page = c.pack.payments[:50]
by_cur: Dict[str, set] = {}
for row in first_page:
    by_cur.setdefault(row["currency"], set()).add(row["amount_minor"])

payments is stored in date order, so payments[:50] is pay_00000 to pay_00049, the oldest fifty. A rendered cell only passes if its digits equal one of those fifty amounts, so the check assumes, without saying it, that the table opens oldest first. Mistral's opens newest first (sort: '-created_at', web/app.js:49). None of its 12 cells could ever match. Our probe even collects the row ids "so the scorer can join against fixtures", in its own comment. No scorer file reads them.

Plotted from the probe's own observations and the fixture file of this run.

I re-graded the run with the scorer's own _money_ok function, copied unchanged. It gives 0.0 as written, 1.0 against the 50 rows actually on screen, and 24 of 24 when each cell is joined to its payment by row id.

Was opening newest first a mistake by Mistral? Its contract fixes the API's default order ("created_at (default, ascending by INSTANT)") and says the Date header toggles, but it never says which way the table starts. Mistral saw the gap and decided, at 11:48 our time: "The grader might check the first page shows specific rows? Unlikely to assume an order. I'll default to '-created_at' (newest first) — most console-like." It bet that the grader would not assume an order, and the grader did. You can argue the UI should inherit the API's default. I'd listen. Even then it's a sort-order quibble. The check charged it as wrong money.

Who did better at money

All five models I compared passed with 1.0: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol (the model behind Jev Router's top Forge score), GPT-6.1 Sol Pro and DeepSeek V4.1 Flash 1017. All of them open their table with sort: 'created_at', and so does our own reference app, which is why our release validation never met a descending default. Grade those five tables against the newest 50 payments and four score 0.0, Sol 0.05.

DeepSeek's formatter is close to a copy of Mistral's:

let digits = String(Math.abs(minor));
if (exponent > 0) digits = digits.padStart(exponent + 1, '0');
const whole = exponent > 0 ? digits.slice(0, digits.length - exponent) : digits;

Same idea, same output. It got 1.0 and Mistral got 0.0. Sol and Sol Pro call Intl.NumberFormat and divide the integer by 10**exp first, and Sonnet does the same with Math.pow, which is fine for amounts this size. So on money code it's a tie between Mistral and DeepSeek. The sort order made the difference.

Mistral did leave one real money bug, in its own test. Its browser self-test checks the first amount against /^[€$¥]\d[\d,]*\.\d{2}$|^KWD [\d,]+\.\d{3}$/, which would reject a correct ¥594 because it demands two decimals after a yen sign. It printed "PASS money formatted" four times, and my guess is that its own first row was never a yen payment. I can't see which row it was.

Without our false critical: about 0.70 and 11th of 44

The score path for this run goes like this. Its checks earned 0.9104. Our false critical multiplied that by 0.6, to 0.5462. Separately, a run that fails some of the 3D checks gets a ceiling on its score. Mistral's partial failures on picking (selecting an object in the 3D view), the coasting camera and the overview's legibility set that ceiling at 0.709. The ceiling formula keeps a score just under the cap, so without the false critical the run would come out at about 0.7048. That's my arithmetic, not a re-score. The board still says 0.5462. It would sit 11th of 44, two places above DeepSeek 1017's 0.6957. The 3D failures, which are Mistral's own, cost it about 0.2 on top.

My audit of that check turned up a second flaw. The yen and dinar "traps" score 0 when no yen or no dinar row happens to be among the cells the probe reads (up to 12). A run whose visible amounts are all right can still pay x0.92. Four ranked runs paid it that way: Elorian Glimpse, Step 5 Preview, Qwen3.8 Omni Flash and GLM-5.3. Across every Gauntlet 7.2 build I could reach, Mistral is the only one the sort assumption hit.

Illustration, generated locally with ERNIE-Image. Our check graded the oldest fifty payments while Mistral's table showed the newest. The red pen was very confident.

Is Mistral Large 4 good for coding? Forge 2.0 builds of 0.1086 and 0.3304

Forge 2.0 tests the same habits harder: the agent reads a long spec, upgrades a working Forge app under a model of Jira's rate limits, and has to prove it all in a local sandbox (our Forge emulator kit) before the calls run out. My piece on the September DeepSeek V4.1 Flash covered this same task, with medians of 0.33 for it and 0.67 for Opus 5.5 in an earlier comparison group. Both Mistral builds and DeepSeek 1017's pair are in the same comparison group on our board. That means the same task, seeds, scorer and engine build. All four asked for the same effort setting.

The board's build overview. Both builds share the same lowest tier.

A Forge 2.0 score is the earned score times a critical multiplier times a reliability factor. Reliability starts at 1 and loses up to 10% for each group of checks whose worst check failed, and it never goes below 0.25. Build 1's reliability sat on that floor (its unfloored product was 0.2194) and build 2's came out at 0.4718. Neither triggered a critical. My reading is that most of the 0.22 gap between the two builds' scores comes from build 1's throttle and the order it ran its migration in.

Build 1 throttled itself and never migrated

The task gives each Jira site a rate budget of 2,400 points an hour, and background work may use 70% of it, so 1,680. Build 1 wrote a gate for that in src/dose.js (lines 76 to 88):

if (dose.bg + estimatedCost > limit) {
  const untilMs = hourStartOf(now) + 3600000;
  await kvs.set(PAUSE_KEY, { untilMs });
  throw new QuotaPaused(untilMs);

The gate is fine. The estimates feeding it were not. The site sent no header saying how many points a call cost, so it added guessed costs to its own counter. It booked 51 points for a board or sprint list page (src/config.js:43, cost: 1 + 50), 105 for a search and 260 for a changelog bulk fetch. On the scoring site its first scheduled run had counted 1,567 points after 21 seconds. The next estimated call would have taken it past 1,680. So it paused for the rest of the hour. Yet the site's own ledger for that run says it had spent 715 points over 27 requests, with list pages costing 1.5 to 2 points each.

That would only have been a nuisance if the migration ran first. It ran last. reconcile() in src/index.js calls reconcileAll, then saveClosed, then migrateV1Rows, and the pause in the first step skipped the rest. Its rescan kept no record of where it had stopped. Each hour it started over and paused in the same place. Our scorer's verdict reads "0/130 historical changes v1 never recorded" and "h2: 0 of 59 v1 rows in scope-ledger", and it traces 15 checks to that one root cause.

The gate tripped twice on the development site too, which our spec sets up the same way as the scoring site but from another seed. At 21:19 our time it wrote "The reconcile hit my own dose gate". After a reset one run got through and it reported "migration complete (88/88)".

Build 2 got the migration right. I think it was more luck than design. Its cost counter under-counted instead (its own words near the end: "my costOf under-counts returned records"). It ran over the background share in 4 of 31 hours, and on our scoring sites it never paused early. In its last minutes its own dev run did pause early. At call 300 of 300 it spotted the ordering problem: "I'm at the call budget (300/300) — this is my final turn". Then it moved the migration to the top with the comment "it is the graded deliverable and needs the fewest points". It never ran that version. It still scored 59 of 59 migrated and 130 of 130 backfilled. DeepSeek 1017's build 2 had that ordering from the start (src/sync.js:409, "Migrate v1's rows first").

Neither build displayed its admin page

Both Mistral builds score 0.00 on the admin panel tier. That one failure takes down the panel labels, the CI web trigger (no secret can exist before the panel's Rotate button creates one) and the admin boot check with it. Build 1's src/frontend/admin.jsx ends with export default AdminPage; and never calls ForgeReconciler.render, so UI Kit had nothing to draw. It found that out at 21:39, on the 298th of its 300 calls, when its only admin-page test printed "(nothing rendered)". Build 2 does call ForgeReconciler.render(<App />) at line 197, and the non-admin view displayed its permission message, but the admin view came back as an empty tree. The probe recorded no error that explains it. And build 2 never ran the kit's UI Kit check on that page.

DeepSeek 1017's build 2 renders the same way (admin.jsx:186) and ran that UI Kit check 14 times before it finished. The habit shows in the counts too: DeepSeek's two builds invoked their resolvers in the sandbox 45 and 30 times, Mistral's 3 and 4, and neither Mistral build ran a browser self-test at all. Build 1 also sent 37 of its 355 tool calls, about one in ten, to a bash tool that doesn't exist (the harness's is called shell), and build 2 sent 19 of 344.

Build 2 had good moments. It had the highest UI-function (0.9957) and reconcile (0.9974) tiers of the four builds in this comparison group. Both are small tiers. Both also beat an Opus 5.5 build from an earlier group. In its last ten minutes it noticed that v1's React bundles were 244 KB against a 150 KB boot budget. It rewrote all three Custom UI pages in plain DOM, to between 102 and 106 KB each.

Mistral Large 4 vs DeepSeek V4.1 Flash 1017: 27 minutes to the first file against 2

Time is where the two models differ most. Mistral's first app-code write came 32, 35 and 27 minutes into its three Gauntlet attempts of 18 and 19 October, and about 49 and 55 minutes into its Forge builds. DeepSeek 1017 wrote its first app file 2.4 minutes into its Gauntlet run and about 4.5 minutes into each Forge build, and it finished both Forge builds with 29 to 37 calls to spare. Mistral ran out of calls in both.

Its median call took 4.5 to 4.8 seconds on Forge against DeepSeek's 4.0. The typical call barely differs. The slow tail is what costs it. Each Mistral run had calls that took 11 to 18 minutes, and in the Gauntlet attempts the calls over five minutes add up to more than half of all the time the model spent answering. DeepSeek's slowest call in any of its three runs took 71 seconds. Across our runs Mistral generated between 55 and 70 tokens a second, while DeepSeek managed 108 to 155.

Illustration, generated locally with ERNIE-Image. 37,853 words of plans, 18 minutes, and the first brick came after. Then the plans were thrown away.

The 18-minute thought that was thrown away

The 18 October attempt that Mistral's servers later ended opened with one thinking call of 37,853 words that ran 1,081 seconds. It was coherent (under 1% of its 12-word sequences repeat). It covered the money rule, the 3D tower heights and the order of the services. Then it stopped at exactly 65,536 output tokens. The harness marked the reply as cut off, and nothing in it was carried out. Its next thought: "Let me continue. I was planning extensively. Time to write code." The harness sent no max_tokens and our model limits list 262,144 output tokens, so the 65,536 cap came from the serving side. Whether that's OpenRouter or Mistral, our logs don't show.

Reasoning was 57% to 81% of everything it generated, depending on the run. On Forge both builds also compacted their context three times, meaning the harness summarised the conversation so far to free room, once the prompt had grown to between about 170,000 and 210,000 tokens. After most compactions it re-read files it had already read ("I need to re-read the v1 sources and my v2 files"). Build 2's own compaction summary says it plainly: "I've spent ~66% of context on exploration; no code".

Those compactions are partly ours. The harness gave Mistral a usable window of 262,144 tokens. It took the smallest Mistral endpoint's 524,288 (the EU one) and reserved half of it for output, on a model whose card says 1M, while DeepSeek got 655,360. DeepSeek sent about 200,000 tokens a call, so under Mistral's limit it would have compacted too. Under its own limit it never did. The summary calls ran about 9 to 11 minutes per Forge build, inside the times charged to Mistral.

What did the long thinking buy? On the Gauntlet, quite a lot: its plans got the money rule right, and its decisions file is careful (a rejected draft is terminal, and any further action on it answers 409 invalid_state). On Forge it bought almost nothing the tests could see. A build that reads for 50 minutes and finds its admin bug on call 298 has no calls left to fix it.

Unexpected role 'tool' after role 'user': the 400 that was our bug

The third Gauntlet attempt ended at 02:05 our time on 19 October, 97 calls in, on this answer from Mistral's API, passed through OpenRouter:

{"object":"error","message":"Unexpected role 'tool' after role 'user'","type":"invalid_request_message_order","param":null,"code":"3230","raw_status_code":400}

Mistral had asked for two tools in parallel, read_image on a screenshot and shell. OpenAI's chat format only accepts text in a tool message ("For tool messages, only type text is supported", in OpenAI's API spec). So since the upstream agent's 1.0 release the formatter has put a placeholder in the tool result and sent the picture as a separate user message right after it. With one tool call, or when the image comes back last, that's harmless. When an image result comes back before another tool result, the request goes out as assistant (two tool calls), tool result, user message holding the image, tool result. Mistral refused the second tool result because a user message stood between it and its call.

Who is right? Mistral's function-calling docs show the order only as a table ("assistant fc.1, fc.2 → tool r.1 → tool r.2 → assistant"), and the error type invalid_request_message_order isn't in their error glossary. Anthropic states the rule in words: "You cannot include any messages between the assistant's tool use message and the user's tool result message." Our request broke the documented order. I count this one as ours.

Three of our runs from 3 and 4 October sent the same order, and all three were answered and scored: Claude Sonnet 5.5 (served by Claude on AWS), GPT-6.1 Sol Pro (OpenAI) and MiMo V2.6 Flash (Xiaomi). All three went through OpenRouter. I only see what the harness sent to OpenRouter, so our logs can't tell whether OpenRouter reordered anything for those hosts. Mistral refused it the first time. Mistral's own coding CLI has an open pull request for the same failure, where a steered user message ends up between a tool call and its result.

We fixed the harness 17 minutes after the 400 (commit a0247fffc2, "keep parallel tool results contiguous when one carries an image"). Within each group of consecutive tool results, the image messages now move after the last result and are merged into one user message. A test replays the exact refused request. It shipped in 3.0.119 of our desktop app on 19 October. The scored fourth attempt most likely ran on 3.0.119, and it never called read_image, so the fix wasn't exercised in a Mistral run.

The second attempt's ending wasn't ours. At 19:40 our time on 18 October Mistral returned "Internal server error" (type unreachable_backend, a 503). The harness resent three times with backoff, the last one failed the same way, and the run ended after 30 answered calls and at least $1.18. The 9 October attempt ended on OpenRouter's side. Three times, while it was writing its main service file, OpenRouter dropped the reply with "Upstream idle timeout exceeded", and the harness gave up after the third resend. Our harness refuses to score a session the provider ended, because grading a half-built tree would publish an outage as the model's score.

What a Mistral Large 4 build costs: about 18 times DeepSeek's, at sale prices

Build 2's bill on its run page.

Its Forge pair billed at least $10.95 for a median of 0.2195. DeepSeek 1017's pair billed at least $0.62 for 0.452, which works out at about $50 per full point (1.0) of median score for Mistral, against $1.36 for DeepSeek.

Those Mistral prices were the launch sale: Mistral's changelog said "Launch pricing: 50% off for 2 weeks" from 6 October, and OpenRouter still showed the sale prices on the 20th. At the regular rates its model page listed then ($1.36 per million input tokens, $4.18 output and $0.14 cached), the same pair would have cost roughly twice as much. Mistral has since cut the list price to within 3% of the sale price, so that estimate no longer holds; the November section near the end has today's numbers.

Where Mistral's money went:

	Mistral b2	DeepSeek 1017 b2
Requests	305	271
Prompt tokens per request	99,182	201,619
Share of prompt served from cache	84.6%	99.7%
Cache-read price per million (OpenRouter list; DeepSeek at the off-peak rate it ran at on Sunday 18 Oct, double at weekday peak)	$0.07	$0.003
Uncached prompt price per million	$0.68	$0.15
Bill	$5.55	at least $0.32

DeepSeek carried twice the context into each call, inside a window two and a half times larger. Nearly all of it was billed as cache reads at a price 23 times lower than Mistral's. At list prices, fresh prompt tokens make up about half of each Mistral Forge bill, though they're only 13% to 15% of its prompt tokens. My guess is that the three compactions per build and the re-reads after them are a large part of that uncached slice. On the Forge builds the compaction summaries also ran on Mistral itself, because our host pin caught the helper calls meant for a small Gemini model. We fixed that while build 2 was still running. It shipped in 3.0.119, after the pair. The fourth Gauntlet attempt's four Gemini helper calls are inside its $4.93.

What the architecture might explain

Mistral published less than I'd like in October. The model card said "a granular Mixture-of-Experts architecture" with "52B active parameters and 1.05T total parameters, and a 1.6B vision encoder", a 1M context and "Status: Public Preview". The Hugging Face placeholder said 49 billion active per token, 52 billion counting embeddings and output layers, with weights expected on 31 October. The launch post deferred the architecture, "additional benchmarks, and our post-training methodology" to the weights release, and said "the reinforcement learning run behind this preview is still in flight". Expert counts, the attention design and a licence weren't published then, and there was no tech report. The weights and the licence came out on 29 October; the November section near the end covers them.

On reasoning, Mistral's reasoning guide documented two values for this model in October, high and none. We sent medium. We do that for all our cloud runs. It's a valid value in Mistral's API schema, OpenRouter then listed only high and none and defaulted to high, and nobody documented what medium became. The 57% to 81% reasoning share suggests it was not thinking minimally, but that's an inference from our token counts.

I have two guesses about the behaviour. A preview whose RL run is "still in flight" may not yet be tuned for long agentic loops, where a long think costs calls and minutes. Our Forge logs look like that. Thorough reading, late writing, no time left to test. And Mistral's 49B active parameters per token (52B with embeddings) are about three times the 16B active during generation listed on the V4.1 Flash card, which the 1017 snapshot updates. That fits the gap in tokens per second we measured, though serving hardware and load matter as much, and our numbers can't separate them. Nothing in the cards explains the admin page or the throttle. Those are decisions in the code. I'd judge it on the code.

Mistral's own lifecycle page adds one more caveat: Public Preview models "allow silent updates". A pinned mistral-large-4-0 today may not be the model we measured on 18 and 19 October.

Where we were wrong

1. Our Gauntlet money check assumed the table opens oldest first and charged Mistral x0.6 for correct money. Joining each cell to its payment by row id fixes it; it isn't shipped, and the board still shows 0.5462.
2. The same check scores a currency that isn't on screen as a failed test. Four other ranked runs paid x0.92 for it.
3. We capped Mistral's context at 262,144 tokens on a 1M model, against 655,360 for DeepSeek, so its six Forge compactions and their minutes and dollars are partly ours.
4. The harness put a user message between two tool results whenever an image came back before another result. Mistral refused it, and that ended a run. Fixed in 3.0.119.
5. Under a Mistral host pin our helper calls went to Mistral too, so its Forge bills include summary work meant for a cheaper model. Fixed in 3.0.119.
6. We asked for reasoning effort medium, which Mistral didn't document for this model at the time, so the effort it really ran at was the provider's choice.

Mistral Large 4 vs DeepSeek V4.1 Flash in November: open weights, general availability and a lower list price

Since I wrote this, Mistral has put the weights up on Hugging Face (29 October). They come under its own Mistral Ecosystem Open-Weight License, which Mistral describes as a modified MIT licence, and anyone who sells the model as a hosted service needs a separate commercial licence. The model card now says GA, generally available, where it said Public Preview. The list price is now $0.70 per million input tokens, $0.07 cached and $2.10 output, on Mistral's card and on OpenRouter, a cent or two above the launch-sale prices we ran at ($0.68, $0.07 and $2.09). So the same tokens behind the at least $10.95 our Forge pair billed would cost at most 3% more today, and the estimate further up of roughly twice that at the old regular rates no longer holds. As billed, the pair cost about 18 times DeepSeek V4.1 Flash 1017's.

The reasoning setting moved too. Mistral now documents low, medium and high, with high as the default and low replacing none, so the medium we sent is a documented value today. Mistral also says the open-weight model had three more weeks of reinforcement learning after the preview, and its Hugging Face page says that model is served on Mistral's API as mistral-large-4. We called mistral-large-4-0 through OpenRouter, which still lists the 6 October snapshot, and the card still says version 26.10, so I can't confirm which weights that name serves now. We haven't run it again, so every Mistral score and bill on this page comes from the preview as it answered on 18 and 19 October.

The plain DeepSeek V4.1 Flash, without 1017 in the name, is DeepSeek's September release, and it's on our boards too. On the Gauntlet it scored 0.591 for $0.25, above Mistral's 0.5462. That gap is mostly our bug: the September run hit a cap of 0.599 because its 3D panel came up empty, and Mistral's run would be about 0.70 without our false critical. Its Forge 2.0 pair scored 0.2893 and 0.379 (a median of 0.3342) for $2.39 together, against Mistral's median of 0.2195 for at least $10.95. Its Forge builds sit in a different comparison group from Mistral's and its Gauntlet run is two weeks older, with our setup changed in between, so I'd treat both gaps as rough. On the boards today (6 November) Mistral's Gauntlet run is 34th of 51 ranked runs and still shows 0.5462. At about 0.70 it would be 11th of 51.

I wrote up separately how Ricursive's open-weight Ripple-65B Preview did on our Gauntlet board next to DeepSeek V4.1 Flash 1017: Ripple-65B Preview benchmark.

Should you put Mistral Large 4 behind your agent?

From these runs, not yet for long agentic builds like our Forge task. It read for close to an hour and ran out of calls, and as billed it cost about 18 times DeepSeek V4.1 Flash 1017. On the Gauntlet, once our check is fixed, it would sit around 11th of 51 runs on today's board. The weights are out now and the API model is generally available, so the next thing I'd want is a fresh run with the effort set to high and a full context window.

If you try it in your own setup, test four things. First, check whether your agent ever puts a user message between tool results (images, steering, injected reminders), because Mistral rejects that request outright. Through OpenRouter, the hosts behind three of our earlier runs answered it. Set the reasoning effort to a value Mistral documents. Watch the time to first write and the slowest calls rather than the median, since the 11- to 18-minute calls are where its time went. And read the failing check before you believe a critical, ours included.

Our desktop app isn't production-ready yet. We run these boards with it because we know what it's made of. I'd like to hear from anyone running Large 4 in a real agent loop: did you get the long first thought too, and did the lowest effort (none in October, low now) make it usable, or just cheaper? Tell me in the comments.

---

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.