DEV Community

Cover image for Gemini 4 Argon: What the Benchmarks Don't Tell You
ali
ali

Posted on Originally published at logicstack.org

Gemini 4 Argon: What the Benchmarks Don't Tell You

Google spent the last two weeks telling developers that Gemini 4 Argon is the most capable model it has ever shipped. The charts are everywhere: bars where Argon edges past Claude Opus 5.5 and GPT-6 Astra, a price tag of two dollars per million input tokens, and a first-place finish on a public arena leaderboard. But the benchmark tables making the rounds were measured at a setting most subscribers will never touch, and that headline price has an expiry date. Here is what the marketing slides leave out, and what actually matters if you are thinking of switching your workloads over.

What Google actually announced

Google unveiled Gemini 4 Argon on 30 September and moved it to general availability this week. Developers can call it through the API at introductory rates of $2 per million input tokens and $10 per million output tokens. Google AI Pro subscribers get Argon's reasoning capabilities inside the Gemini app and in AI Mode in Search. What they do not get is Google Antigravity, the agentic coding environment that edits multiple files, runs tests, and iterates without human prompts between steps; that remains exclusive to Ultra subscribers.

The model ships with a 1,048,576-token context window shared across input, internal reasoning, and output. By default a reply is capped at 65,536 output tokens; longer background requests can chain 64K continuation steps for up to roughly a million output tokens in total. It accepts text, images, video, audio, and PDFs, but replies in text only, and image or audio generation is not supported.

The benchmark tables everyone is sharing

Google says Argon beats GPT-6 Astra and Claude Opus 5.5 on 13 of 19 benchmarks in its own comparison. The scores most quoted by the tech press, reported by VentureBeat against rival models, look like this:

  • DeepSWE v1.1 (software engineering): Argon 77.9%, Opus 5.5 74.2%, GPT-6 Astra 74.1%

  • Zapier AutomationBench (business automation): Argon 51.3%, Opus 5.5 42.5%, Astra 41.4%

  • Terminal-Bench 4.0 (terminal tasks): Argon 57.4%, Opus 5.5 66.4%, Astra not reported

  • Artificial Analysis Intelligence Index: Argon 53, Opus 5.5 58, Astra 53

These are genuinely strong numbers on software-engineering and automation tasks. But read the column headings carefully: Google's own tables, and most press coverage, use Argon's scores at the High reasoning setting. The default is Medium. That distinction changes almost everything, which is where the fine print comes in.

The fine print: those scores were measured in High mode

Argon exposes three thinking levels: low, medium, and high, with medium as the default. Google's published scores were measured in High mode, not the setting subscribers get out of the box. Independent testing makes the gap concrete: on Artificial Analysis's Intelligence Index, Argon scores 52.56 in High mode, within half a point of GPT-6 Astra (52.67), but around five points below Claude Sonnet 5.5 (56.00) and Claude Opus 5.5 (57.62). One bright spot that does survive independent scrutiny: a 15% hallucination rate, the lowest of any model scoring above 45 on the index, versus 51% for Astra at its max setting.

The default-effort picture is murkier. Cursor reports Argon at 52.8% on CursorBench at default effort, on par with Opus 5.5. Scale AI ranks Argon first on its Sixth Sense test at 55.4%. These are credible but preliminary; Arena's text leaderboard ranked Argon first at 1525 based on 4,942 votes, explicitly marked preliminary.

There is a subtler trap for engineers comparing models. All three flagship labs now use the word "medium" for their default reasoning effort, and nothing on any of the three documentation pages says the three defaults spend the same amount of thinking. Never mix sources into one comparison table: Anthropic reports Opus 5.5's Terminal-Bench number at xhigh effort, while Google's numbers come from its own runs at High.

Pricing: cheap now, double later

The $2/$10 rate is an introductory offer, and the honest budgeting move is to price your workloads at the post-introductory rate. Third-party reporting says the price doubles to $4 per million input tokens and $20 per million output tokens, with one outlet pointing to 1 December 2026 and another saying the intro rate runs through the end of 2026. Google has not pinned down the date publicly, so treat any specific cutoff as unconfirmed. For European teams, that works out to roughly €1.80 in and €9.00 out per million tokens at current exchange rates, before VAT.

Why does this matter? Independent cost analysis puts Argon's per-task cost at about $1.99 on the Artificial Analysis index at intro rates, roughly 60% of GPT-6 Astra's. At post-introductory rates that reverses to about $3.98 per task, around 120% of Astra's current rate. If you migrate a pipeline today and forget to re-run the numbers in January, your inference bill quietly doubles. As one comparison site put it: budget for the full price, not the introductory one.

What the API actually looks like: the migration gotchas

Argon changes several behaviours that trip up teams porting code from older Gemini models or from competitors:

  • Reasoning is the only knob that matters. Set thinking_level to low, medium, or high. The classic sampling parameters, temperature, top_p, and top_k, are accepted and silently ignored.

  • Caching is automatic only. Explicit cachedContents calls fail; only implicit caching on prompts of at least 4,096 tokens is supported.

  • Time to first token is slow. Independent benchmarks measure a median of roughly 43-45 seconds before the first answer token on 10k-token prompts. Google's migration checklist recommends HTTP client timeouts of at least 15 minutes for hard prompts.

  • Long output needs background mode. The 64K synchronous output cap means generating up to a million output tokens requires background requests that chain continuation steps on Google's servers.

A minimal, honest call pattern looks like this:

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

# thinking_level: "low" | "medium" (default) | "high"
# High matches the benchmark tables; Medium is what you
# actually get by default. Test both on YOUR prompts.
response = client.models.generate_content(
    model="gemini-4-argon",
    contents="Refactor this ETL pipeline to handle late-arriving events...",
    config={
        "thinking_level": "high",
        "max_output_tokens": 65_536,
    },
    # Google recommends HTTP timeouts of at least 15 minutes
    # for hard prompts against this model.
)
print(response.text)
Enter fullscreen mode Exit fullscreen mode

Note what the snippet does not set: no temperature, no caching configuration. On Argon those are dead controls.

What developers are actually reporting

The press-release story and the developer-forum story are not the same. In a Facebook developer group with 80 comments on the launch, most hands-on testers reported disappointment: outputs judged less accurate than Claude Opus 5.5 on multi-page websites and motion-graphics work, with one tester calling a physics simulator inferior and several agreeing Google overhyped the release. A minority defended Argon as comparable or faster, attributing the preference gap to Claude's design taste. The same thread clarified access confusion: Pro subscribers can use Argon in the Gemini app, not just Ultra users, contrary to the initial reading of the launch pages.

Bloomberg reported that some Google employees felt Argon underwhelms on coding and front-end work relative to its benchmarks, a characterisation Google called inaccurate. Sam Witteveen of Red Dragon AI gave VentureBeat the most balanced verdict so far: a really good model, but not obviously the best one available.

The pattern is consistent with the data. Argon's edge concentrates in agentic coding and business-automation benchmarks (DeepSWE, AutomationBench, CyberBench), where independent tests put it well ahead of Opus 5.5. Terminal-heavy work and front-end code are where Opus still leads and where developers are feeling the gap.

Should you switch?

Not yet on price alone, and not wholesale. The evidence points to a narrower, more useful decision:

  • Worth trying first: business-automation agents, legal and finance knowledge work, long-document analysis, and anything where hallucination rate matters more than raw score. Argon's 15% hallucination rate is the single most defensible number in the whole launch.

  • Test both on your own code: for general software engineering, run Argon at High and Opus 5.5 at high effort against your actual repository tasks before moving anything. The 77.9% versus 74.2% DeepSWE gap may or may not survive contact with your stack.

  • Keep Opus or GPT for terminal-heavy agentic work until Argon proves itself on your terminal benchmarks. 66.4% versus 57.4% on Terminal-Bench is not a small gap.

  • Re-run your cost model at $4/$20 before signing off. A migration approved on intro pricing is a budget surprise waiting for the new year.

What we still do not know

Honesty requires the unknowns on the table. Google's published tables had not been independently replicated as of the subscriber launch. The exact end date of introductory pricing is unconfirmed (1 December 2026 versus 1 January 2027 in different reports). Google's methodology page links Anthropic's Opus 5.5 system card rather than a system card for Argon, so the safety and evaluation methodology is, for now, Google's own account. And it is unclear what High mode costs subscribers in practice once usage quotas kick in. Read the fine print, run your own evals, and do not migrate a production pipeline on a press release.

Top comments (0)