Gemini 4 Carbon landed on October 28 at the top of the Artificial Analysis Intelligence Index v5.0 (59, vs 58 for Claude Opus 5.5 and 57 for GPT-6 Astra). It's only available to cyber defenders and vetted partners right now, so none of us can test it. I still found the comparison table useful, mostly as a reminder of how to read these things.
1. Treat a one-point lead as a tie
Blended indexes average a lot of different tasks. A 59 vs 58 difference tells you the models are in the same tier, not that one is better for your workload.
2. Look for the rows where the winner loses
Google's own table shows Opus 5.5 ahead on OSWorld-2.1 (87.2% vs 81.5%) and GPT-6 Astra ahead on FrontierSWE 2 (65.5% vs 61.2%). If your product is a computer-use agent or a coding agent, the "top model" is not the top model for you.
3. Find the outlier and ask why it's there
Carbon's big gap is in security: 88.2% on SEC-bench Pro, 54.8% on ExploitGym (vs 42.4% and 34.5%), 71% on CWE-bench. The announcement says these were run with guardrails off. The outlier explains the gated release better than any press statement could.
4. Check whether you can actually call it
Pricing is published ($2/M input, $10/M output, same as Argon), but there's no developer access date. A model you can't call is not part of your evaluation, no matter how it ranks.
What I'm doing
Keeping my current setup and adding Carbon to my eval list as "pending." When it opens, it goes through the same small task set everything else does: a couple of refactors, a log-parsing script and one long-context summary.
Curious how others handle this. Do you keep a personal eval set, or do you mostly go by public leaderboards?
Top comments (0)