JetBrains Ranked AI Agents on Real Kotlin Projects. The Token Column Is the Real Story.

JetBrains released the Kotlin Benchmark, its official benchmark for grading AI coding agents on real Kotlin engineering work. Claude Code with Opus 4.7 xhigh leads the first leaderboard at 85.7 percent, with JetBrains Junie and OpenAI's Codex right behind at 81.9 percent. But the resolution rate is the least interesting column on the page.

The number that deserves your attention is tokens per solved task. It ranges from about 66,000 to 777,000 across the top twenty setups. That is a 12x spread in cost for setups whose resolution rates differ by roughly 20 percentage points, and some of the expensive entries solve fewer tasks than setups spending a tenth of what they burn. That column, not the ranking, is what decides which setup makes sense for your team.

What the benchmark actually is

The Kotlin Benchmark follows the SWE-bench methodology, applied to repository-level Kotlin work. The dataset has 105 engineering tasks taken from active open-source Kotlin repositories. Each agent gets a real issue description, has to navigate the project, and produce a working patch. Solutions are verified in containerized environments, and a task only counts as resolved when the patch passes the repo's own tests. No self-reporting, no vibes.

JetBrains built it on the open-source Multi-SWE-bench infrastructure and published the datasets, test harnesses, and leaderboard. Alongside it, they released the methodology page and opened the GitHub repo so teams can inspect the tasks themselves.

Why Kotlin matters here: Kotlin is now the default language for Android, a first-class JVM language for server-side work, and the ecosystem has been poorly served by generic benchmarks. JetBrains already maintained model-focused evals like Kotlin_HumanEval, but those test whether a model knows syntax. This benchmark tests the layer above it: can an agent complete a validated engineering task inside an existing project, the way your team actually works.

The leaderboard, sorted by what it costs you

Here are the results, resorted by my own math into tokens per resolved task. Full disclosure: I have not run these evaluations myself. Everything comes from the official leaderboard, the JetBrains announcement post, and the methodology page. The token math below is mine.

• Claude Code + Opus 4.7 medium: 80 tasks, 5.31M tokens, 66K per task, 4h 54m total. The efficiency champion. Half the tokens of the top entry, 89 percent of the tasks solved.
• Junie + Opus 4.6: 81 tasks, 7.45M tokens, 92K per task, 8h 36m. Second most efficient, one more task than the leader above.
• Codex + GPT 5.3 Codex xHigh: 82 tasks, 8.56M tokens, 104K per task, 8h 1m.
• Claude Code + Opus 4.7 xhigh: 90 tasks, 10.59M tokens, 118K per task, 10h 37m. The #1 entry, and the only one to clear 85 percent.
• Claude Code + Sonnet 4.6 high: 78 tasks, 12.17M tokens, 156K per task, 13h 49m. A cheaper model, an expensive loop.
• Claude Code + Opus 4.6 max: 84 tasks, 13.93M tokens, 166K per task, 12h 39m.
• Junie + GPT-5.4: 72 tasks, 15.29M tokens, 212K per task, 11h 30m.
• Junie + Opus 4.7 max: 86 tasks, 19.98M tokens, 232K per task, 19h 58m. #2 in quality, nearly 2x the tokens of #1, and 20 hours of wall-clock time.
• Gemini CLI + Gemini 3.1 Pro: 69 tasks, 21.03M tokens, 305K per task, 17h 1m.
• Codex + GPT 5.5 xHigh: 86 tasks, 38.78M tokens, 451K per task, 8h 30m. Tied for second place in quality. 3.7x the tokens of the top entry for the exact same 86 tasks.
• Junie + Gemini 3 Flash: 64 tasks, 40.17M tokens, 628K per task, 11h 27m. A Flash model, the most expensive setup on the board after one other. If this does not kill the assumption that small models are the cheap option, nothing will.
• Gemini CLI + Gemini 3 Flash: 47 tasks, 36.53M tokens, 777K per task, 11h 28m. Last place in quality, roughly 12x the cost per solved task of the efficiency leader.

Read that list twice and the ranking stops mattering. Here is what jumps out.

Reasoning effort has terrible marginal returns on the margin you would pay. Claude Code with Opus 4.7 medium spent half the tokens of the xhigh setup and solved 80 of the 90 tasks xhigh solved. The extra ten tasks cost 5.28M extra tokens, which works out to about 528K tokens per additional task, roughly 8 times the average cost per task of the medium setup. If your tasks cluster near the easy end, xhigh reasoning is mostly paying premium prices for tasks medium already solved cheaply.

The same model in a different loop differs by 3.7x. Codex with GPT 5.5 xhigh tied Junie with Opus 4.7 max at 86 tasks. Codex did it with 38.78M tokens, Junie with 19.98M. The agent harness around the model matters as much as the model itself. This is the strongest public evidence yet for something practitioners keep saying: you are not picking a model, you are picking a loop.

Flash models are not cheap. Two Gemini 3 Flash entries sit at the bottom of the token-efficiency table, at 628K and 777K tokens per solved task. The obvious theory: weaker models fail more, and every failed attempt burns a full context of tokens before the verification step says no. Cheap-per-token is not cheap-per-solved-task when the failure rate is high. The real unit of cost is not the token, it is the solved task.

Wall-clock time is a separate budget. Junie with Opus 4.7 max needed nearly 20 hours to finish the suite. Claude Code medium finished in under 5. If your workflow is a developer waiting on an agent between code reviews, a 4x slower loop is a real cost even at identical token counts.

The uncomfortable context: 85.7 percent here, 38.8 percent elsewhere

One number from a different benchmark puts this whole leaderboard in perspective. Yesterday I wrote about Real-SWE, a benchmark that ran 8 frontier models on 10 tasks from private enterprise codebases. The best model there scored 38.8 percent.

The Kotlin Benchmark's top score is 85.7 percent. Same year, same models, same general category of work. The gap is not a mystery, and it is not mostly about language.

• Public tasks. The 105 Kotlin tasks come from open-source repositories. Whether any specific task leaked into training data is unknowable, but the exposure surface exists, and every agent in the leaderboard has almost certainly seen these repos during pretraining or in context.
• Public specs. An open-source issue usually contains enough context to solve it. Real-SWE graded against hidden requirements the original engineer had to discover from tickets, meetings, and institutional knowledge.
• Task scope. Open-source issues are typically session-sized. Real-SWE picked tasks real engineers worked on for weeks.

JetBrains itself says the quiet part in the announcement: the scores are "intended as a signal, not a guarantee for your codebase," and real results depend on your architecture, internal APIs, and validation process. Neither benchmark is lying. They are measuring different things, and the difference between 85.7 and 38.8 is roughly the difference between your public repo work and your private repo work.

What I would actually do with this

If you run Kotlin or Java server-side work and you are evaluating agent setups, this benchmark is the closest public proxy you have. Here is the approach it suggests.

• Pick two setups from opposite ends of the token table, not the top two of the ranking. Something like Claude Code medium and one premium setup, and run each on ten real tickets from your own backlog.
• Count solved tasks, not completed runs. A session that ends in a reverted patch cost you full tokens. Your internal metric should be tokens per merged task, which is the column this leaderboard quietly exposes.
• Watch latency separately from tokens. The 20-hour setup and the 5-hour setup are different products for different workflows, whatever their scores say.
• Demand this of any vendor claim. The next time a vendor tells you their agent "achieves state-of-the-art results," ask which benchmark, public or private tasks, and tokens per resolved task. A leaderboard without a cost column is a marketing document.

The takeaway

JetBrains did the ecosystem a favor here, and not just Kotlin developers. The interesting output of this release is not that Claude Code leads, or that Junie and Codex are close. It is that publishing tokens and latency next to resolution rates makes the quality ranking look almost like a distraction. Once you sort by cost per solved task, most teams' optimal setup moves down the leaderboard, and the assumption that more reasoning and cheaper models always help dies on the same page.

The benchmark is open, the tasks are inspectable on GitHub, and JetBrains plans to expand coverage to Android and Kotlin Multiplatform, add cost and maintainability metrics, and evaluate more open-weight models. If they add a Java edition with private-repo task sourcing, it might be the most useful benchmark in the JVM ecosystem. Until then, use this one the way it is meant to be used: as a signal, then run your own scoreboard on your own tickets.

I write about AI agents, JVM ecosystems, and backend engineering every week. If that is your world, subscribing is free and every post is grounded in linked sources.

Have you run coding agents against your own codebase as a benchmark? What did your tokens-per-merged-task number look like compared to the public leaderboards?