A peak score tells you what an AI system can do. Production cares about what it can do again tomorrow.
We recently got 91% on Terminal-Bench 2.1 using Kimi K3.
Tasks solved: 81/89
Total model cost: $28.72.
This is the part where I am apparently supposed to put the number in very large type, add a trophy emoji, and move on.
Unfortunately, we ran the experiment more than once.
And Kimi does not score 91% every time.
That is where the story gets much more interesting.
91% is a Result. It Is Not a Personality Trait.
It would be very convenient to say: Kimi K3 is a 91% Terminal-Bench model.
It would also be wrong.
We have seen meaningful variance between runs with Kimi and other open-weight models. Sometimes the result is exceptional. Other times it comes down materially. The 91% run was also deliberately different from an official Terminal-Bench submission.
We ran all 89 tasks once. An official submission requires at least five attempts per task. So this was an experimental run, not a judged leaderboard submission. That qualification makes the headline slightly less exciting. It also makes the experiment considerably more useful.
Because the question we actually care about is not “How high can the number go?”
It’s “How much of that performance can we make repeatable?”
Peak Performance and Production Performance Are Different Things
Imagine an agent scores:
91% on Monday.
87% on Tuesday.
83% on Wednesday.
What is its capability?
All three numbers tell you something.
The 91% tells you what the system is capable of under a strong trajectory. The lower results tell you what you might actually have to depend on.
And if I am putting that system into a production workflow, I become considerably more interested in the second number.
Nobody wants an agent whose résumé says “occasionally brilliant.”
Open Models Are Getting Uncomfortably Good
This is what caught our attention.
Even when Kimi comes down from that particularly strong run, the performance range we are seeing from open-weight models inside a well-engineered agent system is becoming surprisingly competitive.
For years, AI performance was mostly discussed as a model-selection problem.
Want better results? Use the better model.
Everything around the model was plumbing. Necessary plumbing, perhaps, but definitely an expensive plumbing.
I think that assumption is starting to break.
As more models move into broadly useful capability ranges, the system around them starts determining how much of that capability actually reaches the user.
And small architectural decisions begin to matter much more than they look like they should.
We Gave Kimi Less to Do
One of the more useful lessons from this run was that better agent design does not always mean giving the model more.
Sometimes it means taking things away.
R-CLI did not use Kimi K3 for everything. Kimi handled the primary reasoning, while delegated subagent work could be routed to DeepSeek V4 Flash.
We also removed 10 tools from the schema because they either conflicted with benchmark constraints or had measured negative effects on performance.
That meant a smaller tool surface, less prompt overhead, and fewer irrelevant decisions.
There is a natural temptation when building agents to keep giving them capabilities.
Another tool.
Another integration.
Another 40,000 tokens of context, just in case.
Eventually your extremely intelligent model is standing in the middle of Home Depot holding 900 tools and wondering which screwdriver you wanted.
Capability is useful.
Choice has a cost.
Context Windows Are Not Storage Units
The same principle applies to context.
Large context windows are useful. That does not mean the goal is to fill them.
In the 91% run, 71.8% of input tokens were served from cache, while median context occupancy was only about 1.5% of the model’s available window. That number is worth sitting with.
The model had a large context window. We mostly did not use it, because the objective is not to make sure the model has seen everything. The objective is to make sure it sees what matters now.
There is a subtle but important distinction between giving a model enough information and making it read the entire filing cabinet before answering every question.
One helps. The other is technically also a strategy.
The Model Did Not Change
None of these decisions changed Kimi’s weights.
We did not make the underlying model smarter. We changed the environment in which it operated.
The harness controlled what context arrived, which tools were available, when work could be delegated, and how much the system should spend. It also used turn timers, budget-aware spend nudges, and a cache-first approach to context. Those choices look small individually. Across hundreds of agent steps, they compound.
A capable model can still underperform if the system keeps giving it irrelevant context or an unnecessarily complicated tool surface.
It can waste enormous amounts of computation following bad trajectories.
It can fail once and never recover. The underlying intelligence may be there.
The harness determines how much of it survives contact with the task.
$28.72 Is Interesting
The full 89-task sweep cost $28.72 which is an impressive number. It is also one I would handle with oven mitts.
Dollar comparisons between AI systems are messy.
Model prices differ.
Caching policies differ.
Commercial agreements differ.
Subscription economics can make apparently simple comparisons considerably less simple. Token consumption gives another useful signal.
Our run used roughly 0.24 million tokens per trial. Several configurations included in our analysis consumed more than one million tokens per trial, while the highest-scoring submission in that comparison used about 2.65 million. That does not make token count a perfect measure either.
But if two systems are attempting similar work and one requires dramatically more model activity to get there, something meaningful is happening in the architecture.
Efficiency is not just model pricing.
It is also how intelligently the system uses the model.
Reliability Is an Engineering Problem
Here is the encouraging part about variance:
It is frustrating.
But some of it creates things engineers can work on.
If one trajectory succeeds and another fails, we can inspect why.
Maybe routing reduces the variance.
Maybe verification catches a weak trajectory before it becomes a failure.
Maybe retry behavior needs improvement.
Maybe a specific behavior can be improved through post-training.
Maybe a different model should handle a particular stage of the task.
The source experiment makes exactly that broader point: inconsistency is not necessarily a model-only problem; the surrounding system can influence how repeatable high performance becomes.
Which means the goal should not simply be:
Make the highest number possible.
It should be:
Make good performance boringly repeatable.
That is a much harder engineering problem.
It is also the one production systems eventually have to solve.
A Benchmark Score Is a Distribution Wearing a Name Tag
This is also why I think AI benchmarks need to mature. A single aggregate number is wonderfully easy to understand. It is also very good at hiding everything underneath it.
Variance matters.
Failures matter.
Retries matter.
Cost matters.
A 91% peak and an 85% repeatable system can tell you very different things.
Neither number is automatically more “real,” because they answer different questions.
One tells you about the ceiling.
The other tells you about reliability.
Production needs both.
The Interesting Competition May Be Moving Above the Model
There will always be another model.
GPT.
Claude.
Gemini.
Kimi.
DeepSeek.
Then six months later something with a name that sounds like either an AI lab or an obscure Pokémon comes out.
The frontier will keep moving, but if several models increasingly occupy the same broad capability range, then the scarce part of the system starts to change.
The advantage shifts toward the teams that can make those models more dependable without wasting enormous amounts of computation getting there.
It moves toward the people who understand how to build systems around intelligence rather than simply purchase access to it.
That is the bet we are making at Backboard.
The 91% run is exciting.
The cost is exciting.
But neither is really the thesis.
The thesis is that open-weight models are getting better quickly, while the difference between an average harness and a great one may only be starting to matter more.
And if that is true, the next major advantage in AI may not come from owning the smartest model.
It may come from making strong performance so repeatable that eventually nobody finds it interesting anymore.
This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.
Top comments (0)