A benchmark called Evo-Bench measured what happens when a language model is asked to improve not its answers but its own operating scaffolding, and found gains of up to 16.6 points that come "closely approaching state-of-the-art human-engineered baselines." The scaffolding models build also transfers: harnesses evolved by one model consistently boost others. The exception is telling -- self-evolution fails on office tasks that require following a specific prescribed workflow.
Key facts
- Top models achieve absolute gains reaching 16.6 points by evolving their own harness, closely approaching human-engineered baselines.
- Evaluated across nine frontier and open-weight models in three domains: Search, Office and General.
- Self-evolved harnesses act as transferable reasoning structures, improving other policy models than the one that built them.
- Primary source: Evo-Bench: Can Language Models Improve Agent Harness? from a RUC AI Box team led by Lisheng Huang, with public code.
The measurement problem
An agent harness is everything around the model: available tools, how context is assembled, what the system prompt says, how results get validated. It matters enormously -- we have reported that the harness, not the model, moved DeepSeek by twenty tasks, and that the same model scores 52 or 81 percent depending on the code around it.
So the obvious next question is whether the model can write that scaffolding itself. The obvious next problem is that measuring it is a trap. Swap in a stronger model and scores rise for reasons that have nothing to do with harness quality. Let the agent tune against the test set and it overfits. Give it a loose protocol and you cannot tell what caused what.
Evo-Bench's contribution is the isolation. The team fixes the policy model and the budget, then applies what they call harness-guided construction: run auxiliary-task evolution to find which tasks are "genuinely sensitive to framework improvements," then apply sensitivity-aware stratified splitting so the validation and evaluation suites generalise across each other. Only tasks that actually respond to scaffolding changes are measured, and any score movement belongs to the harness.
Think of it as testing kitchen layouts rather than cooks. Keep the cook, the ingredients and the clock constant; change only where the knives live and how the counters are arranged; measure the dinner.
What they found
Two results, one encouraging and one strange.
The encouraging one: 16.6 points of absolute gain, closely approaching what human engineers achieve, and -- more interesting -- the scaffolding is portable. The paper describes the synthesised harnesses as "highly transferable reasoning structures, consistently boosting diverse policy models." What one model builds, other models can use. That reframes harness evolution from a per-model tuning exercise into something closer to shared infrastructure.
The strange one: "critical temporal anomalies like early saturation." The agent stops improving long before its budget is exhausted. It finds a local shape it likes and stalls. For anyone reasoning about recursive self-improvement, that plateau is arguably the most important number in the paper, and it is not the headline.
Where it fails, and why that is the interesting part
Autonomous evolution outperforms the artificial harness on General tasks and excels on Search. It "struggles in Office tasks that demand highly specific processing workflows."
That failure pattern is diagnostic rather than embarrassing. Search and general reasoning are open -- there are many good ways to do them, and a model exploring the space can find one. Office work is not open. A specific report has to be assembled in a specific order according to somebody's specific procedure, and there is no way to derive that procedure from first principles because it is a convention, not a truth. The model can invent a better way to think. It cannot invent your company's approval chain.
Why it matters
This is the empirical floor under an argument that has been mostly theoretical. Ryan Greenblatt told Dwarkesh Patel this week that the missing ingredient in AI research is operational taste, not deep insight. Evo-Bench puts a number on a nearby question: models can now do a substantial fraction of the scaffolding work humans have been doing by hand, and they lose exactly where the task is somebody else's process.
Four independent groups converged on this question within a week, including the 161-day self-editing deployment published alongside it. That convergence is itself a signal about where agent research is heading.
The honest caveat
"Closely approaching" is doing real work in that sentence -- the human-engineered baselines are still ahead, and the paper does not claim otherwise. Three domains is a narrow slice of what agents are asked to do. And early saturation means these gains are not a curve you can extrapolate: the agent stops, and nobody yet knows whether that ceiling is a property of the method, the budget, or the models.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)