When an AI agent fails in production, the instinct is to blame the model. Usually that is the wrong place to look.
An agent's behaviour is governed as much by its harness as by the model underneath — the system prompt, the tools it can call, its memory, its verification rules, its runtime policies, and its failure-recovery logic. SWE-agent, Claude Code, Codex, and OpenHands all wrap the same frontier models; what separates a reliable agent from a flaky one is mostly that surrounding layer.
The catch: that layer is almost always tuned by hand. An engineer watches a few failures, forms a hunch, edits a prompt or a rule, and hopes. As Hangfan Zhang of the Shanghai AI Laboratory (lead author of the new Self-Harness paper, arXiv:2606.09498) puts it, the deeper problem is that this paradigm "often lacks a systematic feedback loop." With new models shipping every few weeks, hand-tuning a model-specific harness becomes a treadmill nobody can keep up with.
The idea: let the agent fix its own harness
Self-Harness keeps the model's weights frozen and improves the harness from the evidence of the agent's own execution traces — no retraining, and no dependence on a bigger external model to supervise it. It runs as a three-stage loop:
- Weakness mining — Run a batch of tasks with verifiable outcomes, then categorize the failed traces to find model-specific failure patterns.
- Harness proposal — A "proposer" role turns each failure pattern into a small, targeted edit tied to that specific mechanism — deliberately minimal, to avoid over-correcting.
- Proposal validation — Each candidate edit runs through regression tests and is promoted only if it improves performance without measurable degradation on held-out tasks. Passing edits merge into the next harness version, which seeds the next round.
The acceptance gate is the whole game: improvement without regression, proven on data.
The results
Tested on Terminal-Bench-2.0 (general tool use: artifact management, command use, verification, error recovery) across MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 — freezing everything except the harness — held-out performance climbed 33% to 60% in relative terms. Qwen3.5-35B-A3B went from 23.8% to 38.1%.
What makes it more than a benchmark number is what changed. The edits are specific and legible, not "make the prompt longer":
- MiniMax M2.5 kept exploring dataset configs until it timed out and shipped nothing. Self-Harness wrote a "loop breaker" into its runtime policy — stop and redirect after 50 tool calls — plus a rule to produce an initial version of any required artifact early.
- Qwen3.5 would hit a file-overwrite error, blindly retry the same command, and eventually delete files in confusion. The fix: a strict command-retry discipline — no exact-duplicate commands.
Those are exactly the rules a seasoned engineer would add — discovered and validated automatically.
Why this matters beyond the benchmark
If you ship agents on real workflows, three implications stand out:
- Model upgrades stop being rewrites. Swapping in a cheaper, faster model usually means re-tuning the harness by hand. A self-harnessing loop re-discovers the new model's failure modes automatically.
- Cheaper models cross the reliability line. A 60% relative lift on hard tasks can move a smaller model into "good enough for production."
- Debugging becomes empirical. Ambiguous "the agent looks broken" failures become testable edits with an objective accept/reject gate.
The prerequisite is honest: this only works on tasks where success is machine-checkable. Before you let a loop edit your harness, build the evaluation — a benchmark of real cases with objective pass/fail signals. Without that, "self-improving" is just a word.
This is a syndicated copy. The original, with the real-estate and PropTech angle, is on the VSBD blog. Paper: arXiv:2606.09498.
Top comments (0)