A new agent harness raises Qwen 3.7-Plus from 51.8% to 80.7% on the WeaveBench coding benchmark without changing the model at all. On Terminal-Bench 2.1 the same swap moves it from 69.7% to 77.2%, and on OSWorld 2.0 from 2.8% to 8.3% binary completion. The mechanism is not a better prompt: LongHorizon-Harness keeps task state outside the execution loop and updates it only from facts a separate read-only auditor verified against the environment. The authors' conclusion is that "agent capability is a property of the complete model-harness system."
Key facts
- WeaveBench: 51.8% to 80.7% PassRate with Qwen 3.7-Plus, backbone held constant. Terminal-Bench 2.1: 69.7% to 77.2%. OSWorld 2.0: 2.8% to 8.3%, and 20.6% to 35.3% on a 34-task Claude Opus 4.7 subset.
- Cost: average output tokens per OSWorld task rise from 28.9K to 104K, with the auditor named as the main new expense.
- The harness runs a Manage-Execute-Audit loop; the code is public.
- LongHorizon-Harness, arXiv:2608.01964, with companion paper Model or Harness?.
The diagnosis is more specific than "context too long"
Everyone knows long agent runs degrade. The usual explanation - the context window fills up - is true and not very useful. This paper's account is sharper: existing harnesses let task execution, task state and completion assessment all share the same growing conversation. So when an agent writes "I have installed the dependency and the tests now pass," that sentence becomes part of the record whether or not it is true, and every later decision reads it as fact.
The named failure modes follow from that. Compounding errors: a wrong belief early poisons everything downstream. Goal drift: the original objective gets diluted by hundreds of intermediate steps. Context rot: the useful facts get buried or compacted away.
The fix, in three roles
LongHorizon-Harness splits what a single agent normally does into three jobs.
A manager holds the task state - explicitly, outside the conversation - and chooses the next bounded subtask. An executor starts with a fresh context, does only that subtask, and reports back. Then a read-only auditor goes and looks at the actual environment: did the file change, does the service run, does the test pass. Only after the auditor confirms does the manager update state.
The analogy is a construction site. The old design is one worker keeping the plans, doing the work and signing off on it, all in the same notebook, which gets messier by the hour. The new design keeps the plans on a board by the door, sends the worker in for one job at a time with a clean head, and has an inspector walk the site before anything gets ticked off.
The critical detail is that the auditor is read-only and independent. It does not take the executor's word. That is the entire point: unverified claims never become durable state.
The tradeoff is real and stated
Output tokens per task on OSWorld more than triple - 28.9K to 104K - and the auditor is where most of it goes. That is the honest shape of the result. Better scaffolding does not conjure capability from nowhere; it spends more inference on verification and replanning to recover progress that would otherwise be lost.
Which means "better scaffolding always wins" is the wrong lesson. The right one is that verification has a price, and on long tasks that price is often worth paying because the alternative is finishing confidently wrong.
Where a failure actually belongs
The companion paper, Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures, makes the diagnostic point directly. It defines 41 failure modes organised around the edges between components rather than the components themselves, each tagged with a fault side indicating where the repair belongs. Independent reasoning agents recovered human labels well above chance, with the strongest judge reaching a Cohen's kappa of 0.76 - solid agreement for a subjective labelling task.
Its worked example is the one every heavy agent user will recognise. If a long coding session misses an earlier instruction because context compaction removed it, that is a harness bug: fix the compaction. If the instruction was still sitting in context and the model ignored it anyway, that is a model problem. Identical symptom, opposite fix, and until now no vocabulary for telling them apart.
Why it matters
If a harness swap is worth 29 percentage points on a benchmark, then benchmark tables comparing models are quietly comparing scaffolds too. That reframes a lot of leaderboard discourse - and it makes agent harnesses and scaffolding a first-class engineering surface rather than glue code around the interesting part.
It also connects to the day's other agent findings. SWE-Touch showed agents failing to reconcile a workspace that changed under them, which is precisely a state-verification failure. And two supporting papers reinforce the pattern: AgentStream finds self-evolution reliability varies by scenario and is non-monotonic in model strength, while ScrambleToolBench finds agents can build an accurate internal map of tool behaviour and still fail to use it when the environment shifts, falling back on exhaustive search or stale belief.
The honest caveat
Every number here is author-reported, on fresh preprints, with no independent reproduction. The comparison is also not free of confounds - the baseline is Claude Code, itself a moving target, and "same backbone, different harness" is only as clean as the two harnesses' matched budgets. The result that would settle it is a third party running both harnesses on the same model at matched token spend. Nobody has published that yet.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)