A nine-author study this week pulled a coding agent apart into its components and measured each one across 176 configurations. The findings are less exciting than any vendor slide and more useful than all of them, and they hand the buyer four questions no benchmark answers.
The number on the slide is a car, and you are being sold an engine
Every coding-agent pitch you have seen this year has a number on it. SWE-Bench Verified, some
percentage, up and to the right, usually next to a model name. The implication is that the number
belongs to the model, and that if you buy the model you get the number.
A commenter on this week's Hacker News thread about the harness study put the problem better than
the paper's abstract does. If Car A is faster than Car B, it is not necessarily the engine. It could
be the tyres, the gearbox, the weight, the driver. A coding agent is a car. The model is the engine.
The harness, meaning the loop around the model that decides what it sees, what it can do and when
it stops, is everything else. And the number on the slide is a lap time for the whole car, measured
on a track you do not drive on.
Nine researchers at Fan et al. did the thing nobody selling these tools has an incentive to do. They
held the model fixed, held the execution loop fixed, and varied three harness components one at a
time: planning, action space, and context management. Four models, two benchmarks (SWE-Bench Verified
and Terminal-Bench 2.1), 176 matched configurations, five context-management strategies, four
context-window budgets. Then they looked at the trajectories, not just the scores, to see what each
component actually changed about how the agent behaved.
- 176 matched harness configurations
- 4 models held fixed
- 3 components varied: planning, action space, context
- 5 context-management strategies compared
What they found, translated out of the abstract
The findings are almost aggressively unglamorous, which is how you know they are worth something.
Context management, the machinery that decides what to throw away as the conversation fills up,
matters more the tighter the context budget, and most of its benefit comes from one thing: not
falling over when the window overflows. It does not make the agent smarter. It lets the agent keep
going. The strongest strategy in their comparison was the boring one, a rule-based pass that
deletes obviously stale material before any model-based summarisation runs. And the clever
addition everyone builds, making elided content recoverable so the agent can go back and fetch it,
turned out to be machinery the models rarely used and which yielded no accuracy gain.
Planning, meaning an explicit plan-first step, changes role depending on the model. For weaker
models it is an accuracy scaffold. For stronger ones it stops helping accuracy and becomes a cost
saver, with a small decrease in success rate as the price. Another commenter connected this to
something in the Claude Code changelog: the built-in todo and task-tracking tools were switched off
by default on the newest model generations. The vendor, in other words, appears to have measured the
same thing.
Action space, meaning whether the agent gets a menu of predefined tools or just a shell, splits the
same way. Predefined tools raise success rates for models that are weak at driving bash. Models that
are good at bash do better and cheaper with bash alone, most clearly on command-line-shaped tasks.
The paper does not define "bash-capable" crisply, which a commenter rightly flagged, but the
direction is unambiguous: every tool you add beyond the shell is a claim that needs testing, not a
free improvement.
What each harness component does, by model strength, per the study's abstract. Read the row for the model you actually run.
The trajectory analysis is the part a buyer should care about most. Context management extended
how long the agent could keep working without substantially changing what it did. Planning changed
where trajectories stopped. Action space changed the granularity at which code got written. None of
the three made the engine better. They changed the gearbox, the tyres and the driver, and the lap
time moved accordingly.
Why this matters more than the leaderboard
Put the study next to what the practitioners were saying this week and a picture forms.
Theo Browne's video argued, with some heat, that people who cannot feel the difference between
model generations are prompting badly, and his most useful idea was a picture of the distribution:
every model has a ceiling, which is what the demos show, and a floor, which is what you hit at two
in the afternoon on a boring task. Frontier models mostly raise the floor. ThePrimeagen, the same
week, left a current-generation model on a trivial colour bug, came back forty minutes later, and
found it had spent 330 million tokens and 118 dollars reading the same file over and over. That is
a floor.
A benchmark number is a ceiling measurement of one car on one track. It tells you nothing about the
floor, and the floor is where your money goes. And the study tells you the floor is mostly a
harness property: whether the loop notices it is stuck, whether the context gets cleaned before it
overflows, whether the agent has a shell or a menu, whether there is a plan and whether the plan is
worth its cost for the model you actually run.
The four questions to ask instead
So when the next vendor slide arrives, the number is not the question. These are.
Which harness produced this number, and can I see it? If the answer is "our proprietary agent
runtime", you are buying a car and being told the horsepower. Ask for the loop: what the model
sees, what it can do, how context is managed, when it stops.
What does it do when it is stuck? Ask for the failure trajectories, not the success ones. A good
vendor has them and is proud of them. Ask specifically what happens at context overflow and what
happens after the tenth identical tool call. The study says that is where the benefit of the whole
context-management apparatus lives.
What does it cost per success at the floor, not per success on the benchmark? Your workload is not
SWE-Bench. Take twenty of your own boring tasks, run them, and divide dollars by successes. Include
the runs you killed. The study's own finding, that stronger models do better and cheaper with fewer
tools, is a hypothesis you can test on your codebase in an afternoon.
Which components would I turn off? This is the question the paper actually equips you to ask.
If your model is strong, the plan step might be a cost centre. If it is bash-capable, the tool menu
might be a drag. If your context budget is generous, the elaborate recoverable-summary system might
be doing nothing. A harness with fewer components that you understand beats one with more that you
do not, for the same reason a car you can service beats one you cannot.
A benchmark measures the whole car once, at its ceiling. A buyer needs the floor, per component, on their own track.
The afternoon test
Since the third question is the one that matters and the one a vendor cannot answer for you, here is
the protocol we use, which costs an afternoon and a modest API bill.
Pick twenty tasks from your own recent history. Not the interesting ones. The ones that came in as
tickets and got done without anyone remembering them: a null check, a copy change, a small
migration, a flaky test, a dependency bump that broke something. Ten of them should be the kind of
task an intern would finish before lunch. Ten should be the kind that looks trivial and turns out to
touch four files. Write each one down as a ticket, the way it was actually written, with the same
missing context.
Run each task through the candidate agent with its default harness, on a clean checkout, with a
fixed budget of tokens or minutes, and walk away. Do not steer. Steering is what the demo does and
it is what you will not have time to do at volume. When the budget runs out or the agent stops,
record three things: did it produce a change a reviewer would accept, how many tokens did it use,
and did it at any point loop, meaning repeat an action it had already taken with the same result.
Then divide the money by the accepted changes. That number, dollars per accepted change on your own
dull work, is the only benchmark that will predict your bill. Run the same twenty through a second
candidate and you have a comparison that no leaderboard offers. Run them again with the plan step
disabled, or the tool menu replaced with a shell, and you have reproduced the study's method on the
only codebase you care about. The loops column is the floor made visible; a candidate that looped
on three of twenty will loop on fifteen percent of your work forever, and no ceiling justifies that.
The limit of the study, stated plainly
It was run on a particular set of models that, as one commenter complained, did not include the
current frontier or the strongest open-weight options. The definitions are looser than they should
be. The benchmarks are the benchmarks, with all the training-set contamination questions those
carry. None of that changes the shape of the result, which is that the harness is a first-class
variable and the leaderboards treat it as noise.
The engine matters. Buy a good one. But you are going to spend the next year driving the car, and
the car is the part you were never shown.
Sources
- An Empirical Study of Harness Design for Coding Agents (Fan et al., arXiv 2609.20804) (HN, 217 points, 18 Sep 2026; all study numbers below are from the abstract)
- HN thread on the study (the car analogy and the Claude Code todo-tools observation come from commenters there)
- Please stop using stupid models (t3.gg, video) (18 Sep 2026; the ceiling-and-floor framing)
- How are they losing so bad (ThePrimeagen, video) (15 Sep 2026; the 330-million-token session, as reported on stream)
Originally published on the Levelbrook playbook. Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.
Top comments (0)