Same model, ten times better: the interface did it Chapter 2 · Why the harness around a model matters, and why more isn't better

Imagine spending months trying to make an AI model better.
You upgrade the model. You add more training data. You experiment with prompts. You increase the token budget.
Then someone changes something much less glamorous: the interface the model uses to do its work.
And the results change dramatically.
In one experiment, GPT-4 Turbo resolved 1.31% of software issues when working with a retrieval-based setup. The same model, given a purpose-built interface, resolved 12.47%.
Same model. Same weights. Same benchmark.
Roughly 9.5 times the resolution rate.
No new model was trained. No new checkpoint was introduced. The environment changed.
That result challenges a familiar assumption in AI engineering: when an agent fails, the model must be the problem.
Sometimes it is. But sometimes, we have given a capable model a poor way to interact with the world.
And here's the surprising part: the solution isn't necessarily to build a bigger, more complicated interface.
Sometimes, the extra features are the problem.

1. The experiment that changed the question In 2023, SWE-bench posed a difficult question: could language models resolve real GitHub issues—not merely generate a function from a description, but modify an actual codebase so the maintainers' tests passed? Early results were discouraging. Claude 2 resolved just 1.96% of the benchmark. The obvious conclusion was that the models weren't capable enough yet. Then, in 2024, researchers introduced SWE-agent and asked a different question. What if the model wasn't the only thing that needed improving? Instead of training a new model, they designed an Agent–Computer Interface: a more deliberate way for the model to inspect files, search code, edit changes, and use feedback from its environment. The result was striking. • Retrieval-based setup with GPT-4 Turbo: 1.31% • SWE-agent with GPT-4 Turbo: 12.47% On the SWE-bench Lite split, the comparison was also revealing: retrieval achieved 2.67%, a shell-only agent with a demonstration reached 11.00%, and SWE-agent's purpose-built interface reached 18.00%. These are results from different benchmark splits, so the percentages should not be mixed as though they were one experiment. But within each comparison, the pattern is clear: the way an agent interacts with its environment can materially affect what it accomplishes. The interface wasn't just plumbing around the model. It was part of the system's capability.

Figure 2.1: Hold the model still, and the interface alone can move the result dramatically.
The deeper lesson isn't that SWE-agent's particular interface is universally best. The lesson is that an agent's performance depends on the fit between its model and its environment.
Even the ranking of models changed across setups: Claude 3 Opus performed better than GPT-4 Turbo in the retrieval comparison, but worse inside the SWE-agent interface. A stronger model doesn't automatically benefit more from every interface.
That makes interface design an engineering problem—not a checklist of features to accumulate.

1. Four dials. Four opportunities to get it wrong. The researchers didn't stop after reporting the headline result. They removed or changed parts of their own interface to discover which features actually helped. This is where the story gets interesting. Dial one: How much of a file should the agent see? You might assume that showing the entire file gives an agent more useful context. The experiment said otherwise. • A 100-line file window achieved 18.0%. • A 30-line window achieved 14.3%. • The entire file achieved 12.7%. More context wasn't automatically better. The largest view performed worse than the focused window. Dial two: How much history should stay in context? Again, more wasn't better. • Keeping the last five observations achieved 18.0%. • Keeping the full history achieved 15.0%. A longer history can preserve useful information, but it can also bury the details that matter now. Dial three: Should the agent have an edit command? A dedicated edit command improved performance. • Without the edit command: 10.3%. • With the edit command: 15.0%. • With the edit command plus a syntax-error lint guardrail: 18.0%. The guardrail mattered because it gave the system a way to catch a particular class of mistake before moving forward. Dial four: Should the agent search? Search sounds like an obvious improvement. Why make an agent work without it? Yet the results were: • No search: 15.7%. • Iterative search: 12.0%. • Summarised search: 18.0%. Iterative search performed worse than having no search tool at all. Why? A tool can create new failure modes. An agent may inspect too many results, keep following unproductive leads, or spend its effort exploring rather than fixing the issue. The right tool is not necessarily the tool with the most capabilities. It's the tool that helps the agent make better decisions. Across these experiments, the lesson is the same: every dial has to be measured.

Figure 2.2: More isn't better. File visibility, history, editing, and search all have their own sweet spots.

1. Failure is often misdirection, not a shortage of resources There's another useful finding hidden in the experiment. Across 2,294 task instances, 1,185 trajectories—51.7%—contained at least one failed edit. A failed edit is not just a small inconvenience. It can change the rest of the agent's trajectory. When the agent was making a single edit attempt, eventual success was 90.5%. After a failed edit, that figure dropped to 57.2%. One mistake can send the agent down a much less productive path. This helps explain why simply giving an agent more time or more tokens may not solve its problems. The chapter reports that successful runs tended to finish faster and cost less than failed ones. 93% of resolved instances were submitted before the budget was exhausted. If most successful runs finish within budget, then increasing the budget may not address the main source of failure. More resources help when the agent is genuinely short of resources. But if the agent is heading in the wrong direction, more resources can simply let it travel further down the wrong path. That distinction matters when deciding what to improve. Before increasing a token budget, inspect the trajectory. Did the agent run out of time—or did it lose its way?
2. The counterintuitive twist: simpler agents can win If interfaces matter so much, shouldn't we keep adding features? No. The chapter presents two important counterexamples. Agentless replaced autonomous exploration with a fixed three-phase pipeline:
3. Localise the relevant code.
4. Repair the issue.
5. Validate the result. No autonomous planning loop. No ongoing tool-selection process. On SWE-bench Lite, it achieved 27.33% at an average reported cost of $0.34 per attempt. The chapter notes that a later result reached 32.00%. Then there's mini-swe-agent: approximately 100 lines of Python, using bash as its tool, with a simple execution loop. The project reports a score above 74% on SWE-bench Verified. These results come from different setups, benchmark versions, and evaluation conditions. They should not be read as a perfectly controlled, apples-to-apples leaderboard. Their significance is that they challenge the assumption that sophisticated scaffolding is always necessary. A fixed pipeline can outperform an autonomous loop. A small agent can achieve impressive results without a large collection of specialised tools. So we have three lessons that seem contradictory: • A purpose-built interface can dramatically improve a fixed model's performance. • Extra interface features can make that performance worse. • A much simpler architecture can outperform a more elaborate one. They become easier to reconcile when we stop asking, "How many features does this agent have?" and start asking, "Which limitations does each feature actually overcome?"
6. The uncomfortable benchmark caveat There's one more detail that shouldn't be hidden behind the impressive numbers. SWE-bench results are not a guarantee of performance on arbitrary real-world software tasks. The chapter discusses research identifying several problems in the benchmark: • 32.67% of successful patches in the SWE-Bench+ study involved solution leakage—the fix appeared in the issue report or its comments. • 31.08% of passing patches were considered suspicious because weak tests might not distinguish correct fixes from plausible ones. • A separate differential-testing study found that 7.8% of patches counted as correct failed the developers' own test suite. After problematic instances were filtered, SWE-agent with GPT-4 fell from 12.47% to 3.97%. That's a substantial correction. And it changes how we should talk about the original result. We shouldn't claim that SWE-agent resolves 12.47% of real-world GitHub issues in general. Instead, the more defensible finding is this: on the same benchmark tasks, changing the interface produced a large relative improvement while the model was held constant. If contamination affects both sides of a comparison similarly, the absolute scores may be misleading while the relative comparison can still be informative. This distinction is especially useful for teams evaluating their own agents. You don't need a perfect benchmark to learn whether a change helps your system. You need a fair comparison: run the old and new setups against the same tasks, using the same success criteria. The result tells you whether your change helped your system—not whether it will work everywhere.
7. The harness should shrink as the model improves This brings us to the most useful principle in the chapter. A harness is often built to compensate for what a model cannot reliably do. A rule may prevent a recurring mistake. A specialised tool may make an awkward operation easier. A context-management strategy may stop important information from being buried. Those additions can be valuable. But models improve. The weakness that justified a rule today may disappear in a later model. If the harness never changes, it becomes a museum of old limitations. The chapter's prescription is simple: A mature harness gets smaller when the model improves. This doesn't mean removing every control. It means distinguishing between two kinds of components: • Guides: instructions or specialised mechanisms that compensate for a model's current limitations. These should be retested as models improve. • Sensors: tests, validation, and other mechanisms that reveal whether the work is actually correct. These preserve contact with reality. A guide can become unnecessary. A reliable test remains useful because it checks the outcome rather than assuming the model will get it right. The practical habit is to record why each piece of your harness exists. What failure was it added to prevent? Does that failure still happen with the current model? Can you measure the difference when the component is removed? Don't keep a feature merely because it once helped. And don't remove a guardrail merely because the model sounds more confident.

Figure 2.3: Harness complexity should track the model's remaining weaknesses—not accumulate indefinitely.
Try this: audit one part of your harness
Choose one piece of your agent setup: a long system instruction, a special search tool, a context rule, or an editing guardrail.
Write down the specific failure it was introduced to prevent.
Then ask three questions:

1. Can I still observe that failure with my current model?
2. Do I have a repeatable test that measures whether this component helps?
3. What happens if I remove it and run the same tasks again? If you cannot explain what a component protects against, you probably don't know whether it is helping. Test it before adding another layer. The takeaway The biggest improvement to an AI agent may not come from changing the model. It may come from changing how the model works. But the answer isn't to build the most elaborate harness possible. It is to design an environment that fits the model, measure each component, preserve reliable validation, and remove compensations that no longer earn their place. The real engineering skill is not adding more. It's knowing what to add, what to measure, and when to take something away.

Read the full chapter:
Chapter 2 — The Scaffolding Insight
Next Post: Chapter 3 — The Productivity Paradox
Your turn: What's the most elaborate part of your agent setup—and has anyone measured whether it actually helps?
If you're following this series on building software with AI agents, subscribe to get the next chapter in your inbox.