Harness Engineering Is the New Bottleneck, Not the Model

Eleven coding agents, about four million lines of source, and what a .NET developer should take from it

For most of the last two years, my first question about any new coding agent was “which model is it running?” I asked it out of habit, the way you ask which engine is in a car before you ask anything else. This week I caught myself asking a different question, and I think that small shift is the whole story of late 2026. The question was: “what is wrapped around the model, and can I read it?”

I did not arrive at this from one blog post. Over the last few days, a handful of independent sources landed on the same thesis from different directions: a source-code study of eleven real coding agents, a Google whitepaper, a LangChain guide, a Y Combinator observation, and a newsletter cycle that would not stop talking about harnesses. When that many unrelated people converge, I get suspicious, so I went and checked the primary sources instead of trusting the summaries. Some claims held up well. A couple needed a caveat, and I will flag those as we go.

This is the writeup. It is opinionated, it has a few doubts in it, and it ends with a checklist you can paste into your own repo.

The paper that reads the code instead of the README

The piece that started this is an arXiv paper by Paul Barbaste, Tristan Darrigol, Germain Vu, and Tom Wiltberger, titled “Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents” (arXiv:2609.00006). The subtitle is the important part: “A Source-Code Study of Eleven Systems.”

They did not benchmark models. They read harnesses. Eleven production coding agents: Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, and OpenClaw. Roughly four million lines of Python, TypeScript, and Rust. They also analyzed a twelfth system, Omnigent, as a contrast point because it is a meta-harness.

From that pile of code they extract seven canonical subsystems that every one of these agents has some version of.

+----+-----------------------+-------------------------------------------------+
| # | Subsystem (paper) | What it owns |
+----+-----------------------+-------------------------------------------------+
| 1 | Agent Loop | Alternates inference and action. Owns stop |
| | | conditions and failure recovery |
+----+-----------------------+-------------------------------------------------+
| 2 | LLM Integration | Provider protocols, prompt assembly, caching, |
| | | thinking, routing (my shorthand: model I/O) |
+----+-----------------------+-------------------------------------------------+
| 3 | Tools and Actions | What the agent can do, file editing above all |
+----+-----------------------+-------------------------------------------------+
| 4 | Memory and Context | Rations the context window, persists knowledge |
| | | across turns and sessions (compaction lives here) |
+----+-----------------------+-------------------------------------------------+
| 5 | Safety and Permissions | What runs, what asks, what is forbidden, plus |
| | | execution isolation |
+----+-----------------------+-------------------------------------------------+
| 6 | Orchestration | Spawns and coordinates sub-agents |
+----+-----------------------+-------------------------------------------------+
| 7 | Extensibility | Config, hooks, skills, plugins, MCP |
+----+-----------------------+-------------------------------------------------+

Two findings in the abstract made me stop scrolling. The first: across those four million lines, no agent runtime imports a general-purpose agentic framework, and none of them retrieves code with vector embeddings. The field, in the authors’ words, runs on hand-rolled async loops and deterministic retrieval. If you have spent any time in the “you need a vector database and an orchestration framework” corner of the internet, that is a strange thing to read. The eleven harnesses that people actually use to ship code do not do either.

A small caveat that I think matters: this is about coding harness runtimes specifically. It does not say that embeddings or frameworks are useless for a customer support bot or a document pipeline. It says that when the job is “edit a repo safely,” the people who built the most-used tools chose grep, file reads, and a loop they wrote themselves.

The second finding: the floor is low. Mini-SWE-Agent implements all seven subsystems in roughly a hundred lines. A while loop, one template, one tool, a message list, two limits. The paper even ships a 90-line minimum-viable-harness scaffold in its recommendations section. What separates the production systems from that hundred-line skeleton is not the loop. It is safety, recovery, cost management, extensibility, and the surfaces around the loop.

Read that twice. The core of an agent is trivial. Everything that makes it trustworthy is the harness.

A few other numbers from the abstract, because they say something about where the ecosystem is heading. SKILL.md skills lead MCP in adoption across the eleven systems, nine of eleven versus eight of eleven. The authors also source-diffed the same harnesses across one quarter and report that behavioral policy is migrating out of prompt prose and into configuration. That last one matches my own experience so closely that it deserves its own section later.

Agent = Model + Harness, and the receipt that backs it up

Google’s whitepaper “The New SDLC With Vibe Coding,” by Addy Osmani, Shubham Saboo, and Sokratis Kartakis, states the thesis in a form you can put on a slide: an agent is a model plus a harness. The harness, in their framing, is the instruction files, tools and MCP servers, sandboxes, orchestration logic, guardrails, hooks, and observability.

Slogans are cheap, so what I wanted was evidence. The whitepaper points at a specific result, and it is a good one. A team moved a coding agent from outside the top 30 to the top 5 on Terminal-Bench 2.0 without changing the model. That team was LangChain, and the score went from 52.8 percent to 66.5 percent, a 13.7 point gain, by changing the prompt, the tools, and the middleware around the model.

Here is where I want to be careful, because I nearly wrote this section wrong. In the material I was given, the “top 30 to top 5” jump and the “13.7 points” gain look like two separate pieces of evidence. When I traced them back, they are one experiment: one team, one fixed model, one harness rebuild. That is still a strong result. But it is one receipt, not two, and I would rather you know that than come away thinking there are two independent replications.

Why do I still find it convincing? Because the direction matches everything else. When the model is held fixed and the score moves by 13.7 points, the harness is doing the work. If the model were the whole story, a fixed model could not move.

Three moves from LangChain’s Sydney Runkle

Sydney Runkle, a product manager at LangChain, published “How to Build a Custom Agent Harness” in June. I went in expecting a numbered three-step tutorial. It is not formatted that way, but the shape underneath is three moves, and it is the most practical framing I have seen for someone starting from zero.

Move one: start with the bare agent. A model, your tools, a system prompt, nothing else. She uses create_agent for this.

from langchain.agents import create_agent

agent = create_agent(
    model="anthropic:claude-sonnet-4-6",
    tools=tools,
    system_prompt="you are a helpful assistant..."
)

Move two: add middleware at the points in the loop where you have watched it fail. The guide names the pieces by concern. Context management with SummarizationMiddleware and ContextEditingMiddleware. Retry behavior with ToolRetryMiddleware, ModelRetryMiddleware, and ModelFallbackMiddleware. Policy with PIIMiddleware and HumanInTheLoopMiddleware. Cost control with ModelCallLimitMiddleware, ToolCallLimitMiddleware, and PromptCachingMiddleware. In plain terms: trimming, retrying, redacting, asking a human, and capping spend. Check the LangChain docs for constructor arguments, since those move between releases.

Move three: fit the harness to the task. She calls this task-harness fit, meaning how well your harness matches the actual demands of the task. A coding agent, a research agent, and a support agent should not share a middleware stack.

What I like about this is the order. You do not design the harness up front. You start bare, watch where it breaks, and add a layer for each real failure. That is also, I realized, exactly how I ended up with the harness I run today. I just did not have a name for it.

Why every startup is suddenly building one

There is a market signal here too. The Code newsletter, in its September 14 and 15 issues, reported that Garry Tan has observed nearly every non-hardware startup at a recent Y Combinator Demo Day building a domain-specific harness, and that a core reason is portability: they want to swap GPT for Claude, or the reverse, without a rewrite.

I could not find the original Demo Day remark in a primary source, so treat that framing as the newsletter’s relay of Tan’s observation, not a verbatim quote from me. What I could verify is Tan’s public line that “either you die a system of record or you live long enough to become a domain-specific harness.” Same idea, said more bluntly.

The logic holds up on its own, though, and it is the part that matters for anyone with a codebase. If the model is a swappable component behind an interface, then your durable asset is everything on your side of that interface: the tools, the permissions, the memory, the evals, the domain knowledge. The model becomes a supplier. Suppliers get compared on price and quality and replaced. Your harness does not.

I will add my own doubt here. Tan also argues that the frontier labs will want their harness to be the moat, and that the best case for consumers is model capabilities flattening and commoditizing. Both of those can be true at once, and it is not obvious to me who wins that tug of war. The labs ship their own harnesses. Claude Code is in the paper’s list of eleven. If the lab’s harness gets good enough, the independent harness has less to add. I do not know how that resolves.

Where I felt “harness, not model” firsthand

I build on .NET, mostly Microsoft Agent Framework and Microsoft.Extensions.AI, so my evidence is not from Python notebooks. It is from four specific moments where I changed nothing about the model and the system got better or worse anyway.

The failover wrapper. For about four months I maintained a hand-rolled Polly wrapper around IChatClient. It started at fifteen lines and grew a switch statement every time a provider had a bad day. When Microsoft.Extensions.AI 10.9.0 shipped OrderedFailoverChatClient, it replaced about two hundred lines of policy code. The model did not change. The reliability of the product did. That is harness work, full stop.

The streaming blind spot. The same failover feature bit me in the harness, not the model. On the one endpoint that streams tokens to the browser, a mid-stream failure meant half of one response got stitched to a full second response from the backup provider. Invalid JSON, no exception, no page. No model upgrade fixes that. A commit-delay buffer and a non-streaming path for parseable output did.

The instruction I gave eleven times. I once told a coding agent, in eleven different wordings, never to run destructive git commands without asking. It ignored me often enough that I gave up and wrote an actual pre-tool-call gate. This is the paper’s “policy migrating from prompt prose to configuration” finding, lived. Prose is a request. A hook is a rule.

The forty-one round loop. I ran an evaluator-optimizer loop with no iteration cap and watched the evaluator find increasingly pedantic objections for forty-one rounds. It cost about six dollars before I killed it. The fix was a cap of three rounds. Not a smarter model, a ceiling.

Here is the small piece of .NET I now put in front of any agent I run unattended. It is a spend cap as a DelegatingChatClient, which sits in the same pipeline as telemetry and retries.

using Microsoft.Extensions.AI;

public sealed class SpendCapChatClient(
    IChatClient inner, decimal capUsd, Func<UsageDetails, decimal> price)
    : DelegatingChatClient(inner)
{
    private readonly object _gate = new();
    private decimal _spent;

    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        EnsureUnderCap();
        var response = await base.GetResponseAsync(messages, options, cancellationToken);
        if (response.Usage is { } usage)
        {
            lock (_gate) _spent += price(usage);
        }
        return response;
    }

    public override IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        EnsureUnderCap(); // check before the stream starts; add usage accounting per update if you need it
        return base.GetStreamingResponseAsync(messages, options, cancellationToken);
    }

    private void EnsureUnderCap()
    {
        lock (_gate)
        {
            if (_spent >= capUsd)
                throw new InvalidOperationException($"Spend cap of {capUsd} USD reached, stopping the run.");
        }
    }
}

And the wiring, which is also my local, no-API-bill alternative. The same harness runs against a cloud model or a local Ollama model, and nothing in it changes when I swap.

// dotnet add package OllamaSharp
// ollama pull llama3.1 && ollama serve

IChatClient baseClient = new OllamaSharp.OllamaApiClient(
    new Uri("http://localhost:11434"), "llama3.1");

// Swap the line above for your cloud IChatClient in production.
// The harness below does not change.

IChatClient client = new ChatClientBuilder(baseClient)
    .UseOpenTelemetry(sourceName: "MyAgent")
    .Use(inner => new SpendCapChatClient(
        inner,
        capUsd: 2.00m,
        price: u => (u.InputTokenCount ?? 0) / 1_000_000m * 2.50m
                  + (u.OutputTokenCount ?? 0) / 1_000_000m * 10.00m)) // illustrative prices
    .Build();

That swap is the Demo Day point in about six lines. If the model is behind IChatClient, moving from GPT to Claude to a local Llama is a one-line change, and everything that took me months to get right stays put.

Five free repos to upgrade your Claude Code harness today

If the thesis is right, the cheapest upgrade you can make is not a bigger model. It is a better harness layer, and a lot of people have open-sourced theirs. These five came from the Simplifying AI newsletter’s roundup from September 18. I checked each one against its own README, and I would still tell you to do the same, since install commands in this space change fast.

+----+------------------------+-------------------------------------------------+
| # | Repo | What it adds to your harness |
+----+------------------------+-------------------------------------------------+
| 1 | Superpowers (obra) | A full workflow: brainstorm, plan, TDD, then |
| | | sub-agent execution with two-stage review |
+----+------------------------+-------------------------------------------------+
| 2 | Karpathy guidelines | One CLAUDE.md with four rules: think before |
| | | coding, simplicity first, surgical changes, |
| | | goal-driven execution |
+----+------------------------+-------------------------------------------------+
| 3 | GStack (Garry Tan) | 23 skills and 8 tools that act as CEO, designer, |
| | | eng manager, reviewer, QA, and release engineer |
+----+------------------------+-------------------------------------------------+
| 4 | Ponytail | A minimalism skill that pushes the agent to write |
| | | less code |
+----+------------------------+-------------------------------------------------+
| 5 | Matt Pocock's skills | Skills for real engineering: grilling a plan, |
| | | TDD, bug diagnosis, code review |
+----+------------------------+-------------------------------------------------+

1. Superpowers. Run this inside Claude Code:

/plugin install superpowers@claude-plugins-official

If that marketplace is not available to you, the alternative is /plugin marketplace add obra/superpowers-marketplace followed by /plugin install superpowers@superpowers-marketplace. It enforces a red, green, refactor cycle and deletes code that was written before its tests.

2. Karpathy guidelines. The plugin route:

/plugin marketplace add forrestchang/andrej-karpathy-skills
/plugin install andrej-karpathy-skills@karpathy-skills

Or drop it straight into a repo. One warning from experience: the README’s curl -o will overwrite an existing CLAUDE.md. If you already have one, append instead:

curl -s https://raw.githubusercontent.com/forrestchang/andrej-karpathy-skills/main/CLAUDE.md >> CLAUDE.md

This one is my favorite for the price, because it is four rules in a text file and it is the purest example of “harness, not model.” Zero code, real behavioral change.

3. GStack. It needs Git and Bun 1.0 or newer:

git clone --single-branch --depth 1 https://github.com/garrytan/gstack.git ~/.claude/skills/gstack && cd ~/.claude/skills/gstack && ./setup

Then add a gstack section to your CLAUDE.md, as its README describes.

4. Ponytail. Send these as two separate prompts, one after the other:

/plugin marketplace add DietrichGebert/ponytail

/plugin install ponytail@ponytail

Now the number everyone repeats: Ponytail’s own benchmark reports about 54 percent less code on average across 12 feature tasks, and up to 94 percent on individual tasks. Here is the context I would want before quoting that. Those figures come from the project’s own benchmark, run on real Claude Code sessions in the full-stack-fastapi-template repo with Haiku 4.5, four runs per task, scored by git diff. The 94 percent case is over-built components like a date picker, where a native input does the job. On code that is already minimal, the savings approach zero.

JetBrains ran an independent test across 80 paired tasks and measured a 15.4 percent code reduction, with about 10 percent lower cost and no detectable quality difference. They were explicit that this does not refute the 54 percent on Ponytail’s own task set. It just shows the number does not generalize to every kind of work. My reading: expect a real but modest gain on average, with big wins when your agent tends to over-build. That is still a good trade for a plugin that costs nothing.

5. Matt Pocock’s skills. The README lists two routes. For Claude Code:

/plugin install mattpocock-skills

For other agents:

npx skills@latest add mattpocock/skills

Then run /setup-matt-pocock-skills once per repository, which configures your issue tracker, labels, and docs location. The /grill-me skill, which interrogates your plan before any code is written, is worth the install by itself.

One thing I would do before installing all five at once: do not. Each of these adds instructions to the context your agent reads on every turn. I have watched duplicated instruction files quietly cost me about 1,400 tokens per turn in a different harness because two files had drifted apart. Install one, run a real task, look at what it changed, then decide on the next. Harness layers have costs, and the cost of a bad one is paid on every request.

Where I am still not sure

I would be overselling if I told you the harness is the only thing that matters. A few doubts I have not resolved.

First, the strongest number in this whole debate is a single LangChain experiment on a single benchmark. It is a convincing experiment, and the direction agrees with my own projects, but “one receipt” is what it is.

Second, the paper’s own longitudinal finding worries me a little: convergence turning into imitation. If eleven harnesses are all copying each other’s design, then either that is the right design or nobody is exploring. I cannot tell from the outside.

Third, models keep absorbing harness features. Behaviors that needed a wrapper a year ago now come built in. Some of the work I do today in middleware will probably be a model feature next year. The safe bet is to invest in the parts that are yours specifically: your permissions, your evals, your domain tools, your cost limits. Those do not get absorbed, because they encode what only you know.

None of that changes the practical conclusion. When my agent misbehaves now, I check the harness first, and I am right far more often than when I blame the model.

A checklist for auditing your own agent’s harness

Copy this into an issue or a markdown file and go through it against your own agent. It follows the paper’s seven subsystems, so a gap tells you exactly which layer is weak.

HARNESS AUDIT (score each item: yes / no / not applicable)

LOOP
[] There is a hard cap on iterations or rounds (not just "until done")
[] There is a wall-clock timeout for a single run
[] Failure of a tool call is handled explicitly (retry, skip, or stop)

MODEL I/O
[] The model sits behind an interface, so I can swap providers in one line
[] I have run the same task on a second model or a local one this quarter
[] Streaming and non-streaming paths both have retry/failover behavior I tested
[] Prompt caching is on where the provider supports it

TOOLS
[] Every tool has a narrow, single purpose and a clear name
[] Tool output is size-limited before it enters the context
[] File edits go through one well-tested mechanism, not ad hoc string replace

MEMORY AND CONTEXT
[] I know what happens when the context window fills (trim, summarize, fail)
[] Persistent memory entries can expire or be corrected
[] Instruction files are deduplicated (CLAUDE.md and AGENTS.md do not drift)

PERMISSIONS
[] Destructive actions are blocked in code (a hook or gate), not only in the prompt
[] Untrusted content (web pages, tool output) is marked as data, not instructions
[] Code execution runs in a sandbox with no network by default
[] A human approval step exists for irreversible actions

SUB-AGENTS
[] Each sub-agent has its own scope, tools, and budget
[] Parallel fan-out has a ceiling, and someone owns the stop condition

EXTENSIBILITY AND OBSERVABILITY
[] Behavior rules live in config or skills, not buried in one giant prompt
[] I can see every model call, tool call, token count, and cost per run
[] There is a per-run spend cap that actually stops the run
[] I have at least one regression eval that fails when the harness gets worse

SCORING
0-8 yes: you have a prompt, not a harness
9-16 yes: you have a harness with holes, fix loop, permissions, and cost first
17+ yes: you are ahead of most, keep evals honest

If you only fix three things this week, fix these: the iteration cap, the destructive-action gate, and the spend cap. Together they cover most of the expensive and embarrassing failures I have had, and none of them needs a better model.

Sources

• Barbaste, Darrigol, Vu, Wiltberger, “Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents: A Source-Code Study of Eleven Systems,” arXiv:2609.00006
• Osmani, Saboo, Kartakis, “The New SDLC With Vibe Coding,” Google whitepaper
• Sydney Runkle, “How to Build a Custom Agent Harness,” LangChain blog, June 2026
• JetBrains AI blog, “Ponytail Skill for Claude Code: Does It Really Cut Tokens,” July 2026
• The Code newsletter, “AI startups are betting on domain-specific agent harnesses” (Sep 14) and “You need to build a custom agent harness now” (Sep 15)
• Simplifying AI newsletter, “These 5 Free Repos Will Give Your Claude Code Superpowers” (Sep 18)
• MKWritesHere, “Harness Engineering: The Anatomy of a Modern AI Coding Agent,” and Mehmet Özel, “Google’s New SDLC Guide Draws a Hard Line Between Vibe Coding and Agentic Engineering,” both on Medium