Harness Engineering: The Part of the Agent That's Actually Yours

Rohan Jain — Sat, 29 Aug 2026 23:43:24 +0000

Two years of building agents, and almost every fix that held was outside the model.

My agent did something dumb, so I went looking at the model. Benchmarks, leaderboard position, public opinion on Twitter.

What actually fixed things was boring: a reranker, a retry ladder, a state schema, a gate in CI. Code I wrote around the model, not the model itself.

Training, fine-tuning, RLHF, DPO: all of that is GPU, time and data intensive, which keeps it with big companies and frontier labs, so I built agents instead. Once I'd exhausted the "just get a better model" instinct, I moved on to swapping checkpoints, bumping the context window, rewriting system prompts. Those helped, but the gains were small and they didn't hold. Then I started fixing the scaffolding, and things stayed fixed.

Models are like people

Having worked at a financial institution, I'd seen firsthand the mistakes people make. No matter their tenure or subject knowledge, the mistakes were usually silly ones, in steps they had performed flawlessly a hundred times before. The causes varied: exhaustion, a slip of the finger, skipping a basic step.

I built controls and guardrails to eliminate these many times over, learnt that no matter how good someone is, they will still make mistakes, and it's my job to make sure there are guardrails to catch them and redirect them onto the right path.

Some of you have probably already guessed the parallel. Models fail for different reasons than people do, not fatigue, but ambiguity, distribution shift, and no memory of the hundred times they got it right. But the remedy is identical. You want the best operator you can get, and that alone does not guarantee fail-safe operation. You need crash barriers that stop the thing from doing something hazardous.

I eventually built those same loan servicing controls into an agent, a multi-agent system for syndicated loan documents that ingests a Credit Agreement or a Notice PDF and executes the lifecycle action end to end. It shows up a few times below, because it's my clearest example of a harness where a mistake means a real drawdown against a real facility.

And as a bonus: if your harness is strong enough, a cheaper model can do the same work.

Agent = Model + Harness

The definition that stuck with me is from Vivek Trivedy's write-up at LangChain:

Agent = Model + Harness

The first part is usually not in our hands. So let's talk about the second.

The harness is everything that isn't the weights. Retrieval, tool schemas, the control loop, the sandbox, what persists between turns, what gets thrown away when you run out of room. The nice thing about the early days of something new is that you can put almost anything under its umbrella without much scrutiny, which is exactly what I've been doing.

Hold the model fixed and the numbers still move

Between March and April 2026, Claude Code got noticeably worse and nobody outside the company could say why. Anthropic's postmortem named three changes, none to the weights and none to the API. My favourite is the smallest: a system prompt line capping text between tool calls at 25 words. That text is the model's scratchpad, so capping it shortened the thinking along with it. Their own ablations measured roughly a 3% drop in coding quality from that one line.

The research agrees. In a controlled comparison, changing harness at a fixed model moves SWE-bench pass@1 by 8.5 to 13 points, while changing model at a fixed harness moves it 2.5 to 5. The within-model harness range is bigger than the within-harness model range: a strange sentence to write and a hard one to unsee. That holds for big harness swaps, though. Adjacent leaderboard rows mostly differ by noise.

Five places my own harness was the problem

Birgitta Böckeler's harness engineering write-up is the framing I use now. Guides run before the agent acts and raise the odds it gets things right first time. Sensors run after and let it self-correct. Each can be computational (deterministic, cheap enough to run on every change) or inferential (an LLM judging). Almost every bug I shipped was a missing guide or a missing sensor.

1. What it reads before it writes

In the loan servicing agent, a Notice gets validated against four Credit Agreement clauses: conditions precedent, permitted purpose, notice mechanics, repayment conditions. It kept validating against the wrong one and approving things it shouldn't have. I blamed the model's reasoning. Retrieval was handing it the wrong text.

A repayment conditions clause and a conditions precedent clause are near identical in embedding space and completely different in effect, and because Credit Agreements are drafted from templates the whole corpus is written in one voice. That is exactly where semantic similarity stops discriminating. The fix was hybrid retrieval: full text search alongside vectors, so the defined terms a clause turns on carry weight instead of being averaged away.

GraphBench is my controlled version of the claim. Two pipelines, same FAISS lookup, same Neo4j subgraph, same Mistral-7B, only the triple selection differs. Across 500 HotpotQA questions that alone moved exact match from 3.2% to 5.0%. The agent's ceiling is set by what reaches its context; a better model reasons better over the wrong document and still gives you a wrong answer.

Computational guide.

2. What it does when it's wrong

My Self-Healing Code Agent would hit a failing test, try a fix, fail again, then keep trying the same shape of fix until it ran out of budget. It had no failure ladder. Retry was the entire recovery vocabulary.

What replaced it: confidence-aware routing, so a low-confidence diagnosis goes to a blind retry rather than a targeted repair, because a confident wrong theory is worse than no theory. Parallel repair strategies via LangGraph's Send(), picking a winner by tournament. A ReAct debugger that tests hypotheses and checks whether repairs are converging. And a stop: max iterations, then a human.

On the benchmark in the repo, eight tasks, llama3 via Ollama on local CPU: first-pass success 3 of 8, or 37%. After the repair loop, 7 of 8, or 87%. Same model throughout. The 50 point gap is the loop.

A retry is not a recovery policy. Diagnose, route on confidence, fan out, escalate, hard stop is a recovery policy.

Computational sensor, plus a control policy.

3. How hard it thinks, and where

I assumed the rule was cheap model for easy steps, expensive model for hard ones. The Self-Healing agent inverted that. The cheap local 3B runs the generator, the thing that writes and repairs the code. Claude runs adversarial test generation, root cause analysis and the critic.

That looks backwards until you notice what each role does. Generation is recoverable, because everything downstream exists to catch its mistakes. Diagnosis is not: a wrong diagnosis sends the next iteration in the wrong direction and nothing catches it. Spend where a mistake is unrecoverable, not where it's most likely. The loan servicing agent applies the same rule across three tiers: Sonnet orchestrates and owns routing, GPT-4o-mini extracts against fixed templates, Haiku validates and executes.

Set this wrong and you get a system that feels dumber with no error anywhere in your logs. Anthropic ran that wrong default across an entire product for a month.

A guide, and a load-bearing one.

4. What it runs without asking

The Self-Healing agent executes generated code, which is where I was most naive. I started with permission checks inside the agent loop. That is not a security boundary: a check inside the harness only sees the call the agent chose to make, and anything reached another way never touches it. The boundary that holds is the one the operating system puts on the process. Mine is a Python subprocess with rlimit caps, which bounds CPU and memory but is not deny-by-default on filesystem and network. Naming that gap is more useful than pretending I closed it.

Gates sit on top of the boundary, not instead of it. The gate answers should this happen. The sandbox answers what is even possible. In the loan servicing agent every gate has a threshold rather than a vibe:

Any field below 0.75 confidence stops and shows the reviewer the inference next to its source text.
Interest is recomputed in Python on ACT/360 and checked against the notice with a $30 tolerance band. The model doesn't get to do the arithmetic.
Risk escalation to High stops for a human. Every drawdown stops, unconditionally.
The transaction log is append-only. No deletes anywhere.

That last one is a constraint rather than a sensor: the control that keeps working when everything else is broken.

Constraints and gates.

5. Whether you would even notice

Read what Anthropic says about detection. The caching bug got through human code review, automated review, unit tests, end-to-end tests and internal dogfooding, and their own evals didn't reproduce it at first. Users reported degradation in early March; the last fix landed April 20. These are people with more eval infrastructure than any of us. The bug was invisible because it lived in the seam between context management, the API and extended thinking, and nobody had a sensor pointed at that seam.

Three things I do because of this. Harness config goes in version control, separately from code, so git log prompts/ is the first place I look when behavior moves. The agent never grades itself alone: spec-blind tests before the code exists, adversarial tests after, and self-reported and reference-validated kept as separate columns. Something survives the run: an append-only row per notice recording which gates fired and what the human decided, so drift has a record to diff against instead of a vague sense that things used to be better. Anthropic's users had that vague sense for six weeks. The piece I'm still building is the merge gate.

Inferential sensor. Expensive, probabilistic, worth it.

The benchmark trap

I'm currently building ToolSmith, where I fine-tune a small 4B model to get better at calling tools. It will probably beat a bigger general-purpose model on my own test set. That won't mean much. A model tuned for one sandbox winning inside that sandbox is the expected result, not proof it's good in general.

Even a proper score on a public tool-calling benchmark wouldn't tell me what I need to know. Those benchmarks check whether the model writes a correctly formatted tool call. They don't check what happens when your harness loads forty tool definitions at startup and fills the context before the user has typed anything. They don't check what the loop does when the call fails.

A benchmark measures the model. Your agent is the model plus everything around it. The number is still worth having. It just won't tell you the five things above.

What I'd tell myself a year ago

When output degrades and you didn't change the model, the harness moved. Read the changelog before you rewrite the prompt.

Write the failure ladder before the happy path. Retry is not on it.

Spend your expensive model where a mistake is unrecoverable, not where it's most likely.

Put your harness config under eval, or you'll find out it moved when a user tells you.

None of this is free: the harness moves cost from a token bill to an engineering one, which is a trade you might want and might not.

And the budget part: a 3B model went from 37% to 87% on my benchmark without a single weight changing. The models will keep getting better on their own schedule. The environment around them is the part that's yours.

If you're building agents and you've got a harness failure story, I'd like to hear it. Which of these five cost you the most?

Sources and further reading

Vivek Trivedy, The Anatomy of an Agent Harness, LangChain
Birgitta Böckeler, Harness engineering for coding agent users, martinfowler.com
Anthropic, An update on recent Claude Code quality reports
Ryan Lopopolo, Harness engineering: leveraging Codex in an agent-first world, OpenAI
Zhang et al., Stop Comparing LLM Agents Without Disclosing the Harness, arXiv
Terminal-Bench leaderboard rankings: luck or skill?, LessWrong
Lilian Weng, Harness Engineering for Self-Improvement

My projects referenced

Loan Servicing Agent: multi-agent system for syndicated loan documents. LangGraph, Neon Postgres with pgvector, confidence-gated human review, ACT/360 validation, append-only audit log.
Self-Healing Code Agent: generates Python, adversarially tests it, diagnoses failures via a ReAct debugger and repairs iteratively. 37% to 87% across 8 benchmark tasks. Live demo on HuggingFace Spaces.
GraphBench: open-source library benchmarking GraphRAG against GNN-RAG on multi-hop QA, with the scaffold held fixed. pip install graphbench-kg.
ToolSmith: post-training Qwen3-4B for tool-calling via LoRA SFT and step-level GRPO with verifiable sandbox rewards. Ships an MCP server and a Gradio Space.

Originally published on Medium.

DEV Community: Rohan Jain

Harness Engineering: The Part of the Agent That's Actually Yours

Models are like people

Agent = Model + Harness

Hold the model fixed and the numbers still move

Five places my own harness was the problem

1. What it reads before it writes

2. What it does when it's wrong

3. How hard it thinks, and where

4. What it runs without asking

5. Whether you would even notice

The benchmark trap

What I'd tell myself a year ago

Sources and further reading

My projects referenced