DEV Community: Ian Johnson

The Governance Gap Is a Boundary Problem

Ian Johnson — Wed, 08 Jul 2026 15:24:45 +0000

One of my early factory prototypes had a clean audit trail. Every action an agent took, recorded. Who did it, what it touched, and when it ran. I was proud of it. Then I watched an agent remove a status enum from a database column. Not a value I wanted changed. It was a status that other records depended on. The audit trail caught it. Logged the delete, the actor, the timestamp. Caught it perfectly, and caught it after the status was already gone.

That’s the moment the “governance gap” stopped being a phrase I’d read in other people’s posts and became something I’d built by accident. I had governance you could read. I didn’t have governance that could say no.

This is what I learned about that gap by building a tool to close it. The tool is open-refinery, a control plane that governs how AI-driven work reaches repos, models, and tools. The lesson is simpler than the tool. The gap everyone’s naming isn’t a missing policy. The gap is that nothing makes an agent respect it. And a policy nothing enforces is a suggestion.

The gap is between the policy and the work

People use “governance gap” to name the space between the policy you wrote and the agent doing the work. You have a policy. The agent runs. The policy and the run never meet, so nothing makes the agent respect it.

The common fix is to add something. A policy document, an approval step, or a content filter. Each of these is a thing you write down. None of them is a thing that stops an action.

Here’s the distinction that took me three prototypes to see. An audit log observes: it tells you what happened. Governance has to gate. It must stop the thing before it happens. My prototype with the clean audit trail was all observation. When the agent deleted that status, nothing stood between the intent and the delete. The log wrote the event down a moment too late to matter.

Once you see governance as gating instead of recording, the question changes. It’s no longer “what policy should we write.” It’s “where does the policy get enforced before the action does.” That question has a location in it. Most governance talk doesn’t.

A gate has to live somewhere

A gate is a control that runs before an action and can refuse it. To run before the action, it has to sit between the thing that wants to act and the thing being acted on. It needs a seam. In most agent setups, nobody drew the seam.

So the first real decision in open-refinery was to split two things that usually run together.

The harness does the work. It’s the agents, the prompts, the memory, the tools, the orchestration. It lives in your application. It decides what to try.

The platform governs how work reaches a target — the repos, the models, the tools the harness wants to touch. The platform does none of the work. open-refinery executes nothing. It sits on the seam and decides whether an action reaches its target.

The harness never touches the target directly. Every action passes a gate that can say no, and the platform owns that gate.

Draw that line and the gate has a home. The harness asks to do something. The platform checks the policy. The action reaches the target, or it doesn’t. Skip the line and your policy floats free of the work. That was the prototype I started with: the policy lived in one place, the actions happened in another, and nothing connected them.

None of enforce-before-execute is new. RBAC, IAM, and Kubernetes admission controllers are all examples. What’s new is who’s on the other side of the gate. A service either has permission or it doesn’t. An agent reads your policy in prose and decides how to apply it. The boundary is what lets you stop trusting that interpretation: the policy gets enforced on the seam, not left to the agent to honor.

The boundary is the whole idea. Everything else here is a consequence of it.

Make the loop boring on purpose

I wanted the platform to run without a human watching. Lights-out. I also wanted every action fully auditable. Those two goals fight each other the moment the system makes a choice you can’t explain.

You can’t audit “the model decided to.” You can audit a function that takes recorded inputs and returns a recorded output. So the core loops in open-refinery are plain code, not agent choices. The policy is encoded there, deterministically, where the agent can’t reinterpret it away. There are two loops: transition, which moves a work item from one stage to the next, and produce, which runs an action against a target. The agent fills in the inputs. The code decides what happens.

This audits the mechanism, not the judgment. Which input the agent picks is still the agent’s call. Determinism doesn’t make the decision safe. It makes the decision inspectable, and it puts a gate between the decision and its effect.

Every action runs the same sequence:

authorize -> enforce policy -> execute -> record

The order is important. Authorize before the run, or you can’t prevent it. Record after the run succeeds, or a crash mid-action loses the event.

The check itself is small:

decision = enforce(role, action, resource, mode="strict")

if decision.denied:
    raise PolicyDenied(decision.reason)

A policy names who may do what to which resource. In strict mode, nothing’s allowed unless a policy allows it. In audit mode, everything’s allowed unless a policy denies it. Strict mode fails closed: if nothing explicitly allows an action, it’s denied. Audit mode fails open: an unknown action runs. Which one you pick is the whole difference between a gate and a log. Audit mode is where you learn what your agents do. Strict mode is where you decide what they may do. That enum from the top of the post? A denied action under a strict policy on the status resource. One line, and it never happens.

Human oversight is a dial, not a switch

One of the things I got wrong early was treating human-in-the-loop as on or off. Either a person approved every action, which didn’t scale, or nobody did, which was the prototype that deleted the status. Both are bad. The useful setting sits between them, and it should be a setting.

open-refinery has five oversight levels, set per process. At L0, manual, every action needs a human to approve it. L1 and L2 loosen the grip: assisted, then supervised, where only the gated steps wait for a person. L3 is autonomous, no approval. L4 is dark, fully lights-out. You pick the level for a process based on how much you trust it — not for the whole system at once.

What stays the same across all five is the machinery. The gates run the same way at every level, and the records share one schema. What changes is how many approval events the trail carries, and how many actions wait for a person before they execute. Because the loop is deterministic, moving the dial doesn’t change what gets checked or how it’s recorded; only who has to say yes, and how often.

The boundary shows up again here. The dial sits on the platform, not in the harness. You can turn a process from supervised to dark without touching a line of agent code, because the approval step lives on the seam.

How to tell whether a policy is real

Once the platform could gate actions, I needed to know which policies were actually doing anything. A policy you wrote and a policy an agent respects are not the same policy.

open-refinery models this with claims. A governance claim is a statement about a repo: agents may not delete production data, all migrations get reviewed. Each claim can be backed two ways. An instruction states the intent. A gate enforces it. A claim is covered only when it has both.

That gives three states, and the middle one is the whole problem.

Instruction and gate: covered. The policy is written, and something stops violations.

Instruction, no gate: partial. You wrote the policy. Nothing enforces it. This is the governance gap, made countable. It was the most common state I found. It’s the one that feels safe and isn’t.

Neither instruction nor gate: an imitation surface. It reads as governed. It isn’t.

open-refinery scores a repo on this. How many claims have a gate, and not just an instruction? I won’t pretend I have clean fleet-wide numbers to wave at you. What the model buys me is the ability to ask the question per repo instead of assuming the answer. Partial coverage isn’t a failure of intent. It’s a gap you can name, which means you can close it.

A record you can’t quietly rewrite

Gating decides what happens. Provenance decides whether you can trust the record of what happened. These are different jobs, and I discovered that after ran them together at first.

An audit log you can edit isn’t evidence. If someone can delete the record of the delete, the record is worth nothing. So the records in open-refinery are immutable, and each one chains to the one before it with a hash. Change a past event and every hash after it breaks. One function walks the chain and checks it:

verify_chain(events) # raises if any link's hash doesn't match

This doesn’t stop someone from doing something bad. It stops them from hiding that they did. I want to be honest about which one this is: tamper-evidence, not tamper-proofing. It only holds if two things are true: someone actually checks the chain, and its head is anchored somewhere the writer can’t reach. Put the log and the verifier on the same box an attacker owns, and they can rewrite an event and recompute every hash after it. A local chain raises the cost of quiet tampering. It doesn’t make it impossible.

The gate isn’t one thing in one place, either. It shows up at three seams. The transition gate checks a work item moving from one stage to the next. The invoke gate checks an executor about to run a command. And there’s a call the harness makes before it acts at all:

POST /authorize { role, action, resource, intent: "deploy to prod" }

The harness asks before it deploys. It gets back an allow, or a PolicyDenied. The policy does nothing until it sits on one of these seams. A policy with no seam is the partial-coverage case from the last section, seen from the code side.

The policy itself needs governing

This is the part the governance-gap conversation skips. If governance lives on the boundary, then changing the boundary is itself an action. So who governs that?

In my early prototypes, the policy lived in a config file. Anyone who could edit the file could change it, and there was no record of who changed what, or why. I’d built a system that gated every agent action and left the policy itself wide open. That’s not a small hole. That’s the same gap, just moved up one level.

In open-refinery, a policy change is a governed action like any other. It walks an approval chain. It lands in the same audit trail. Every version of a policy records its author, its timestamp, and what changed. Ask “who changed this policy, and when, and why,” and there’s an answer.

A policy file anyone can edit with no record isn’t governance. It’s a suggestion with a version history, at best. Governance that can’t govern itself doesn’t hold. This is the second-order problem, and it’s where the boundary earns its keep: moving the seam is an action, so it passes a gate too.

Put one gate on one policy this week

You don’t need a platform to start closing your own governance gap. You need to stop treating your audit log as a control.

Pick one policy you’ve written for your agents. A real one: in a doc, a prompt, or a README. Find where it’s enforced. If the honest answer is “it isn’t,” you’ve found a partial-coverage surface on your own system. That’s the gap, and now you can see it.

Add one gate: a check that runs before the action and can refuse it. Start with the action you’d least want an agent to take without you watching. For me that was a destructive database change. Pick yours.

Make the policy change leave a trace. Even a commit with a reason in the message is a start on second-order governance. The goal is that “who changed this policy” always has an answer.

Do those three things and you’ve moved from governance you can read to governance that can say no. That’s the whole gap, and it closes one gate at a time.

Charter and Harness: What Each One Is

Ian Johnson — Tue, 30 Jun 2026 23:47:29 +0000

It seems that every time I think I know the shape of the harness, a new fact presents itself that I have to account for. I recently wrote a book Harness Engineering. But new information has caused me to rethink my model a bit. I plan to leave the term harness alone, and tighten up the definition. In addition, I add the term charter.

What follows is a chapter I’m working on for the book to bridge the gap in my model. I’m offering it completely free because I think it’s the best definition I’ve had so far. Your feedback is very welcome.

Clone a repository that someone has tuned for agentic coding and open the .claude folder at its root. What you find is mostly prose. A CLAUDE.md that explains the project. Rule files with names like testing.md and migrations.md. A few skills, each a short markdown file describing a workflow. Maybe a settings.json wiring up a hook. Almost none of it is code that runs on its own. It is a stack of instructions, written for a reader who happens to be a model.

Now start the agent: Claude Code, or whatever the team standardized on. It reads that folder, holds it in mind, and goes to work on your repo. The folder did nothing. The agent did. There, in a single clone, sit the two things this book is about: the material you author, and the engine that acts on it. This chapter names both and says what each is made of. The next chapter draws the line between them. Here the job is to fill the two boxes.

The charter: what you author

The charter is everything you author to specify how the agent should behave in your project. That .claude folder is most of it. The word is borrowed from its plainest sense: a charter is a founding document that says how a thing is to be governed. Your repo’s charter governs the agent that enters it. You write it, you review it in pull requests, you amend it when it turns out wrong, and it governs every agent that works the repo until you change it.

A charter comes in four kinds of material, and naming them now saves arguments later.

Guides instruct before the agent acts: the CLAUDE.md, the rule files, the slash commands, the prose half of a skill. A guide is a sentence the agent reads and weighs against the situation, like prefer composition over inheritance, and justify a base class in the PR description. No graph expresses that. It is judgment, written down.

Tool definitions declare what the agent can reach for and how it is wired: the mcp.json entry, a tool’s schema, a skill’s script, or a plugin’s config. One artifact can span families: a skill’s prose is a guide, but its script a tool definition. The definition is yours and lives in the repo. The thing it points at, the installed server or the plugin binary, is external, and the next chapter sorts that seam.

Sensors tell the agent whether the work it just did is acceptable: the test files, the lint config, the type-checker settings. You authored them, so like a hook they are charter even though they run. A sensor reports; it does not refuse.

Gates refuse: the pre-commit hook, the branch-protection rule, the blocking reviewer. A gate is a sensor wired to a stop.

Guides, tool definitions, sensors, gates. If something in your charter is none of those four, it is probably not earning its place.

Most of a charter is prose, and that is the point. A rule written as a sentence covers cases no control-flow graph would, because the agent weighs it instead of branching on it. Some of a charter is code you author, a hook or a gate, and that code is still charter: you wrote it to state a standard, so it is a specification the engine runs, never the engine itself. A charter is also a living thing. It is versioned in the repo, reviewed like any change, and amended as the project teaches you where it was wrong. The rule you write today because an agent ignored a convention is a line in a document you will keep editing for as long as the repo lives.

Here is the whole of it, as a definition you can hold: the charter is your repo’s .claude folder, all of it (the rules, the hooks, the agents, the skills, the slash commands, the settings), plus the dependencies you declare: plugins, policy packs, shared skills, tools such as MCP servers. Those are two questions about a thing, not one. The families above sort by role: is it a guide, a tool definition, a sensor, a gate? Origin is the second question: did you write it by hand in the folder, or adopt it as a declared dependency? A declared dependency keeps its declaration in your repo, which is charter, while its installed body lives outside, which is not. Declaration in, source out. Either way the test does not change: you authored or adopted it to specify behavior.

If you have been in this field a while, you may have called this folder “the harness.” It is the charter. The word slipped for an understandable reason, and the next chapter traces it; for now, hold the line that the folder you author is the charter, and the engine that reads it is the harness.

The harness: the engine that applies it

The harness is the engine that applies the charter against the model. The field has a shorthand worth borrowing: an agent is a model plus a harness. The model predicts text and nothing more. The harness is everything wrapped around it to turn prediction into work, and the charter is what the harness consults while it does. So the relationship runs in a line. You author a charter; the harness applies it to drive the model; the model produces a result, a diff or a document or an answer. Charter, harness, model, result. The charter is the input you control. The harness is the only stage that acts. The next chapter takes that line apart; here it is enough to see the shape.

An engine that does this has jobs, and Part B of this book takes them one at a time. It assembles the context the model sees each turn, choosing what to load and what to drop, and invokes the tools your charter defined. The permissions you set decide which files it may touch and which commands it may run. The gates fire, the sensors report, and the loop turns with bounds and a kill switch until the work is done or the budget is spent. Every one of those is the engine at work, not a standard you wrote down. The standard is the charter; the doing is the harness.

You get most of a harness rather than build it. Claude Code is one, Cursor is another, and a LangGraph orchestration your platform team stood up is a third. The vendor ships the engine, you point it at your repo, and it reads your charter on the way in. For most teams that is the entire relationship with the harness: choose one, configure it, move on. The teams that build their own, a custom orchestrator or a queue that runs agents across a fleet, are doing harness engineering proper, and Part B of this book is written for them. Part A is for everyone, because everyone writes a charter.

A harness can even ship its own charter-like pieces: Claude Code’s built-in slash commands, a default skill, baked-in rules. Those are harness, because the vendor authored them, not you, and they do not sit in your .claude folder. Your charter can override them. Drop a fix-bug skill into your folder and the agent runs your bug workflow instead of the vendor’s default; write a rule that bends a built-in command to your process, and your version wins. The built-in is harness, your override is charter, and the agent meets the charter.

One part of the charter couples to the harness more tightly than the rest: workflow. A rule or a hook is a primitive the harness simply reads, but how work gets defined, prioritized, and completed is a process the harness has to register and run, not merely read. That is one reason a team builds its own harness, when the workflow it needs is one a stock harness cannot express. Part A’s chapter on contracts is where you define the workflow; Part B’s on orchestration is where the harness runs it.

Two words, two boxes: the charter you author, the harness that applies it. They are not the same kind of thing, and it is a mistake to treat them as one. Where exactly the line falls, why a hook you wrote is charter even though it runs, why the orchestrator is harness even though you might own it, is the next chapter. For now, hold the two boxes apart and know which one takes most of your time. For most of this book, it is the charter.

What’s in each box

Run this against your own repo.

Your charter — everything you authored or adopted to specify behavior. Most of it is the whole .claude folder; the rest is declared dependencies. By role:

guides: CLAUDE.md, rules, skills, slash commands (what instructs)
tool and agent definitions: mcp.json, tool schemas, sub-agent and plugin config (what the agent reaches for)
sensors: test files, lint and type-check config (what reports)
gates: pre-commit hooks, branch protection, required reviews (what refuses)

By origin: written by hand in the folder, or a declared dependency (plugin, policy pack, shared skill, tool such as an MCP server) whose declaration is charter and whose installed body is external.

Your harness — the engine you run, mostly not yours to write:

the coding agent (Claude Code, Cursor, Aider)
any framework or orchestrator above it (LangGraph, a custom runner)
the loop that runs work on repeat
the runtime that fires your hooks, invokes your tools, and enforces your permissions

A thing on the first list, you maintain, and it is where your reliability comes from. A thing on the second, you mostly choose. Telling which list something belongs to is the whole skill the next chapter sharpens into a single test.

The charter is the half of the work with your name on it. The rest of this book is about writing it well.

Additional Resources

Harness Engineering: Building Reliable Workflows Around Non-Deterministic Agents

Ian Johnson — Fri, 26 Jun 2026 21:27:58 +0000

Hello everyone. I just released a book about harness engineering. It’s on LeanPub. Pay what you want, starts at $10.

https://leanpub.com/harness-engineering

What does it cover?

The discipline, the taxonomy, the behavioral system, the reliability layers, rules, guides, corpus, playbooks, rule cascades, architecture as communication, tools, skills, agents, plugins, MCP, evals, delivery orchestration, continuous delivery with agents, PR as checkpoints, policy libraries for team and org extension, patterns and anti-patterns, harness debt, post-mortem, harness curation, loop engineering. Everything I have learned over the past six months with hands-on experiments with a real production codebase.

Why so cheap?

We have a problem right now with junior and mid level engineers feeling the pressure of AI. I think we have an obligation to help them — and I think that means providing a path to senior. How could they get to senior if the agent does everything for them?

We must give juniors the support they need to become seniors.

One of the goals with this book is to support these folks in their growth by showing the big picture, and then anchoring it with cited works of software engineering canon. This book gives both and has an appendix specifically for juniors with how to operate and grow in the world of agentic coding.

If you are concerned about losing your job or not being able to find a job doing software engineering because of LLMs, then this book is for you!

There’s more

The book also covers scopes from single contributor to enterprise and provides actionable advice whether you read a particular part or the whole thing. It offers benefits to anyone looking to learn how to get better results out of their coding agents, regardless of experience, role, project, team, company, or organization.

I hope it serves you well.

This just rearranged my view on harness engineering

Ian Johnson — Thu, 25 Jun 2026 23:16:14 +0000

I just read this article — I recommend you do as well.

Loop Engineering Is Just Software Engineering. We Have a Name for That.

I’ve been doing a lot of work on harnesses, but I have been focusing at the project level. I’ve only recently started to delve into LangChain/LangGraph. I built my own daemon that implements this with a shell on my local machine. It works, but it could be better. I had always thought of these as two layers and they’ve become crystal clear to me now.

When I read this, I immediately want to try it, but I do not have access to an API key yet. Also, I don’t really want to clobber my organization with API pricing. So I asked claude to build a prototype in Python given the description. I tested it, and it worked pretty well.

But if you’re already at the API point (and you probably are if you are looping), here’s the tool: https://iii.dev/

This idea is important

The single most important takeaway from this article is that we’ve already solve the problems of loop engineering with distributed systems. Instead of getting stuck in loops or overpaying, you can take advantage of the queueing pattern — dead letter queues, circuit breakers, retries, and so on.

This idea is dynamite.

It had slipped past me because I was focused on the project level of the harness and loop engineering has been mainly focusing on creating loops with agents, not how to do it. So diagrams and explanations tend to lead to synchronous or daemon sorts of solutions. In hindsight, it seems so obvious.

It also removes the orchestrator problem: the fact that different subagents about bound to a main orchestrator thread replicates Brooks’ warning for organizations in the form of agents. But with the distributed system, there is no orchestrator — that’s been reduced to state. Brilliant!

Vocabulary

It also shifts my vocabulary. I’m currently writing a book about harness engineering: https://leanpub.com/harness-engineering

I was focusing on the repository level, but now the delivery harness (as I was calling it) builds one level up. This will probably cause me to have to re-arrange my definitions, but here’s where my head is at.

Layers of the Harness

I visualize the harness like an onion. But the harness I’ve been talking about has been code, and the delivery harness is a service. That means that personal still sits at the center, wrapped by the project (or repo) harness, and then wrapped with team and org harnesses that implement policies just like software libraries, and then wrapped in the delivery or orchestration harness. I’ve used some tools like this recently, like Hivemind (https://cohen-liel.github.io/hivemind/). It’s great, and there are plenty of others to try, but this has changed my entire outlook on the type of architecture the delivery harness should use. I think he’s right — it’s a distributed system.

Thank you Mike Piccolo!

Reading the Agent Log Like a Detective

Ian Johnson — Wed, 24 Jun 2026 16:21:53 +0000

The agent shipped a broken migration last month. The PR was clean, the tests passed locally, the reviewer signed off in twelve minutes, and the deploy to staging green-lit on the first try. Production started erroring within four minutes of the merge because the migration assumed a column existed that had been renamed two weeks earlier.

The first instinct was to blame the model. Bad agent, didn’t read the docs. The second instinct was to add a rule. Always check the current schema before writing migrations. Both instincts felt productive. Neither would have fixed it.

What fixed it was reading the transcript. The agent had, in fact, looked at the schema file. The schema file in the repo was correct. It had been updated when the column was renamed. But the agent had also loaded a database/CLAUDE.md rule that described the column under its old name as the canonical reference. The rule was stale. The agent followed the rule, but the rule was wrong.

You could call this a model problem — the agent trusted a stale rule over the live schema file sitting right next to it. But that’s not the lever you can pull. The fixable cause was in the harness (the rules, skills, hooks, and tools wrapped around the model) and here it was a rule that lied. It was visible in the log if you read it forensically.

That’s what this post is about. Reading agent transcripts as evidence, not as narrative. The five failure modes that show up over and over. And the habit of treating every wrong commit as a case to investigate before reaching for a fix.

Replay, not blame

The first move when an agent ships bad code is to resist the urge to fix and move on. The fix is downstream. The question worth asking is how did the agent decide to do that. The answer lives in the transcript.

Most modern agent harnesses save full transcripts somewhere. Claude Code stores them in ~/.claude/projects//, Cursor keeps recent ones in its UI, and most CI-driven agent runs log to a build artifact. The transcript is your scene of the incident. It records what the agent loaded, what it read, what it ran, what it produced, and in what order. Everything you need to reconstruct its reasoning is in that file.

The discipline is to treat the transcript the same way an SRE treats a stack trace: not as a list of complaints, but as the most reliable evidence of what happened. The agent’s text answers in the transcript are sometimes hand-wavy. The tool calls are not. The tool calls are facts.

Read the tool calls first.

The five failure modes

Across the agent transcripts I’ve reviewed, the same five failure modes keep producing wrong-code incidents. Each one has a cause in the model but each also has a cause in the harness, and the harness is the half you can change. All five are visible in the log.

Missing context

The agent never read the file that mattered. It wrote a migration without checking the schema. It wrote a test without checking the test conventions. It added a new module without checking how other modules in the same area were structured.

How to spot it in the log: scroll back to the start of the task and look at the Read and Glob tool calls. Note which files the agent loaded. Compare to the files the change touched. The pattern is: the agent edited file X, but never read file Y, where Y was the file that defined the convention X had to match.

The fix isn’t in the model. The fix is to make sure the relevant context loads automatically. Either by giving the rule a sharper path scope (so it loads when the agent touches the area), by adding a skill that walks the agent through the right files in order, or by adding a hook that injects the relevant context at the start of tasks of that shape.

The agent did what it was told to do. The instruction set didn’t include read Y first. That’s a harness gap.

Contradicted rules

The agent followed a rule that was wrong. The rule was outdated, or it referenced a pattern the codebase no longer uses, or two rules in the harness said different things and the agent picked the one that produced the bug.

How to spot it in the log: look at the rules the agent cited in its reasoning. If the rule cited matches the code change exactly, but the change is still wrong, the rule is the bug. Cross-check the rule’s claims against the current code. Rule says use UserRepository, code uses UserService is a smoking gun.

The fix is the rule-rot audit. Rules drift faster than people think. A rule written six months ago can be confidently citing a class that was renamed in April. The agent trusts the harness; if the harness lies, the agent ships lies.

The contradicted-rules failure mode is the one I find most embarrassing because the agent did exactly what the harness said to do. The agent isn’t wrong. The harness is wrong. And the harness was wrong because I didn’t audit it.

Hallucinated APIs

The agent called a function that doesn’t exist. Used an option that isn’t supported. Imported from a module that doesn’t export what it imported.

How to spot it in the log: search the agent’s output for function names and option names, then grep the actual codebase or library for them. The mismatches are obvious. The agent might have hallucinated findOrFailWithLock when the real method is findOrFail and there’s no lock option at all.

The fix has two parts. Short-term, tighten the harness with documentation links or type signatures pointing at the real reference is the cheapest cure. Longer-term, give the agent tools instead of documentation. An MCP server that exposes the library’s actual API (so the agent can call it instead of guessing) ends the hallucination class. A type checker run as a sensor (an automated check wired into the harness that fails the run when something’s wrong) catches the hallucinations that slip through.

The recurrence pattern worth noting: hallucinations cluster around APIs the model is confidently familiar with from training but where the project has chosen to use the library differently than the common convention. The model isn’t making up the function from nothing; it’s substituting the public-API version when the project actually wraps it.

Scope creep

The agent changed more than it was asked to. The task was fix a bug in the login flow. The diff also reformatted three unrelated files, renamed a helper for clarity, and updated four imports because the rename touched them. The bug fix is correct. The reformatting is harmless. But the PR is now many times the size it should be, and the reviewer’s eye is going to slide past the actual fix.

How to spot it in the log: look at the Edit and Write tool calls. Compare the files touched against the files the task description called out. The gap is the creep.

The fix is a rule and a sensor. The rule says touch only what you must, clean up only your own mess. The sensor is a pre-PR check that fails if the diff is wider than a budget; N files, M lines, whatever your numbers happen to be.

Scope creep is one of the most common failure modes in my experience and the most invisible. It doesn’t break anything. It just makes every PR worse to review and every regression harder to track down.

Premature commit

The agent committed before the work was done. The tests pass because the agent skipped writing the failing test first. The acceptance criteria look met because the agent declared them met without verifying. The PR description claims feature X is implemented when, under the hood, feature X is a stub.

How to spot it in the log: look for the moment the agent decided the work was done. Compare against the actual evidence of doneness. Did the agent run the test it claimed proves the fix? Did the test exist before the change, or did the agent write it after seeing the change pass? Did the agent verify the acceptance criteria, or did it skip the verification step and write the PR description from memory of what the task asked?

The fix is procedural. A skill called finishing-work that walks the agent through the verification steps before allowing a commit. Perhaps a pre-commit hook that fails if the changed files don’t have at least one paired test change. An agent that has to produce a completion report (see my earlier post on post-mortems for agent runs) before declaring done.

Premature commit is the failure mode where the agent looks most productive and is least useful. The PR ships, the work isn’t finished, and the next person to touch the code finds out.

Reading the log: a worked example

Here’s a transcript fragment from a real run I investigated. The task was to add a new field to a user profile.

> User: Add a `pronouns` field to the user profile. Optional string, nullable.
[Read .claude/CLAUDE.md]
[Read app/Models/User.php]
[Glob database/migrations/*_users_*.php]
[Read database/migrations/2024_05_12_add_email_to_users.php]
[Write database/migrations/2026_06_18_add_pronouns_to_users.php]
[Edit app/Models/User.php] // added 'pronouns' to $fillable
[Bash: php artisan migrate - pretend]
[Edit app/Http/Resources/UserResource.php] // added pronouns to toArray
[Bash: vendor/bin/pest tests/Feature/UserResourceTest.php]
[Write app/Http/Controllers/UserController.php] // added validation rule

The PR shipped. The reviewer signed off. Production threw an error the next morning, from the admin panel: Undefined array key 'pronouns'. The admin user view read $user['pronouns'] straight off the resource array, and the resource never put it there.

Reading the transcript: the agent loaded the project’s CLAUDE.md, looked at the User model, looked at a recent migration as a template, and wrote a similar migration. It also added the field to the User resource and the controller validation. It did not read app/Http/Resources/AdminUserResource.php, which the admin panel uses and which lists its fields explicitly, key by key. Unlike the model’s $fillable, the admin resource doesn’t auto-pick up new columns. A field exists there only if someone names it. The agent never opened the file, so pronouns was never added, and the admin view that assumed the key blew up.

The failure mode: missing context. The harness had no rule that said user-profile fields appear in three places: User, UserResource, AdminUserResource. The agent followed the convention from the two places it knew about and shipped a partial change.

The fix wasn’t be smarter, agent. The fix was a path-scoped rule in app/Http/Resources/CLAUDE.md that named the resource files that needed to stay in sync. The rule went in. The sensor (a quick AST check that any new $fillable field appears in both resource files) went in alongside. The next pronouns-shaped task didn’t repeat the bug.

That’s the loop. The transcript pointed at the gap and the harness change closed the gap. The model wasn’t the variable.

What forensic reading teaches you about your harness

The payoff of reading transcripts isn’t fixing the specific bug. It’s the patterns that show up after twenty or thirty of them.

You start noticing which files the agent rarely reads, and asking why. You notice which rules it cites a lot (they’re load-bearing) and which it never cites (they’re dead weight). You notice the directories where transcripts go wrong most often — usually the directories with the worst seams, the least context, or the most conventions that aren’t written down.

There’s a cost the failure modes hide from each other. The fix for missing context is almost always a new rule and every rule you add is a future candidate for contradicted-rules rot. Fix enough gaps by piling on instructions and you build the exact stale-harness problem that produced failure mode two. So the audit cuts both ways: you read transcripts to find the rule you’re missing, and you read them to find the rule that’s now lying. A harness isn’t a pile of rules that only grows. The transcripts tell you which rules to add and which to retire, and the second list matters as much as the first. The transcript is the highest-resolution feedback signal the harness produces.

Two practical reading habits help.

Read at least one failed transcript a week. Not all of them. One. Pick the one that produced the most surprising failure. Read it end to end. Note the failure mode. Ask whether the harness has a gap there. The discipline isn’t to fix everything. Rather, it’s to develop the muscle.

Keep a transcript notebook. A file in the repo. Each entry: link or paste of the transcript, the failure mode (pick from the five above), the gap in the harness, the change that fixed it. After ten entries, the notebook becomes a map of where your harness is weak.

The notebook doesn’t have to be pretty. It has to exist.

The thing the model isn’t responsible for

A theme runs through every failure mode above. You can argue each one back to “the model should have known better”. But arguing about whose fault it is doesn’t change the rate. Changing the harness does. Missing context is a closable harness gap. Contradicted rules are stale rules. Hallucinated APIs are weak documentation or missing tools. Scope creep is an unset constraint. Premature commit is a missing procedure.

This isn’t a defense of the model. The model is fallible, and the next generation of models will be more capable than this one. The point is that the cheap improvements live in the harness, not in the model. You can’t retrain the model from your laptop. You can fix a rule, scope a rule, add a sensor, sharpen a tool, and write a procedure. All five are at your fingertips. All five reduce the rate of the same bug shipping again.

Forensic reading of transcripts is the practice that tells you which one to fix.

Read one transcript this week

Find the last commit your agent made that you wish it hadn’t, or the last PR that came back with a “this isn’t quite right” review. Open the transcript that produced it.

Read the tool calls first. Note which files the agent loaded and which it should have loaded. Find the rule it cited or skipped. Cross-check the agent’s claims against the actual code. Pick which of the five failure modes you’re looking at.

Then write the harness change that would have stopped it. Whether a new rule, a path-scope, a sensor, a skill or a tool. Ship it the same day.

After three or four passes, you’ll start seeing the same failure mode keep recurring. That’s the signal. The pattern is the rule waiting to be written. The transcript is just the place the pattern lives until you spot it.

The model can’t read its own transcript. You can.

What Breaks When You Skip the Harness

Ian Johnson — Tue, 23 Jun 2026 14:47:49 +0000

The agent kept running the tests, watching them go red, scrolling up to find the failure, and then running the tests again because it had already lost the output. Third time in one session. The model wasn’t the problem. The session’s memory of its own output was. I had Claude tee the test command to a log file. After that it read the log instead of re-running.

That was one fix on one project. The pattern repeats. If the model keeps producing bad code, what is actually broken?

The model isn’t the problem

Output quality is the sum of two things: the model and the surroundings.

In the teams I’ve worked with, the obsession is always the model. They upgrade. Switch tools. New IDE plugin every month. They read every benchmark. The defects keep landing in the same places.

The surroundings get almost no attention. Those surroundings are the harness: the files, rules, tools, and feedback loops that wrap the model inside a real project. A CLAUDE.md with conventions. Skills — codified workflows the agent can run by name. MCP servers. Scoping rules. A feedback.md that records what the team has corrected. Pre-commit gates.

Without those, the model is guessing.

Five things teams blame on the model, each pointing at a missing piece of the harness, each with a first move you can ship this week. Some of the bigger fixes (codifying a team’s lore, writing a useful project-specific MCP, etc.) are quarter-of-work to do properly. The first move is week-sized. The first move gets you started.

APIs that don’t exist

The model writes a call to client.users.list({ since: lastSyncAt }). The argument doesn’t exist. The real API takes a Unix timestamp on a different endpoint. Code compiles. Test fails. The engineer reads the docs. The bug gets filed as “the model hallucinated.”

Except the model didn’t hallucinate from nothing. It wrote what it learned from training data and didn’t verify. Nothing in the session told it to.

The fix is grounding. Give the model a way to read the real docs in the moment.

Two grounding moves do the work.

First: an MCP server for library docs — Context7 is one. The model fetches current docs and writes against what’s there. When it fails, it’s almost always because the rule telling the agent to fetch first wasn’t in CLAUDE.md.

Second: a project-specific MCP for internal APIs. Every team I’ve worked with has one or two services nobody got around to documenting well. An MCP that serves the OpenAPI spec — or a search over the service’s source — puts real signatures in front of the model.

A single rule in CLAUDE.md ties it together. “Fetch docs before writing library code.”

Smallest first move: add Context7. Add the rule. Try it on the next library task.

Code that doesn’t match yours

The model writes a new service. The file lands in the wrong directory. Errors throw instead of returning. The naming is camelCase in a snake_case codebase. The diff looks foreign.

The reviewer sees it and rewrites it. The next PR has the same problems.

The model has no reference for “how we do it here.” The rules live in three engineers’ heads and one out-of-date README.

Part one is a CLAUDE.md that names the project’s conventions in plain words. Where files go. How errors propagate. Naming style. Testing framework. Logger. Time library. One or two sentences each. The bare minimum.

Part two matters more: worked examples.

When we moved a service to hexagonal architecture, the agent kept writing new code on the wrong side of the ports-and-adapters boundary. The old code was still everywhere and gave it permission to keep doing what it had always done.

The model copies texture from examples better than from prose. A skill or a feature doc with one full worked example , such as a real PR, a real file, or a real test, gives it something to imitate. Three examples beat a thousand words of rules.

The catch: if the codebase itself is inconsistent (legacy module is snake_case, new code is camelCase), the model picks up the contradiction and writes both. The worked example has to come from the side of the codebase you want the agent to copy.

Smallest first move: pick the workflow your team runs most often. Write one skill for it. Drop one full example into the file.

The same bug, in every session

You correct the agent on Monday. It used the wrong logger. You tell it to use the project’s logger. It does. Friday, new task, wrong logger again.

The agent didn’t learn. The session ended. The correction went with it. The fix is a feedback loop the agent can read. Not a chat log, but a file.

Create a feedback.md. When the team corrects the agent on something that’s a rule, not a typo, write it down in three lines. Context. Mistake. Rule going forward. Tell the agent to read it before every relevant task.

The first entry in mine was about flaky E2E tests. The team’s rule was fix them or delete them. Never live with them. The agent kept patching around them instead. After the third time I corrected it I stopped repeating myself and wrote the rule down.

Who writes to it: anyone on the team. The cost of adding an entry has to be smaller than the cost of fixing the same thing twice. Otherwise nobody writes.

The promotion rule: when the same correction shows up in feedback.md three times, lift it into CLAUDE.md or into a skill. Three is arbitrary, pick a threshold and stick with it. I chose three because of the rule of three. The point is that nothing useful stays buried in the feedback file forever. Every promoted entry is one more rule the model reads at the start of the next session.

Smallest first move: create the file. Write one entry today. Add one line to CLAUDE.md that says read it first.

The knowledge stays in heads

A senior engineer is on a call, explaining why the payment service has its own retry logic that is different from the rest of the system. The new engineer nods. The agent in everyone’s terminal doesn’t, because the agent wasn’t on the call.

Two weeks later the agent writes a new payment integration. It calls the shared retry helper. It has no idea there’s a reason not to. The PR ships. The on-call gets paged.

This is tribal knowledge. The lore of the system. It lives in Slack scrollback, in senior engineers’ heads, in a Notion or Confluence page.

The fix is codification. Three kinds of files do the work.

A glossary captures domain vocabulary. The word “session” can mean three different things in one system — auth, user, analytics. The glossary picks one definition per term and points to it. One paragraph each.

A feature doc holds the why of a part of the system. Why payments has its own retry logic. Why we picked Postgres over MySQL. Why the queue uses Redis Streams and not Kafka. Short notes a new reader can scan in a minute.

Skills hold workflows. The steps to ship a database migration. How we add a feature flag. The onboarding checklist for a new service joining the monitoring stack.

The hard part of this fix is not the format. The hard part is getting the senior engineer to write the feature doc instead of explaining it on calls. That’s a social problem, not a documentation problem. Pair on the first one. Make the doc the deliverable.

A billing module on one of my projects had quiet invariants about how transaction state combined. The agent made an arithmetic change that looked clean in the diff. It would have produced wrong totals in production. The reason the change mattered wasn’t in the code. It was in the team’s heads.

Smallest first move: pick the one piece of lore most often re-explained to new engineers. Write it down as a feature doc this week. Reference it in CLAUDE.md.

Reviewers as the harness

Every PR re-litigates the same things. Naming, test coverage, error handling, conventions, and so on. Same three reviewers, same six comments. PRs take two days to merge. The reviewers are tired.

The reviewers have become the harness. Humans are doing what a script should do.

The fix is gates the machine can run.

A pre-commit hook catches the same six review comments before the diff exists. Match the gate to the team’s actual repeated comments. It could be naming-convention nits that needs a linter rule, missing-test-file nits that needs a coverage gate, or risky-file-changed nits that needs a CODEOWNERS-style script. The point is to read the team’s review history and turn the top three repeat offenders into machine checks.

A review skill or review agent reads the diff and flags the things a linter can’t see.

A code-health check (CodeScene is what I use) gives a numeric score for maintainability and blocks regressions. The score settles the argument over whether a diff makes the code worse in seconds.

The worst case I’ve watched was naming inconsistencies during a migration. Two valid styles in the codebase, the model picking both, sometimes in the same PR. We added sensors that flagged the old style in new files and ran them in CI. The drift stopped showing up in review.

The work reviewers should be doing (design, intent, what the PR is actually trying to do) gets crowded out by everything machines could catch and don’t.

Smallest first move: pick the review comment your team has left most often this month. Write one pre-commit hook for it. Push it in a PR this week.

Write three rules this week

The model isn’t the problem. The surroundings are the work.

Three things to do this week.

Open CLAUDE.md. Write three rules: the three things you’ve corrected an agent on this week. That’s the start of a harness.
Create feedback.md. Write one entry. Tell the agent to read it. The feedback loop now exists.
Pick one workflow you’ve run through an agent more than three times this month. Make it a skill. The third time you reach for it, it’s already written down.

The harness rots too. CLAUDE.md drifts as the project changes. Skills go stale. feedback.md swells to four hundred entries and turns into noise. Two skills contradict each other and the agent picks one at random. Most teams who tried a harness and quit do so at the rot stage, not the setup. Treat it like the code it wraps: audit on a schedule, delete what no longer applies, promote what has earned its place.

This isn’t the only thing that matters. A great model still beats a mediocre one on hard problems. I don’t have a clean benchmark for the bigger claim. What I have is a preference: given the choice between a great model in a bad harness and a mediocre model in a good one, I take the mediocre model every time. The model’s mistakes I can debug. The drift I cannot.

Watch for two weeks. The defects don’t vanish. Instead they show up earlier at pre-commit, in the feedback file, or in the review skill instead of in CI or production. That’s the win.

Tools Are Harness Too

Ian Johnson — Mon, 22 Jun 2026 15:30:29 +0000

I had a rule in my project CLAUDE.md for months that said always check Code Health before committing. The agent followed it about 70% of the time. The other 30%, it forgot; usually on the longer tasks, when its context window was filling up and the rules at the start of the prompt were losing weight against the code at the end. I added emphasis. The 70% became 80%. I added a section break. 85%. I rewrote the rule in capital letters with three exclamation points. Same 85%.

Then I gave the agent a Code Health tool. Not a rule. A tool — an MCP server entry point called code_health_safeguard that returned a structured analysis of the staged diff. The next day, the agent ran it before every commit. The day after that. The day after that. The compliance went to 100% and stayed there.

The difference between a rule and a tool is the difference between telling the agent what to do and making the right thing easier than the wrong thing. Both belong in a working harness. Tools are the half nobody talks about.

Rules tell, tools enforce

A rule is a paragraph. The agent reads it, weighs it against the other things in its context, and produces output. If the rule wins the weighing, the agent follows it. If something else wins (the code’s existing pattern, a more recently mentioned constraint, the user’s apparent intent in the prompt) the rule loses.

The probability the rule wins is not 100%. It is some number less than 100%, and the number drops as the context grows. Rules in long-running agent sessions are like introductory remarks at a long meeting: present, recorded, mostly forgotten by hour three.

A tool is different. A tool is an affordance: a thing the agent can do, exposed at the same level as everything else the agent does. The agent’s working surface is I can read files, edit files, run shell commands, and call these specific functions. A tool slots into that list. The agent doesn’t have to weigh a paragraph against another paragraph; it sees a function and decides whether to call it.

The decision shifts from should I obey this rule to should I use this capability. The shift matters because the decision happens at a different layer in the agent’s reasoning. Capability decisions are cheaper than rule-weighing decisions. They survive the long-context drift.

When a tool beats a rule

Not every rule should become a tool. The translation has a cost. Tools take time to write, they introduce new surface area to maintain, and they only earn their place when they replace a rule that fails often enough to be worth replacing.

A rule belongs as a tool when:

The rule produces a check the agent can run. Always verify the schema before writing a migration is a checkable property. Be thoughtful about naming is not.
The check has a deterministic output. Run the linter and pass is a yes-or-no answer. Match the team’s style is a vibes call.
The check fails silently when skipped. A rule the agent can ignore without anyone noticing is a rule that begs to become a tool. The tool is the gate that closes the silent skip.
The check is run often. A check the agent should run on every commit is a tool. A check the agent should run on the rare cross-project migration is a skill or a rule.

For everything else, rules are still the right unit. Tone, naming conventions, design preferences, when-to-use-X-vs-Y judgment calls: those are paragraphs, not function calls. Trying to encode them as tools is the path to a brittle harness with thirty checks and no taste.

The Code Health example, end to end

To make the trade-off concrete, here’s the rule-to-tool conversion that started this post.

The rule, in CLAUDE.md:

## Code Health is authoritative

Before suggesting a commit, run a Code Health check on the changed files.
If any file regresses below 9.0, refactor before committing. Do not commit
code with a Code Health below the project floor.

The rule is fine. It’s specific, it names the threshold, it tells the agent what to do. It just doesn’t fire reliably under load.

The tool is the CodeScene MCP server’s pre_commit_code_health_safeguard. It exposes itself to the agent as a callable function. The function takes the staged diff and returns a structured response — files checked, scores before and after, regressions flagged, suggested refactorings. The agent doesn’t have to remember to do the right thing; the right thing is one tool call away, and not calling it now looks conspicuous next to the call to git diff --cached that the agent makes anyway.

The rule still lives in CLAUDE.md, with one important change. It now says call pre_commit_code_health_safeguard before suggesting a commit. The rule names the tool. The tool does the work. The rule is shorter and the agent’s compliance is higher.

That’s the conversion shape: a rule that names a tool, and a tool that does the check the rule used to describe. Both layers are present. Each is cheaper than it would be alone.

Tools as constraint design

The other reason tools matter is that they let you design constraints that rules can’t express.

Consider the constraint the agent should never delete production data. As a rule, it’s a sentence: don’t run destructive operations against production. The rule is correct and entirely advisory. As a tool, it’s the inverse. The absence of any production-database-write capability in the tool list at all. The agent literally cannot call what isn’t exposed.

This is the difference between advisory constraint and constraint by construction. Rules are advisory. Tools are construction. The harness designer chooses where on that spectrum to put each constraint.

The rule of thumb I use: constraints that are critical to safety belong as constraints by construction. Tools the agent shouldn’t use shouldn’t be in the tool list. Permissions the agent shouldn’t have shouldn’t be granted. The advisory layer (rules) is for guidance; the construction layer (tools, permissions, capabilities) is for safety.

When I’m reading a harness, the first place I look isn’t the rules. It’s the tool list and the permission allowlist. The tool list tells me what the agent can do. The rules tell me what it should do. The safety properties live in the first answer, not the second.

What an MCP server gives you

MCP (the Model Context Protocol) is the standard most modern agents use to expose tools. The mechanics are simple: a server runs somewhere (often as a subprocess of the editor), declares a set of capabilities to the agent, and handles requests when the agent calls them.

The capabilities are arbitrary. They can wrap a CLI, an HTTP API, a database, a local file system, a third-party service, or pure computation. Whatever can be wrapped becomes a tool.

A few of the MCP servers I find load-bearing in my own work:

CodeScene. Code Health checks, refactoring guidance, hotspot detection. Replaces a long rule about complexity with a tool that gives concrete numbers.
Context7. Real-time library documentation. Replaces the failure mode where the agent hallucinates an API by giving it a way to look up the actual one.
The project’s own MCP server. Custom checks the team cares about. A “show me the current schema” tool. A “validate this against the API spec” tool. A “list the existing repositories” tool.

The third one is the one most teams skip and shouldn’t. The project’s own MCP server is where the team’s domain constraints get encoded as capabilities. A validate_contract tool, a verify_acceptance_criteria tool, a check_naming_convention tool — each takes a rule that used to live in CLAUDE.md and turns it into something callable.

The tradeoff: tools cost more

The honest cost of tools is that they’re harder to write than rules. A rule is a paragraph in a markdown file. A tool is a function with a typed schema, error handling, an installer, and a test surface. Writing a good tool takes an afternoon. Writing the rule it replaces takes ten minutes.

The afternoon pays back, but only for the right tools. The rules to convert are the ones with high cost (the rule has been violated multiple times, each violation has a real cost) and high check-ability (the check is mechanical). The rules to leave alone are the ones with low cost or low check-ability. They’re cheaper to keep as rules and live with the occasional miss.

There’s a second tradeoff. Tools accumulate. Every tool added to the agent’s surface is another option it considers. A tool list that grows to fifty capabilities slows the agent’s decision-making and pushes the relevant tools further out of immediate reach. Tools, like rules, deserve their own audit cadence in the same way rule-rot eats at rules, tool sprawl eats at agents.

Keep the load-bearing tools. Delete the ones the agent never calls. Watch the same pattern as the rules: which tools earn their slot, which sit unused.

Tools as the third layer of the harness

If rules are the first layer and sensors are the second, tools are the third. Each layer does work the others can’t.

Rules communicate intent and shape the agent’s reasoning. They’re cheap. They’re advisory.
Sensors detect when intent isn’t followed. They’re checks at the workflow’s edges. They’re load-bearing.
Tools make right things easy and wrong things impossible. They’re capabilities. They’re construction.

A working harness uses all three. The rule says do X. The tool makes X a callable capability. The sensor verifies X happened. Each layer has a different cost and a different power. The mistake is to load everything into one layer because that layer is comfortable.

The teams I see with rules-only harnesses ship the same bugs over and over with great-sounding documentation. The teams I see with tools-only harnesses ship code that compiles but doesn’t match the team’s conventions. The teams I see thriving have a balance — rules for the guidance, tools for the capability, sensors for the gate.

The seams between rules and tools

The most powerful arrangement is when a rule references a tool by name. The rule loads into context, communicates the intent, and tells the agent which tool to reach for. The tool, when called, produces the deterministic answer the rule was hand-waving toward.

## Verifying schema changes

Before writing a migration that adds, drops, or alters a column, call
`db_schema_dump` to get the current schema. Confirm the columns you reference
exist and are typed as you expect. Cite the schema dump in your reasoning so
the reviewer can see you verified.

That’s a rule paired with a tool. The rule does what rules do: it explains the intent, names the situations it applies to, and tells the agent how to handle them. The tool does what tools do: it provides the actual capability — fetching the current schema, in a structured form, deterministically.

The pair is stronger than either piece alone. The rule alone would be ignored on long contexts. The tool alone wouldn’t know when to be called. Together, they cover the case.

Convert one rule to a tool this week

Pick a rule from your CLAUDE.md that names a check. Always run the linter. Verify the schema first. Check the test coverage. Anything that says always X where X is mechanical.

Look at whether there’s an MCP server that already does X. If so, install it, expose the tool to the agent, and rewrite the rule to name the tool. If not, look at the size of X. If it’s small (a shell command, a script), write a one-tool MCP server that exposes the script. Most agent harnesses make this surprisingly easy now.

Ship the tool. Update the rule to name the tool. Watch the next ten agent runs. The rule that was complied with 80% of the time becomes 100% almost overnight, because the agent isn’t being asked to remember anymore. The capability is right there.

Rules tell. Tools enforce. The harness with both is the harness the agent doesn’t drift out of.

Harness Engineering — Building Reliable Workflows Around Non-Deterministic Agents

Ian Johnson — Sun, 21 Jun 2026 22:55:28 +0000

Six months ago this was a blog post. Then a second blog post. Then Bridle, Sellier, intent-driven-delivery, and eventually Keystone.

Now it’s a book: Harness Engineering — Building Reliable Workflows Around Non-Deterministic Agents.

What surprised me writing it: the experiments earned their place. Bridle taught me the flywheel. Sellier taught me defaults beat flexibility. IDD taught me the team layer can be reified. Keystone is the synthesis they pointed at.

It’s on LeanPub now. Pay what you want, starts at $10.

https://leanpub.com/harness-engineering

Sensors: The Other Half of the Harness

Ian Johnson — Sat, 20 Jun 2026 15:15:50 +0000

The pre-commit hook caught a migration last Tuesday that would have shipped to staging green and broken production on the next deploy. The migration dropped a foreign key constraint without naming a backfill plan, and the rule that said every constraint change needs a backfill plan had been sitting in migrations.md for four months. The rule didn’t stop the agent. The rule didn’t stop the human reviewer either. The diff was 38 files and the constraint drop was a single line. What stopped it was a four-line shell script wired to pre-commit that grepped the staged migration for DROP CONSTRAINT and exited non-zero if no--backfill: comment followed it.

That four-line script is what I want to talk about. Most of what gets written about agent harnesses is about rules — what CLAUDE.md should say, where to put it, how to scope it, when to version it. Rules are half the harness. The other half is the set of checks that fire when a rule gets broken. I call them sensors, and they get talked about a lot less than they should.

The asymmetry in the conversation

Read any of the well-circulated posts on agent harnesses and count the words spent on each side. Rules are the centerpiece. Sensors are glanced over, as if they’re so obvious they don’t deserve their own treatment. They aren’t obvious, and the assumption that they’re already there is the most expensive assumption in the field.

The asymmetry has a reason. Rules are easier to write. A rule is a paragraph. A sensor is a script, a hook, a CI step, a pre-commit config, a custom check. Rules sit at the cognitive level a writer naturally operates at; sensors sit at the level of plumbing. The first one is a lot more fun to write than the second.

The asymmetry also has a cost. Rules without sensors are vibes. The agent reads them, claims to follow them, and gets graded on whether the resulting code looks like it followed them. Looks like is the failure mode. The agent is a probabilistic system. It will obey a rule most of the time and skip it some of the time, and the times it skips are the times the rule mattered most: the awkward branch, the migration nobody likes to think about, or the edge case the rule was written to catch.

A sensor changes the contract. The rule is no longer a hope. It’s a falsifiable check.

What a sensor actually is

A sensor is anything that can detect whether a rule was followed and signal that detection in a way the workflow respects. The shape is narrow:

It runs deterministically.
It returns a clear pass or fail.
It fires at a point in the workflow where its result still matters.
It’s cheap enough that nobody is tempted to skip it.

Most of the sensors in a working harness aren’t fancy. A grep with set -e. A line in a linter config. A pytest fixture that asserts the database fixture hasn’t drifted. A pre-commit hook that runs mypy on changed files. A GitHub Actions job that fails the PR if the migration directory has a file without a paired rollback. None of these is impressive on its own. The point isn’t the individual sensor; it’s the discipline of having one for every rule that matters.

Five places a sensor fires

There are five workflow positions where sensors earn their pay. Each catches a different class of failure. A working harness has sensors at most of them.

Pre-edit

The earliest sensor is the one the agent encounters before it writes code. The rule is loaded into context; the sensor is something the agent can run to check its own work before producing output. A type stub generator, a schema dump, a --dry-run command that simulates the change. The agent doesn’t need approval; it just needs to see what its proposed change would do.

Pre-edit sensors are the rarest of the five, because they require giving the agent tools, not just rules. The pay-off is that they catch errors before any bytes hit disk. The agent that can run prisma validate on a proposed schema change is going to make fewer broken commits than the agent that can only read the rule that says make sure your schema is valid.

Pre-commit

Pre-commit is where most teams put their first sensor, because Git makes it easy. A .pre-commit.config.yaml or a husky hook fires the moment the agent (or human) tries to land a change. The check has the staged diff to look at, the rest of the repo for context, and a hard exit code that the workflow respects.

This is the sensor layer for rules that constrain the artifact — what the code looks like, not what it does. Format, lint, type, dead code, banned imports, naming patterns, the migration backfill check above. Cheap, fast, local. The agent that breaks a pre-commit hook either fixes the violation or doesn’t commit. Either outcome beats the agent commits and nobody notices.

Pre-PR

A pre-PR sensor is one that runs after the commits exist but before the diff goes up for review. The check has more to work with: a full branch, a base ref, a set of changed files that’s no longer one commit at a time. It’s the right layer for cross-file checks. Did you change this API without updating its consumers? Did you add a new migration without bumping the schema version? Does the test coverage diff drop more than two points?

Pre-PR runs cheap on CI and slow locally. Most teams run it on the branch push, which catches the agent’s work right as it’s offering the work for human attention. That timing matters. A check that fires after the human has started reviewing is a check that’s been skipped.

Post-merge

Post-merge sensors are the ones that run against main. They aren’t strictly part of the agent’s loop, but they’re part of the harness because they detect when the loop produced something that broke once integrated. Smoke tests, end-to-end suites, schema-diff jobs that compare staging to production, query-plan monitors that fire when a slow query joins the rotation.

The instinct is to think of post-merge as CI and not as part of the harness. That’s a mistake. The agent’s behavior is shaped by what fails after it shipped. A post-merge sensor that catches a regression and gets traced back to a missing rule is the most valuable kind of sensor in the whole stack. It tells you something the inner sensors missed, and it tells you what to add upstream.

Drift

The fifth and most-skipped sensor is the one that checks whether the rules themselves are still true. The codebase moves. The framework version bumps. The pattern the rule used to describe gets replaced by a new pattern. The rule is now wrong, and nobody notices because nothing fires.

A drift sensor is the check that wakes up periodically and asks: does this rule still match reality? Sometimes that’s a literal grep for the pattern the rule references. Sometimes it’s a count of how many files in the codebase still match the rule’s example. Sometimes it’s a last-modified audit that flags any rule older than six months for review. The cadence is monthly or quarterly; the goal is to keep the harness honest as the code beneath it changes.

I’ll write a whole post about rule-rot soon. The short version: a drift sensor is the difference between a harness that’s six months old and helpful and a harness that’s six months old and lying.

The pairing rule

If a rule matters, it has a sensor. That’s the principle. It sounds rigid, and it is, on purpose.

The discipline works because it forces a real question every time a rule gets written: how would I know if this got violated? If the answer is I’d see it in code review, the rule is on probation. Code review is slow, expensive, and inconsistent. It is also the wrong layer for rules that produce mechanical violations.

If the answer is the linter would catch it, the rule probably doesn’t need to be a rule. The linter is the rule. Just configure the linter and skip the prose.

The interesting cases are the rules where the answer is nothing currently checks for this, but I could write a check. Those are the rules that earn a sensor. The sensor doesn’t have to ship the same day the rule ships, but it should be on the list and have a date.

The rules that aren’t worth a sensor are the rules that aren’t worth writing down. Be thoughtful is not a rule. Bullet points should be sentence fragments unless they form a list of full sentences is also not a rule, because nothing catches a violation and the cost of writing it down is more than the cost of fixing the rare slip in editing.

A rule with no sensor is either too vague to matter or important enough that someone should pair it within the month. Either way, the absence of a sensor is information.

What good sensors share

The sensors that survive in production look alike. They share five traits.

They run in seconds. A pre-commit hook that takes 90 seconds gets bypassed within two weeks. A pre-PR check that takes 12 minutes gets the --skip-checks shortcut added to the team Slack. The check has to be fast enough that running it is cheaper than figuring out how to avoid running it.

They produce clear messages. A failed sensor that prints EXIT 1 is a failed sensor that’s about to get ignored. A failed sensor that prints migrations/20260620_drop_users_index.sql dropped an index without a paired backfill comment. Add a comment starting with --backfill: explaining the plan, or annotate the migrations with --no-backfill-needed: is a sensor that teaches the agent (or the human) how to fix the violation.

They run locally. A check that only exists in CI is a check that the agent finds out about ten minutes after it pushed. Running the same check locally before push closes the loop. The pre-commit framework and npm run check-style scripts both make this easy; the discipline is to make sure CI runs the same scripts the developer can run.

They cite the rule they enforce. Every sensor message that earns its weight names the rule. *Failed: rule MIGRATIONS-04 (foreign-key changes require a backfill plan).* The agent reads the message; if the rule is wrong, the agent can find it and propose a change. The sensor is the gate, but the rule is the explanation.

They fail loud and pass quiet. A sensor that prints OK checked 47 migration files in 1.2s every commit is a sensor whose output everyone learns to ignore. Pass silently. Fail with a paragraph. Save the human’s attention for the cases that need it.

Where sensors live in the project

A working sensor layout has four physical homes. Each holds the sensors for a different workflow position.

.husky/ # or .pre-commit-config.yaml
  pre-commit # fast, staged-files only
  commit-msg # commit message linting
.github/workflows/
  pr-checks.yml # pre-PR, runs on push
  post-merge.yml # post-merge, runs on main
.claude/
  hooks/ # agent-facing pre-edit checks
  sensors/ # one shell script per drift check
scripts/check-*.sh # the actual checks, invoked from above

The shape isn’t sacred. What matters is that each sensor has a single home and the workflow knows where to call it. The anti-pattern is the same check running in three places with slightly different logic and the team unable to remember which one is authoritative.

The sensors themselves go in scripts/ (or wherever your project keeps tooling) and get called by whichever runner needs them. That separation means the same check can fire from pre-commit, from CI, and from a drift-audit cron without three copies. One script, three invocations.

Personally, I prefer to wrap these scripts in make tasks. The benefit that I receive is that the agent always knows to use make. And that makes everything easier to find and more consistent. I use make in CI too. Usually, I work with docker and so this is a natural pair for me. Isolate the environment with docker; isolate the commands with make. That makes it easier for the human and the agent to work with the repository.

Two sensors worth writing this month

The two highest-value sensors for a team starting from zero are both small.

The migration safeguard. Whatever your migration tool is, the team’s hardest-to-catch bugs ship through it. A pre-commit script that knows the patterns your migrations are supposed to avoid — DROP COLUMN without paired backfill, ADD COLUMN NOT NULL without a default, schema-breaking renames without a two-phase plan — catches more production issues than any rule on its own. The agent will write migrations that look right and miss the constraint. The sensor sees what the human reviewer doesn’t.

The diff-budget gate. A pre-PR check that fails if the diff exceeds N files or M lines. The number is yours to pick. I use 15 files and 400 lines, which roughly maps to a PR a human can review in one sitting. The rule that says keep PRs small is exactly the kind of rule that gets nodded at and then drowned in the next month. The sensor that fails the build at file 16 is the rule with teeth.

Both are under fifty lines of shell. Both pay for themselves the first week.

The anti-patterns to watch

A few sensor failure modes recur often enough to name.

The unrunnable sensor. It lives in CI, takes 14 minutes, and depends on a secret only the deploy bot has. Nobody on the team can run it locally. By the time it fires, the human reviewer has already mentally signed off. The fix: factor out the local-runnable piece and run that on every commit.

The sensor that always passes. Someone wrote it months ago, the inputs changed, the assertions still hold, the failure case is no longer reachable. The sensor is a green light glued to the dashboard. The fix: every sensor needs at least one known-bad fixture that the test suite runs against it to confirm it still detects the violation it was written for.

The flaky sensor. It fails sometimes and passes the rest. The team learns to rerun. Within a month, the rerun habit has spread to the other checks, and a real failure gets bypassed because it looked flaky. The fix: a flaky sensor is broken. Pull it from the gate, fix it, or delete it. Do not leave it firing.

The rule with no sensor. The whole point of this post. A rule that lives in CLAUDE.md and has no check is a rule that gets followed when the agent feels like it. The fix is the pairing rule above: if the rule matters, pair it within a month.

The sensor with no rule. Less common but worth naming. A check fires, a developer doesn’t know why, the rule it’s enforcing isn’t written down. The check is correct on its own terms but unteachable. The fix: cite the rule in the failure message. If you can’t, write the rule.

Pair one rule to a sensor this week

Pick one rule from your CLAUDE.md — the one you find yourself reminding the agent about most. Open a file. Write a check. Wire it to pre-commit. Ship it.

The right rule for the exercise is one with a clear violation pattern. Always use the Result type for fallible operations is a fine candidate; you can grep for throw in the changed files and warn. Match the existing code style is a poor candidate; the violation pattern is too diffuse for a small sensor to catch.

When the sensor catches its first violation, two things happen. The rule becomes load-bearing. The agent now has to satisfy it instead of just acknowledge it. Additionally, you find out whether the rule was right. Sensors that fail a lot on changes that look reasonable in review are sensors enforcing rules that need to change. The sensor is the feedback loop; the rule is the hypothesis.

Add a second one next month. By the end of the quarter your harness will feel different. Not because the rules are smarter, but because the rules can be checked.

The rules are the part everyone writes. The sensors are the part that makes them true.

The Accidental Framework

Ian Johnson — Fri, 19 Jun 2026 15:23:25 +0000

The thing I built was a Go binary that copied markdown files into a repository. I called it an installer. The README called it an installer. The CLI verb was init. For the several months in early 2026, that framing held. Then I sat down to plan the 1.0 release, started typing, and could not write the next paragraph without using the word framework. Not in the marketing-blurb sense. In the load-bearing sense: a small runtime, a set of named extension points, conventional file layouts, a resolver that picked a winner among competing sources, a lockfile, a migration runner, per-backend adapters. The thing I had been calling an installer for four months had been doing framework work the whole time, and I just had not noticed.

This post is about that moment of recognition and what came after. It is about the refactor that turned an installer-shaped framework into a framework-shaped framework. The most surprising part was not the rewrite. It was how little of the concepts had to change. The names were already right. The directories needed to move. The runtime needed a physical line down the middle between framework code and content. But the abstractions I had been using (guides, corpus, sensors, actions, playbooks, adapters) held up under the new framing without revision. The work I had done naming things over the previous months turned out to be the work that made the refactor cheap.

I want to walk through how that happened, what tipped me off, and the one wrong turn the refactor almost made before backing out of it. The story has a tidy ending; the lesson does not, and I will get to that.

The shape was there before the word

Keystone started as keystone init — a binary you ran in a fresh repo and it laid down a harness/ directory full of markdown. Rules for the agent. A small lifecycle. A few sensors that ran your existing lint and test commands. Nothing especially framework-ish. The 0.1 README said installer four times.

Then I started adding things. Each addition felt like a feature, not an architectural shift. The commit log reads like a feature log:

0.2: install-time options. Pick your agent. Add a target.
0.3: restructured the harness into corpus, guides, sensors, flywheels. Named taxonomy for the first time.
0.4: a kind taxonomy on guides and sensors. Different kinds, different load behavior.
0.5: a migrate command for forward-compatible upgrades. The harness had a schema now, and the schema had a version.
0.10: policy plugins. Org-level rules that projects pulled in.
0.11: the policy cascade. Org → Team → Project. Strict and non-strict layers.
0.12: playbooks. Ordered chains of actions.
0.13: sensors as a tier-aware policy kind.

Read that list once and it looks like product growth. Read it twice and the shape gets harder to miss. By 0.13 there was a runtime with conventions, plugins, a cascade, a lockfile, migrations, and per-agent rendering. The README still said installer.

The thing about a shape forming under a name is that the name is the last thing to change. Each individual commit looks like a small feature; the architectural mass piles up underneath and nobody is watching the mass. I was not, anyway. I was writing the next feature.

When I opened a fresh document called PLAN-10.md and tried to summarize where things stood, the first sentence I wrote was: Convert Keystone from a harness installer with org policy plugins into a harness framework. I stared at that for a while. The conversion was not a conversion. It was an admission.

What an installer was pretending not to be

Here is what makes the installer label dishonest in retrospect. An installer drops files. It is done. The user takes it from there. The framework that Keystone had become did all of this on top:

It defined named extension points. Guides, corpus, sensors, actions, playbooks, adapters per agent. Each one had a path convention, a frontmatter contract, and a load behavior. That is an API, not a directory listing.
It resolved conflicts across sources. The same /.md could exist in the project, in a team plugin, in an org plugin. Exactly one won. The resolution order mattered, and the ordering rules were stable across versions. That is a runtime, not a copy step.
It tracked drift. A lockfile pinned per-source SHAs and per-file hashes. The binary would notice when plugin files had been edited under it. That is integrity enforcement, not file-laying.
It migrated old layouts forward. keystone migrate would walk an installed harness, apply numbered transforms, and bring it to the current schema. That is a schema migrator, with all the implications of having a schema.
It rendered the same harness into multiple targets. Claude Code’s CLAUDE.md shape, Codex’s AGENTS.md, Cursor’s .cursor/rules/. The adapter layer translated one source into many. That is a code generator, not a paste-once-and-done.

Take any one of those in isolation and you can argue it was a feature an installer happened to include. Take them together and the argument falls over. Installers do not carry adapter layers. They do not have lockfiles. They do not have migration runners. They certainly do not have a cascade resolver with strict and non-strict overrides.

The give-away, when I went looking, was the layout of the source tree. Go files at the repo root next to a harness/ directory full of markdown that the binary embedded. No physical boundary. A single Go module mixing the runtime with the content the runtime shipped. If you asked me to point at “the framework” and “the content” in the 0.x tree, I would have had to do it with words, not directories. That is a tell.

The vocabulary did the heavy lifting

Here is the part of the story that surprised me most. I was bracing for a rewrite — the kind where you sit down with a blank internal/framework/ directory and start naming concepts from scratch, then translate the old code into them, then translate again because the first names were wrong. That is the usual cost of recognizing a shape late.

It did not happen. The abstractions held.

When I sat down to write the ports-and-adapters layer of the 1.0 plan, I had a table to draft. One row per port. For each port, a name, a path convention, an activation rule. I wrote it expecting to discover gaps. Here is what I found instead.

Every port already had a name. Every name was already in use. Every path was already conventional. Guide meant a rule loaded on every turn at guides//.md. Corpus meant reasoning loaded on demand at corpus//.md. Sensor meant an automated check at sensors/.md. Action meant a single unit of lifecycle work. Playbook meant an ordered chain of actions. Adapter meant the per-agent binding at adapters//…. The table I sat down to invent had been the truth of the code since half-way to 1.0. I was writing it down, not designing it.

Writing it down still mattered. The contract was implicit until that table existed. New ports would have slipped in by accident the same way old ones had. But the design work had already happened, one commit at a time, while I thought I was just shipping features.

That is the angle worth chewing on. The names I had given things while building features carried more architectural weight than any single commit they appeared in. Guide and corpus drew the line between rules-loaded-every-turn and reasoning-loaded-on-demand. Sensor drew the line between an automated check and an aspirational rule. Action and playbook drew the line between an atomic unit of work and an ordered chain of them. Every one of those lines was a load-bearing distinction in the runtime, and I had named them all before I knew they were load-bearing.

The corollary is the part to hold onto: when the refactor came, I did not have to invent a single new concept. I had to relocate the code that implemented the concepts. The concepts themselves stayed put. Naming had front-loaded the design work. The refactor was a directory move on top of a vocabulary that was already correct.

This is the underrated payoff of taking names seriously while building. If you have named the abstractions well in version 0.3, you can refactor the runtime in version 1.0 without renaming anything. If you have named them badly, version 1.0 starts with a six-month tax to fix the words before you can fix the code. The cheap version of a hard refactor is the one where the user-facing vocabulary is already correct and only the implementation moves.

It also changes the social cost of the refactor. A user who learned guide and corpus in 0.3 still knows what those words mean in 1.0. The blog post they wrote about the harness in 0.5 is still accurate. The wiki page in their team’s Notion has not gone stale. Refactor-without-renames is the kind a user does not feel.

Writing it down made it real

Once I admitted what was happening, the next move was to make the framing physical. Not “we say it is a framework now.” That is marketing. The framework had to look like a framework when you opened the source tree, and behave like one when you read the code.

The plan went through six phases. Each had a small handful of commits, none too clever, each one a small structural improvement. Order mattered, because some moves enabled others.

The phased refactor, in order. Each arrow is a phase the next one depends on.

A few things about the order are worth pulling out.

Phase 0 was the cheapest and the most important. Before any code moved, I wrote eight architecture decision records and one port contract per abstraction. Each ADR was a single page: context, decision, consequences, alternatives considered. Each port contract was a single page: path convention, required frontmatter, cascade behavior, an example, the command that scaffolds it. The total page count was around twenty. Writing them took two days. They turned out to be the most useful pages in the whole project, because every subsequent phase referenced them. When a phase started, I re-read the relevant ADR; when a phase finished, I re-read it again to check we had stayed inside the lines.

Phase 1 was the physical line down the middle. Every Go file under the repo root got moved into internal/framework/. The CLI entrypoint moved to cmd/keystone/. The template tree relocated to internal/framework/scaffold/templates/. After Phase 1, you could point at the framework in the directory tree without using words. That is a small thing that turns out to be a big thing. Two months later, when a contributor asks “is this framework behavior or content?”, the answer is a path, not a paragraph.

Phases 2 through 5 were small structural improvements stacking on top of that line. JSON-only config (Phase 2), because YAML had drifted across half a dozen schemas during 0.x. Vendored read-only plugins (Phase 3), because the plugin model had been “edit the markdown in harness/policies/” and that turned out to be the wrong default. Projects would silently diverge from upstream and nobody would notice. Conventions, generators, and a doctor command (Phase 4), because once you have named ports you can write generators that scaffold an adapter for each port. Per-port token budgets (Phase 5), because once you have named ports you can also count tokens per port and tell the user when one is bloated.

Phase 6 was the documentation phase. An upgrade guide for 0.x users. A compatibility doc spelling out what 1.x promised to keep stable. Nothing structural changed in Phase 6. The point was to make a contract that the next year’s worth of changes had to respect.

Each phase landed in a separate set of commits. Each commit was small enough to read in one sitting. None of them were a “big refactor commit.” That is the texture of a refactor that respects the work already there: many small surgical moves, each one preserving behavior, with the architecture emerging from the sum.

The plugin draft we threw away

I want to talk about a wrong turn, because every refactor has one, and the ones that do not get talked about tend to be the ones that bite later.

The first draft of the 1.0 plan made the built-in defaults (universal guides, lifecycle playbook, default sensors, per-agent adapters) into first-class plugins shipped embedded in the binary. The plan read:

The universal engineering corpus/guides, the lifecycle actions, the task playbook, and the default sensors all become first-class policy plugins — same shape as user-installed plugins, loaded by the same engine.

The promise was symmetry. Built-ins and external policy would travel the same pipeline. One loader, one cascade, no special cases. It was clean. It was elegant. It also had a giant cost buried in the elegance: the moment defaults are loaded as plugins, editing a default stops being “edit a markdown file in your repo” and becomes “fork the embedded plugin or override it from a higher layer.”

That breaks the Rails-style ergonomics I wanted at the project layer. The whole point of conventions over configuration is that the conventional file is just sitting there in the repo, your repo, your git, ready to be edited like any other file. If the default lives inside the binary and only appears in the consumer’s repo as a shadow that overrides it, you have turned a one-line edit into a four-step debugging session. Where is this rule actually coming from? Is the override winning? Why is my edit not taking effect? I have seen that pattern in tools that load defaults from inside their distribution, and the answer is always the same: it is a constant low-level tax on every user, paid forever.

I caught it because I sat with the plan for a day before starting any code. Two of the ADRs (number five, Conventions, not plugins, and number two, Framework / client boundary) ended up being the place where the wrong turn got walked back. The decision was: defaults are scaffolded into the consumer’s harness// on keystone init, from embedded templates. From that moment on, defaults are project content. The user edits them as markdown files in their own git. There is no override mechanism for defaults, because there is nothing to override. There is just one file sitting on their disk.

Plugins still exist. They do one job: share policy across projects. Read-only, vendored, hash-verified, drift-reset on the next run. They are not the mechanism for shipping defaults, and the mechanism for shipping defaults is not the mechanism for sharing policy. Two concerns, two mechanisms. The earlier symmetry was buying elegance with the user’s debugging time.

The lesson I want to draw from this is narrower than “avoid premature symmetry.” It is: when two mechanisms produce a surface that looks identical to the user, you should still ask whether the editing model is identical. If the user edits both the same way, symmetry is paying its rent. If the user edits one of them and merely reads the other, you have two concerns that happen to share a shape, and unifying them costs you the editing UX of the one the user actually edits.

I had to throw away two days of plan-writing to walk that back. It was worth it. The walkback is what made the 1.0 surface usable.

Code moved, concepts did not

When 1.0 shipped, I went back through the changelog to count. The framework had moved every Go file into a new location. It had dropped YAML. It had rewritten the cascade resolver. It had added a vendored plugin model, a doctor command, a budget command, a port-level scaffold generator, and per-agent adapter regeneration. The repo layout looked nothing like 0.x.

The user-facing vocabulary had not changed.

A 0.x user who never read the 1.0 plan could open the new docs and find every word they already knew. Guide still meant rules loaded every turn. Corpus still meant reasoning loaded on demand. Sensor still meant an automated check. Action and playbook still meant what they meant. The only new word at 1.0 was port, and port was a name for something the user already understood without the name. Namely, the named slot a piece of content lives in.

This is the payoff I underestimated when I was naming things in 0.2 and 0.3. The cost of naming a thing well at the start is that you have to actually sit with it for a few minutes and ask whether the name says what the thing does. The benefit is that later, when you refactor the runtime, you do not have to renegotiate the contract with every user who learned the old word. The user’s mental model does not change. Only the implementation does.

A refactor that does not rename anything is the cheapest kind for a user. They keep their docs, their wiki, their training onboarding. They keep the words they say out loud when describing the product to a teammate. The internals can move freely as long as the names stay fixed.

How shapes form under names

Step back from Keystone for a second. The pattern is broader than one project.

Tools grow into frameworks. They almost always do, if they live long enough. The path looks like this: a small utility solves a small problem. It picks up a configuration file. The config file grows extension points. The extension points need naming conventions. The naming conventions need a resolver. The resolver needs ordering rules. The ordering rules need a way to opt out, then a way to opt back in. Suddenly there is a runtime, and the runtime has a contract with everything plugged into it, and the contract is a framework whether you call it one or not.

The interesting question is when to notice. Too early and you are over-designing: naming abstractions before you have three concrete uses for them, building cascade rules for a single tier, writing port contracts for one port. Too late and the runtime has accumulated implicit contracts that nobody wrote down, and the cost of writing them down is paid in surprise regressions when you try to clean up.

The way I think about it now: the moment you find yourself adding a resolver (anything that picks a winner among multiple sources of the same concept) you are in framework territory. Lockfiles and migrations are also strong signals. Per-target rendering (the adapter layer) is a hard signal. Any one of those, taken alone, is not enough. Two of them together is. Three of them together and the question is not if you are building a framework, it is whether the framework is going to be honest about itself in its README.

If you only catch it at three, the way I did, you are not in trouble. You are in the spot where naming the shape costs you a few days of writing and a few weeks of refactoring, and the user-facing surface comes out untouched on the other side. If you catch it at six or seven signals deep, the cost is higher, because by then there are real users with real assumptions and the cleanup runs into compatibility costs.

Three things to do this week

If you are reading this and recognizing your own tool in the description, here is what I would do this week, in order.

Write the port table. Open a new document. List every named extension point in the tool. For each one, write the path convention, the activation rule, and the shape of the file (or config) that goes there. Do not stop until the table is exhaustive. The table is the easiest possible draft of the framework contract. If it takes an hour, you were already a framework and your vocabulary was holding the design up; if it takes a week and you keep discovering anonymous concepts, you have found the work that needs naming first. Either way, this is the highest-value hour you can spend on the project this week.

Draw the physical boundary. Look at your source tree. Can you point at “framework code” and “content the framework happens to ship” using a path, or do you have to use words? If it is words, move directories until it is a path. This is the single most clarifying refactor for a maturing tool, and the tests will catch you if you break anything along the way. One day’s work for a small project, a week for a medium one. From the moment it lands, every conversation about “where does this go?” gets shorter.

Write ADRs for the decisions you have already made. Not the future decisions, but the ones already locked in by the code. Three to ten ADRs of one page each. Context, decision, consequences, alternatives considered. Future you will thank present you the next time someone asks why X is the way it is. The act of writing them often surfaces a wrong turn you can still back out of cheaply, exactly like the plugin draft I walked back. Better to find that during ADR-writing than during a refactor PR.

After those three, you will have a clear picture of what you have, where it lives, and why. The work that comes next is the work that actually changes things — the renaming, if any, the directory moves, the new scaffolders, the migration path for existing users. That work is real and it takes time. But you can scope it, plan it, and ship it in phases that do not break users, because the contract you wrote down in the first three steps is the contract every subsequent phase has to honor.

The thing I keep coming back to from the Keystone refactor is how much of the work had been done already by past versions of me when I named guide and corpus and sensor and action in different commits weeks apart. They were not designing a framework. They were each adding one feature and giving it a name. The framework emerged from the sum of those names. The 1.0 refactor was the moment I looked up and noticed.

If you have been adding features and naming them well, your framework is already there. It is just waiting for you to write it down.

Keystone

Keytsone is now 2.1.1! It is open source and MIT-licensed. I am very open to feedback, so if you have any, please create a discussion or an issue on GitHub. It has a real chance of landing.

A lot landed between the 1.0 cut and 2.1.1, and all of it sits on top of the framework/client split 1.0 introduced. The shape of the project didn’t change. The surfaces around it did.

2.0 was the big one, shipped June 17. A new primitive taxonomy ran through the harness end to end. An in-binary MCP server moved keystone onto the same wire its agents already speak, so there’s no separate process to babysit. A localhost dashboard surfaced live state for the first time. An eval suite came with it. The disk layout moved alongside all of that, and the upgrade was still one command: keystone migrate.

2.1.0 followed the next day. The old patches mechanism retired in favor of a versioned migrations subsystem with paired Up/Down transforms under migrations//, run with keystone migrate up | down | status. The plugin→policy rename finished across Go source, JSON schemas, and docs in the same release. Installs that hadn’t migrated yet started warning and continuing instead of breaking, so the upgrade path stays soft.

2.1.1 reshaped the dashboard. It’s now an HTMX SPA with a single

swap target, real back/forward navigation, and fragment responses keyed off HX-Request. The 14 single-purpose nav links collapsed into five sections: Observability, Harness, Sources, Flywheels, Quality. SSE topic narrowing means a widget only refreshes when its own path changes. A per-session audit log lands at .keystone/state/audit/session--.jsonl, opened with O_CREATE|O_EXCL so nothing ever overwrites a prior run. Cmd+K (Ctrl+K on Windows and Linux) opens a search popover from any page.

If you’re still on 1.x, the upgrade is the same as the install (except Brew, which has an upgrade subcommand. It carries you the whole way to 2.1.1. To bring a project harness up-to-date with the new core framework structure, run keystone migrate up. This will patch core files only and will never touch user-edited files.

The Harness Is Also Onboarding

Ian Johnson — Thu, 18 Jun 2026 15:14:48 +0000

A new engineer joins the team. Their first question (even if they don’t know it): Where should I read first to understand how things work here? I started to answer with the usual list — the README, the architecture doc, the deployment runbook — and stopped halfway through. The most useful document in the repo, by a wide margin, was the project’s CLAUDE.md. I had not written it for an agent, not a human. It still answered most questions better than the docs I had written for humans.

After reading, a new team member will have sharper questions than the previous three hires had asked in their first week. The questions are sharper because the harness had taught him the team’s conventions, naming, escalation patterns, and what not to do. The things that take months to absorb from code review and tribal memory had been compressed into a file he read on day two.

The harness is also onboarding. We had not designed it that way, but it was working that way anyway, and once I noticed it I started writing it that way on purpose.

This is the post about what changed when I noticed. Not a lot, in terms of effort. A lot, in terms of who the file was for and how the team treated it.

The README is aspirational; the harness is forensic

A harness captures the things the team has learned, mostly by paying for them. Every rule has an incident behind it. Every annotation explains a choice that was not obvious at the time. Every scope adjustment reflects a misread that the team made, noticed, and corrected. The file accumulates the way a scar accumulates; one event at a time, each event leaving a mark.

A traditional onboarding doc is written before the team has made the mistakes. It says what the architecture is supposed to look like, what the deploy process is supposed to be, what the conventions are supposed to be. It is aspirational. The harness is forensic.

The forensic document is more useful for a new hire because it tells them what actually happens when you do the wrong thing, not what you are supposed to do in theory. The new hire’s first PR is going to break three conventions. The harness tells them which three, and why, before they write the PR. The architecture doc does not. It cannot, because the conventions did not exist when it was written.

The accidental onboarding works because the harness is honest about the team’s reality in a way the docs built for humans tend not to be. A new hire does not need to know what the system should be. They need to know what happens when they touch it.

What the README leaves out

The README has the project’s pitch and the commands to run. The architecture doc has the diagrams. The contributing guide has the PR template. None of these tell a new engineer what the team has learned to be careful about.

The harness does. Here are four kinds of lines the harness tends to carry and the human docs tend not to:

A rule that says in the migrations directory, always test against a real database. That line tells the new hire this team has paid for a mocked-migration mistake. They learn the lesson without paying for it.

A rule that says errors in this service propagate as Result types, not exceptions. That line tells the new hire about a convention that pervades the codebase and would otherwise be invisible until their first review came back red.

A rule that says do not edit anything in legacy/; coordinate in #team-platfrom first. That line tells the new hire about a political-technical boundary that nobody put in a doc but everyone on the team knows.

A rule that says the test for this module hits a real Redis; bring it up with docker-compose up redis. That line gives them the exact command they would otherwise spend twenty minutes searching for.

The harness is dense. Every line is something the team chose to say. A new hire reads density they can use; they do not read fluff. That density is also why the harness wins against the README on day two. The README has the bird’s-eye view, but the new hire does not need a bird’s-eye view to ship their first PR. They need the things that will get them in trouble.

Writing for two readers at once

Once you accept the dual role, a few writing choices change. None of them make the harness longer. They make it more useful per line.

Annotate the reason. A rule that says use the structured logger is enough for the agent. A rule that says use the structured logger; we had a parsing outage in February because grep-style logs broke the alert pipeline tells the new hire the why. The agent does not need the why to apply the rule. The new hire does. The cost is one extra line. The benefit is a harness that explains itself the first time it is read.

Name the conventions, not just the rules. The harness gets sharper for the agent if rules are imperative — always X, never Y. The harness gets sharper for the new hire if the conventions have names. We call this the import-flow contract is more useful for a human than a list of three imperatives in the import directory, even if the latter is more usable for the agent. The trick is to do both: name the convention in the section header, then list the imperatives below.

Make the boundaries explicit. The harness should say which modules are owned by which sub-team, which directories require coordination, which areas are stable and which are in flux. The agent uses this to know where to be careful. The new hire uses it to know who to ask. The same line does both jobs.

Keep it readable end to end. A harness that is sectioned and skimmable is a harness a new hire can actually finish in an hour. A harness that is one unbroken list of rules is a harness the new hire gives up on at minute eleven. The agent does not care about pacing. The human reader does, and the human reader is the one who closes the file if it loses them.

The dual-role harness is not longer than the agent-only harness. It is the same length, with two or three extra lines per major rule and a structure that supports both readers.

The onboarding flow that puts the harness first

Our current first-week flow for a new engineer:

Day one: environment. Clone the repo, install dependencies, run the test suite locally. This is the boring stuff and it has not changed. The harness does not help with this; a good make bootstrap does.

Day two: read the project CLAUDE.md end to end. Read it slowly. Ask questions in the team channel as they come up. The questions become annotations the harness was missing. The whole day is for this.

Day three: read one subdirectory’s CLAUDE.md. The one closest to the engineer’s first task. The path-scoped harness has the local conventions, which are denser and more specific than the project-level rules. By the end of the day the engineer has a working model of the area they are about to change.

Day four: a small PR, with the agent assisting. The harness catches the failure modes the engineer would otherwise walk into. The human review is on the substance of the work, not on the conventions, because the harness handled the conventions before the PR was opened.

The first week ends with the engineer having absorbed more of the team’s accumulated practice than they would have in two weeks under the previous flow. The cost of writing the harness for the dual role was small — three or four hours of editing, spread over a couple of evenings. The return compounds on every subsequent new hire.

A new hire’s questions are a free audit

The thing I underestimated, before noticing this pattern, was how much signal a new hire’s questions are for the harness itself.

A new hire asking why does this module have a different error pattern than the others is telling me the harness does not name the convention clearly enough, or does not name it at all. A new hire confused about which directory a particular kind of code belongs in is telling me the harness does not encode the structural rule. A new hire surprised by a CI failure is telling me the pre-commit hooks are not communicating their constraints clearly. None of these are gaps in the new hire’s understanding. They are gaps in my document.

I treat the new hire’s first month as a free audit of the harness. Their questions are the gaps. Their confusions are the rules that need to be sharper. I patch as the questions come in, which means by the time the next new hire arrives, the document is a measurable amount better. The cycle has run six times now and the file has gotten better every single round.

This is the second dividend of the dual role. The harness is onboarding. The onboarding feedback is harness improvement. The flywheel runs in both directions, and it runs for free. The new hire is going to ask the questions either way.

What this does to the architecture doc

The first time I noticed all this, I asked whether the architecture doc was still earning its place. The answer turned out to be yes, but a narrower yes than before.

The architecture doc has the diagrams, the high-level shape of the system, the names of the services and how they communicate. It is the map. The harness is the ground truth: the conventions, the constraints, the practices the map does not capture.

The new hire reads the map first to know what they are looking at, then the ground truth to know how to walk on it. Both are still required. The harness did not replace the architecture doc; the harness made the architecture doc a smaller document, because the harness was carrying the load the architecture doc had been trying to carry and failing.

The map is short. The ground truth is detailed. That division turns out to be the right one. The diagram I used to update every quarter to keep up with the conventions is now stable for a year at a time, because the conventions live in the file that gets edited every week anyway.

Read your harness like a new hire this week

If you are running an AGENTS.md or CLAUDE.md or any equivalent file on a team project, three things to do this week.

Open the harness and read it cold. Pretend you joined the team last Monday. You do not know what the codebase does. You do not know who owns what. Read every line. Note every sentence that assumes context a stranger would not have. That list is your patch list. Annotate the reasons. Name the conventions. Fix the unexplained jargon. Do this in one sitting if you can; the cold-reader perspective decays fast.

Hand the patched harness to the next hire instead of writing them a new onboarding doc. Sit with them while they read it. Write down every question they ask out loud. Their questions are your next patch list. Do not defend the existing document. Edit it.

Move one section out of the architecture doc and into the harness. Pick the section that has gone stale fastest — usually the conventions section, or the section listing which team owns which module. Put it where the file is actually updated. Delete the now-empty section in the architecture doc, or leave a one-line pointer. The architecture doc gets smaller. The harness gets one more thing it is honest about.

After those three, the harness is doing on purpose what it had already started doing by accident. A document that is read by both the agent and the new hire has to earn its place twice. The rules that earn their place twice are the rules worth keeping. The rules that earn their place only for the agent are still load-bearing, but the dual-role audit makes them visible, which is the first step to making them better.

The harness is doing more work than you think. Notice the second job. Then write it on purpose.

Keystone 2.0 — A Worthy 2.0

Ian Johnson — Wed, 17 Jun 2026 19:41:27 +0000

https://www.tacoda.dev/keystone/

A major version should mean something. If you ship 2.0 and a user opens the repo to find the same shape with a new number on it, you have wasted their attention. The shelf is already full of frameworks that did that. So when I started planning the next Keystone release, the question I kept asking was simple: what would make a developer say, out loud, “this is a different tool now”?

Two answers held up. The first was shape. Keystone 1.x had the right ideas (guides, corpus, sensors, actions, playbooks, adapters) but the taxonomy stopped just short of being a real framework vocabulary. A user could not look at the abstractions and immediately know where the next thing they needed lived. The second was observability. The harness was healthy, but you had to take my word for it. There was no operator view. No dashboard. No way to see, at a glance, what the agent saw.

2.0 fixes both.

A real framework vocabulary

Keystone is the agent harness framework. That has been the pitch since 1.0. The Rails analogy is the right one. A working set of components, conventions, and slots, so the team building on top isn’t inventing the world from scratch each Monday morning.

2.0 makes the vocabulary explicit. Eleven primitive kinds in two layers : Framework — guide, corpus, sensor, action, playbook, eval, source — and Agent — rule, skill, subagent, command, persona. Every file in .keyston/harness/ carries canonical frontmatter declaring its kind, id, and per-kind required fields. The walker emits a single .keystone/INDEX.json that every tool reads first. You stop searching the directory tree for where a thing lives. You ask the index.

This is the part that makes 2.0 feel like a different tool. The old harness/ layout worked. The new .keystone/harness/ layout teaches.

An operator view

keystone web serve opens localhost:4773. The dashboard is fourteen pages of insight into the harness you just installed: home, metrics, insights, primitives, policies, investigator, sources, verify, prune, inbox, flywheels, evals, search, graph. Same binary. HTMX plus SSE; fsnotify on .keystone/ swaps fragments when files change. Open it in a browser, edit a guide in your editor, and watch the dashboard update without a refresh.

A harness you can see is a harness you will actually maintain. Before 2.0, “is the harness healthy?” was a question you answered by reading files. Now it’s a tab you keep open.

The other things worth knowing

A few more pieces ship in 2.0 that earn their own mention.

A built-in MCP server. keystone mcp install --agent cladue-code writes .mcp.json in one shot. Twenty-one tools, four prompts, resources for index, primitives, sources, and skills. The same binary that authors the harness now dispatches it to the agent over the model-context-protocol. One source of truth, one runtime contract.

Evals with baseline diffs. A new framework primitive lives at .keystone/harness/evals//EVAL.md. Static and sensor levels in 2.0; agent level reserved for 2.1. The interesting verb is keystone eval run --baseline — it materializes the ref in a git worktree, runs both sides, and diffs the results into a regression report. Your harness gets its own test suite, and the suite knows what “last week” looked like.

A slash-command surface in 2.0.1. Every Keystone action ships as a /keystone:* skill, projected into .claude/skills/ on init. /keystone:bootstrap, /keystone:learn, /keystone:synthesize, /keystone:audit, /keystone:spec, /keystone:orient, /keystone:review. The agent already knows how to call them. You just say the word.

There’s more: keystone search over every primitive, keystone graph --format mermaid|dot for a relationship view, keystone watch for an fsnotify loop that re-indexes on save, the plugin → policy rename, and the retirement of --harness-root in favor of a fixed framework path. The website has the full tour.

Moving from 1.x

One command. keystone migrate moves harness/ to .keystone/harness/, renames plugins/ to policies/, rewrites keystone.json to the v2 schema, regenerates the index, and refreshes host projections. It is idempotent. Pair it with keystone snapshot save --label pre-2.0 for insurance and the rollback is a single restore away.

Try it

brew install tacoda/tap/keystone
keystone init

The full walkthrough (every primitive kind, every CLI verb, the MCP tool surface, the dashboard tour) lives at tacoda.dev/keystone. Keystone is MIT-licensed and agent-agnostic; Claude Code, Cursor, Codex, Aider, Continue, Cline, Goose — whichever one you’ve already settled on, the same harness drives it.

2.0 is the version where the framework shape and the operator view both showed up. 1.0 was the right idea; 2.0 is the one that earns the number.