DEV Community: Ian Johnson

The middle path between a stock agent and a custom harness

Ian Johnson — Mon, 27 Jul 2026 17:23:15 +0000

Here’s a conversation I keep watching play out. An engineering lead wants their team on a coding agent. Compliance won’t sign off on the stock one — no audit log, no way to block writes to secrets, no way to make destructive shell commands ask twice. So the options on the table narrow to two: keep waving people off the tool they want, or build a custom agent from scratch on top of an SDK. Two engineers, a quarter, and now the team owns a product nobody wanted to own.

The conversation is the same at every company I’ve seen it in. It’s the wrong choice on both ends. Stock coding agents are close enough that the gap feels small. A custom harness is far enough that you don’t want to cross the distance. There’s a middle path that seems to have gone unnoticed, and I want to make the case for it.

The middle path is pi, an OSS terminal coding agent that ships the boring parts of the harness and hands you the interesting parts as first-class extension points. Custom tools, tool-call gates, commands, skills, prompt templates, the system prompt itself. All of it lives in your repo as normal TypeScript and Markdown. No fork. No SDK-level plumbing. If you delete .pi/, you’re back to a stock coding agent.

I spent a weekend testing this claim against five real workflows. Three coding cases, two non-coding cases. The non-coding ones matter, and I’ll get to why.

What each end costs you

Stock agents are the fastest thing to adopt. They also have a small surface. You get some combination of a system prompt override, MCP servers, and a permission list. That’s usually it. When you hit a real workflow constraint (an auditor who wants a log, a platform team with an opinionated deploy pipeline, a research group that wants a locked-down read-only reviewer) the shape of the tool doesn’t bend.

The other end is building your own. Every SDK sample makes it look easy. The tool loop is fifty lines. And then you need streaming. And cancellation. And session storage that doesn’t corrupt on Ctrl-C. And a TUI that doesn’t jitter. And compaction that doesn’t drop the last user message. And a model router that handles four providers. And permission gates. And a way to update the model catalog without shipping a release. Six months later you have a product, and the product is worse than the OSS agent you started with, and the two people who understand it are the only two people who can touch it.

The cost curve is below. Stock (red) is cheap up front and expensive the day a real constraint lands. Custom (black) is expensive up front and expensive forever. There ought to be a floor between them. Harness assembly (green) is that hypothesized floor.

What pi actually gives you

Here’s the distinction the whole hypothesis rests on. Pi is nominally a coding agent. But only two things about it are coding-specific:

The four built-in tools: read, write, edit, bash.
The default system prompt.

Both are replaceable. The built-ins go away with --no-builtin-tools. The system prompt is a flag or a file. Everything else pi ships is not coding-specific. It’s just what any competent terminal agent needs.

That’s the observation. If those two coding-specific pieces are replaceable, then pi isn’t really a coding harness. It’s a general terminal-agent harness that happens to ship with a coding-agent profile enabled by default.

Six strategies, laid out in a taxonomy

Every “custom harness” a team actually wants is really some combination of six moves. Pi exposes all six as normal repo files.

A. Repo context. AGENTS.md or CLAUDE.md at the project root, or a .pi/SYSTEM.md if you want to replace the system prompt globally. Zero code. This is where house style and always-on rules live.
B. Prompt templates. Markdown files under .pi/prompts/, invoked as /name. For workflows a human kicks off, such as “do a security review of X.”
C. Skills. Markdown runbooks under .pi/skills//SKILL.md that the model loads when a task matches. For playbooks: how we deploy, how we review a notebook, how we cite a source.
D. Tool gates. A TypeScript extension that hooks pi.on(“tool_call”, ...) and can block or modify any tool call before it runs. This is where compliance and blast-radius controls live.
E. Custom tools and commands. pi.registerTool() and pi.registerCommand() in an extension. For wrapping internal CLIs, deploy pipelines, ticketing systems, or anything else the agent should be able to do that isn’t a shell command.
F. Launch profile. A wrapper script that starts pi with a specific tool allowlist, a specific system prompt, and specific resources preloaded. This is how you ship a “mode.”

The cost of each roughly goes A < B ≈ C < D ≈ E < F. A team starts at A and adds strategies as pain shows up. None of them require touching pi’s internals.

Five workflows, put through the taxonomy

I set up a repo, pi-harness-assembly, with five cases. Three coding, two non-coding. The point wasn’t to build production tools; it was to see whether the taxonomy holds up when you point it at real needs.

Case 1: a regulated shop. Every tool call logged to disk, writes to .env and prod-config paths hard-blocked, a confirmation before destructive shell commands run. This is strategy D plus A. One extension file, about 130 lines including comments. It reacts to tool_call events, appends a JSONL audit line, checks the path against a protected-paths list, and prompts the user for dangerous bash patterns. The auditor gets their log. The team keeps the agent they wanted. If someone hates it, they delete one file.

Case 2: an internal platform team. They want a canonical deploy workflow (preflight, canary, verify, promote) that the agent can execute but can’t skip stages of. This is strategies E, C, and B together. An extension registers a deploy_service tool that shells out to the real deploy pipeline. A /deploy command kicks off the flow. A skill under .pi/skills/deploy-runbook/SKILL.md contains the actual playbook: run the stages in order, stop before prod promote, don’t batch services. The playbook lives in Markdown so the platform lead can edit it without touching TypeScript, which matters more than any technical detail on this list.

Case 3: a research group. They want a locked-down “explain and critique my notebook” mode — read, grep, list, nothing else. Strategies F, C, and B. A bin/pi-review launcher runs pi --tools read,grep,find,ls, expands a prompt template, and preloads a skill with a review rubric (correctness, reproducibility, statistical soundness, performance). No writes possible. The model describes changes instead of making them. Twenty lines of shell and two Markdown files.

Those three are the interior of the hypothesis. Coding workflows, coding tools, coding-shaped constraints. If pi is a middle path, it should handle them, and it does.

The two non-coding cases are where the hypothesis either generalizes or breaks.

Case 4: a web scraping harness. Not a coding agent. read, write, edit, bash are all gone. The extension registers fetch_url and save_record. A /scrape command starts a job. A skill contains the extraction playbook: probe with a headers request, fetch text-mode HTML, identify the repeating entity, save one JSONL record per entity, paginate carefully. The launcher passes --system-prompt “$(cat .pi/systems/scrape.md)” to reframe the agent’s identity for that invocation only, so it doesn’t leak into the coding cases in the same repo.

Case 5: a research assistant. Also not a coding agent, and structurally different from scraping. Where scraping extracts rows from known URLs, research finds sources, reads them, and synthesizes with citations. Four tools: web_search, fetch_source, add_note, list_notes. Every claim gets a citation to a URL the agent actually fetched because the skill mandates it, the extension can’t enforce it, and that’s fine because the skill is where the rule belongs. Notes accumulate as Markdown under research-notes/.md across sessions, and list_notes gives the agent a way to discover its own prior work. Long-term memory built out of the filesystem.

What surprised me

The three coding cases were what I expected. Small extensions, plain Markdown, working workflows. Not novel; this is what pi is for.

The two non-coding cases are what changed my mind about the scope of the claim.

Case 4 and Case 5 use the same six strategies as Cases 1–3. Same taxonomy, same file layout under .pi/, same shape of extension. The only new move is passing --system-prompt “$(cat …)” in the launcher instead of relying on the default. That’s it. Two unrelated domains — scraping and research — produced structurally identical harnesses.

That symmetry is the finding. If the six strategies applied to coding but not to anything else, the hypothesis would be “pi is a good coding harness with room to customize,” which is a much smaller claim. What I actually saw is closer to: pi is a general terminal-agent harness with a coding profile as its default, and repointing it at a different domain costs one extra file (the replacement system prompt).

The other thing worth naming: the same person can maintain all five cases. The extension files are 60 to 200 lines each. The skills are Markdown a domain expert can edit. There’s no shared framework code because pi is the framework. The team doesn’t own a harness product; they own a small pile of configuration.

Where the hypothesis is fragile

I want to be careful here because I’ve seen “just use OSS X and extend it” arguments fall apart in production. A few honest caveats.

The extension API is TypeScript. If your team doesn’t write TypeScript, every strategy above D costs more than I’m making it look. You’d still get A, B, and C for free, and that’s not nothing. The deploy runbook and the review rubric are Markdown, but the enforcement layer needs code.

Extensions run with full system access. A malicious pi package can do anything your shell can do. This is the same threat model as installing an npm dependency, and the same mitigation applies: read the code before installing anything from a stranger. Pi’s docs are direct about this. Teams with tight supply-chain controls need to account for it.

Skills are guidance to the model, not enforcement on the model. The scraping skill says “don’t spider off-domain unless the user asked.” A determined model can still do it if the tools allow it. If a constraint has to be true, put it in a tool gate (strategy D), not in a skill.

And the whole argument rests on pi staying maintained. It’s an OSS project. The “expensive forever” costs of the build-it-yourself path come back if the project you’re depending on goes cold, you inherit the maintenance you were trying to dodge. That’s not a reason to avoid the middle path, but it’s a risk worthy of consideration.

What I’d do Monday morning

If you’re on a team that wants a coding agent and can’t ship one because of a real constraint, here’s the smallest version of the experiment.

Pick your single sharpest pain point. Not five. One. If it’s compliance, that’s a tool-gate extension: sixty lines. If it’s a specific workflow you keep hand-holding the agent through, that’s a skill: a Markdown file. If it’s “I need a locked-down mode for the intern,” that’s a launcher script and a --tools flag.

Write only that. Ship it to one person. See if it holds up for a week. If it does, add the next thing. If it doesn’t, delete it and you’re back to stock pi with no sunk cost.

The reason to bother with the middle path isn’t that it’s technically clever. It’s that it lets you match the agent to the actual constraint without buying a product you don’t want to own. Every strategy in the taxonomy is reversible. You can start at A on a Tuesday and be back at stock pi by Friday if it wasn’t worth it. That property — cheap to try, cheap to unwind — is the whole point.

The reason I care about this is that I keep meeting teams who’ve decided the choice is binary. It isn’t. The audit-and-guard extension in the test repo is 135 lines. The deploy workflow is 81. The scraping tools are 165. None of them are impressive engineering. That’s the point. The interesting work happens above them, in the skills and the workflows and the constraints that actually match the team. The taxonomy exists to keep the code layer small enough that the team can afford to focus on the layer that matters.

The book I wrote because “harness” meant four different things

Ian Johnson — Thu, 23 Jul 2026 15:22:55 +0000

You’ve probably felt this. You point the same agent at two of your repos with the same lazy prompt, and one gives you a tight change you can review over coffee while the other hands back a sprawling PR that quietly broke something three files away. Same model, same day. Different repo.

That gap is the whole reason I wrote Harness Engineering: Building Reliable Systems Around Non-Deterministic Agents. It’s out, it’s long, and this is the pitch: what it covers, what it’ll teach you to do, and why it’s worth the shelf space.

The problem it’s about

A coding agent is non-deterministic by design. It also writes faster than your team can socially enforce its standards. A human writes a hundred lines a day; an agent writes thousands, and review still moves at human speed. Most of what it writes lands before anyone reads it carefully. And it inherits whatever’s already in the repo and amplifies it; the agent doesn’t change your standards, it enforces them, so the lower the standards, the more enforcement you get.

You can’t make the model deterministic. So make the work around it deterministic. That’s the wager the whole book rests on. Reliability isn’t “the agent never varies” — it’s variance that stays inside a band you can read, debug, and ship. You narrow that band three ways: narrow the input (scope plus acceptance criteria that can actually fail), narrow the output (gates that reject anything outside the band), and narrow the recovery (decide the next move after a failure before the failure happens).

Here’s the part that turns it from craft into a decision a business can sign off on: every bit of determinism you add moves the trust boundary. The question stops being “do I trust the model?” (you never fully will) and becomes “how much unreviewed agent output am I willing to ship, and behind which gates?” That’s not a vibe. That’s a number a team can own. The book’s job is to give you the levers to set it where you want it, on purpose.

Shaping the charter

The charter is everything you author to say how the agent should behave, and it’s the bulk of the book. This is where most teams have the most slack and the fastest wins.

You’ll learn to write a lean CLAUDE.md — under two thousand words, because a bloated one is just token budget you’re setting on fire. You’ll tier your rules into iron laws, golden rules, and preferences so the agent knows what’s non-negotiable and what’s taste. You’ll build an ontology so one term means exactly one thing across the repo, because the fastest way to get inconsistent output is to let “job,” “task,” and “run” all mean the same thing in three files. And you’ll get honest about a trap I see constantly: aspirational rules teach the agent to lie. A CLAUDE.md that describes a codebase you wish you had is worse than none because it makes the agent confidently obey instructions the rest of the code contradicts.

The book covers the Claude primitives directly — CLAUDE.md, rule files, skills, sub-agents, hooks, and MCP servers — and, more usefully, when to reach for which. A skill and a sub-agent solve different problems; a hook and a gate live at different points in the loop. I keep the stock harness (what the tool actually gives you) separate from my own conventions (the rule tiers, the patterns I’ve found useful) so you can tell what’s load-bearing from what’s just my preference. Every tool you hand the agent is a capability grant: grant the smallest one that closes the gap. And a limit that only logs is not a limit.

Building the harness: loops, DAGs, RAG, and a coding agent from scratch

When the stock agent can’t express your workflow, you build one. Part V is the engineering of that.

The first fork is the shape of control flow: DAG or loop. A DAG is a fixed graph of steps: predictable, testable, easy to reason about, and rigid. A loop lets the agent decide when it’s done: flexible, and dangerous if the exit condition is soft. The book teaches you to pick deliberately and to engineer the loop when you need one: real termination conditions, budgets, and the recovery move baked in, so “the agent decides when it’s done” doesn’t mean “the agent runs until your bill does.”

RAG gets treated as what it actually is — retrieval as a harness job, not magic. What you put in the context window, how you rank it, how you keep it from crowding out the instructions that matter. Feeding the agent more isn’t the goal; feeding it the right thing at the right token cost is.

Then the operational spine: sandboxing and permissions, prompt injection inside a grant you’ve already given (the attack people forget), model selection and routing so you’re not paying frontier prices for a formatting pass, cost and resource caps, and evals so you can measure the harness instead of hoping. It ends with building a coding harness end to end — there’s a from-scratch Go harness you can clone and read, because a harness you can’t read is one more thing you’re trusting blind.

Orchestrating multiple agents

One agent is a harness problem. Many agents is orchestration, and it’s the last third of the book.

This is where “vibe coding” breaks at team scale. The fix is intent contracts — a scoped statement of what a unit of work must do, with acceptance criteria that can fail. This can be a full spec, or one much smaller in scope. An acceptance criterion that cannot fail isn’t a criterion, it’s a wish. From there: the six phases a piece of work moves through and the interrupt protocol — when an agent hits ambiguity it doesn’t guess, it files an interrupt, the team-level version of raising an exception. You’ll cover graph engineering, a reconciler daemon that keeps the whole thing converging, and the two ends of the spectrum: dark factories (fully automated, running unattended) and lights-on factories (automated but observable, with an audit trail you can read after the fact and proactive approval processes). The rule that keeps a factory honest: one that can edit its own gates has no gates.

It’s grounded in real tools, so it’s not architecture astronaut stuff. It’s patterns that run.

What you walk away able to do

Audit a repo in 20 minutes with a 17-item, 34-point instrument and get a gap score before your next agent session — then close the cheapest gap first.
Author a charter your team actually follows , from CLAUDE.md to rule tiers to ontology to bounded tool grants.
Build a harness in code — loop or DAG, retrieval, sandboxing, routing, cost caps, evals — when you outgrow the stock agent.
Stand up orchestration for many agents without it becoming a factory you can’t see into.
Keep all of it healthy over time with flywheels, post-mortems, charter debt tracking, versioning, and evals.

Every chapter ends with one copyable artifact: a checklist, a hook script, a worksheet, a code module. There’s a 30-day starter plan in the appendices. The template is what survives the read.

The honest part

The patterns aren’t mine. Tests, types, naming, bounded contexts, characterization tests, continuous delivery — they predate the agent and belong to Brooks, Parnas, Fowler, Evans, Beck, Feathers. What’s mine is the assembly: two named disciplines, a shared vocabulary that ends the “harness means four things” argument, a maintenance pattern, and a working tool.

And the bet is built to last. If your reliability strategy depends on a specific model’s behavior, that strategy has an eight-month half-life. This one bets on the repo, not the model — so it doesn’t expire the next time a new model ships.

It’s big: 62 chapters and a total of 866 pages. Not a weekend read. But it’s built to be used. The book is grounded in an engineering discipline: no vibes.

The curator doesn’t stop writing code — the curator stops writing code the agent can write. If that reframe lands for you, this is the long version.

Read it on LeanPub: Harness Engineering: Building Reliable Systems Around Non-Deterministic Agents

The Governance Gap Is a Boundary Problem

Ian Johnson — Wed, 08 Jul 2026 15:24:45 +0000

One of my early factory prototypes had a clean audit trail. Every action an agent took, recorded. Who did it, what it touched, and when it ran. I was proud of it. Then I watched an agent remove a status enum from a database column. Not a value I wanted changed. It was a status that other records depended on. The audit trail caught it. Logged the delete, the actor, the timestamp. Caught it perfectly, and caught it after the status was already gone.

That’s the moment the “governance gap” stopped being a phrase I’d read in other people’s posts and became something I’d built by accident. I had governance you could read. I didn’t have governance that could say no.

This is what I learned about that gap by building a tool to close it. The tool is open-refinery, a control plane that governs how AI-driven work reaches repos, models, and tools. The lesson is simpler than the tool. The gap everyone’s naming isn’t a missing policy. The gap is that nothing makes an agent respect it. And a policy nothing enforces is a suggestion.

The gap is between the policy and the work

People use “governance gap” to name the space between the policy you wrote and the agent doing the work. You have a policy. The agent runs. The policy and the run never meet, so nothing makes the agent respect it.

The common fix is to add something. A policy document, an approval step, or a content filter. Each of these is a thing you write down. None of them is a thing that stops an action.

Here’s the distinction that took me three prototypes to see. An audit log observes: it tells you what happened. Governance has to gate. It must stop the thing before it happens. My prototype with the clean audit trail was all observation. When the agent deleted that status, nothing stood between the intent and the delete. The log wrote the event down a moment too late to matter.

Once you see governance as gating instead of recording, the question changes. It’s no longer “what policy should we write.” It’s “where does the policy get enforced before the action does.” That question has a location in it. Most governance talk doesn’t.

A gate has to live somewhere

A gate is a control that runs before an action and can refuse it. To run before the action, it has to sit between the thing that wants to act and the thing being acted on. It needs a seam. In most agent setups, nobody drew the seam.

So the first real decision in open-refinery was to split two things that usually run together.

The harness does the work. It’s the agents, the prompts, the memory, the tools, the orchestration. It lives in your application. It decides what to try.

The platform governs how work reaches a target — the repos, the models, the tools the harness wants to touch. The platform does none of the work. open-refinery executes nothing. It sits on the seam and decides whether an action reaches its target.

The harness never touches the target directly. Every action passes a gate that can say no, and the platform owns that gate.

Draw that line and the gate has a home. The harness asks to do something. The platform checks the policy. The action reaches the target, or it doesn’t. Skip the line and your policy floats free of the work. That was the prototype I started with: the policy lived in one place, the actions happened in another, and nothing connected them.

None of enforce-before-execute is new. RBAC, IAM, and Kubernetes admission controllers are all examples. What’s new is who’s on the other side of the gate. A service either has permission or it doesn’t. An agent reads your policy in prose and decides how to apply it. The boundary is what lets you stop trusting that interpretation: the policy gets enforced on the seam, not left to the agent to honor.

The boundary is the whole idea. Everything else here is a consequence of it.

Make the loop boring on purpose

I wanted the platform to run without a human watching. Lights-out. I also wanted every action fully auditable. Those two goals fight each other the moment the system makes a choice you can’t explain.

You can’t audit “the model decided to.” You can audit a function that takes recorded inputs and returns a recorded output. So the core loops in open-refinery are plain code, not agent choices. The policy is encoded there, deterministically, where the agent can’t reinterpret it away. There are two loops: transition, which moves a work item from one stage to the next, and produce, which runs an action against a target. The agent fills in the inputs. The code decides what happens.

This audits the mechanism, not the judgment. Which input the agent picks is still the agent’s call. Determinism doesn’t make the decision safe. It makes the decision inspectable, and it puts a gate between the decision and its effect.

Every action runs the same sequence:

authorize -> enforce policy -> execute -> record

The order is important. Authorize before the run, or you can’t prevent it. Record after the run succeeds, or a crash mid-action loses the event.

The check itself is small:

decision = enforce(role, action, resource, mode="strict")

if decision.denied:
    raise PolicyDenied(decision.reason)

A policy names who may do what to which resource. In strict mode, nothing’s allowed unless a policy allows it. In audit mode, everything’s allowed unless a policy denies it. Strict mode fails closed: if nothing explicitly allows an action, it’s denied. Audit mode fails open: an unknown action runs. Which one you pick is the whole difference between a gate and a log. Audit mode is where you learn what your agents do. Strict mode is where you decide what they may do. That enum from the top of the post? A denied action under a strict policy on the status resource. One line, and it never happens.

Human oversight is a dial, not a switch

One of the things I got wrong early was treating human-in-the-loop as on or off. Either a person approved every action, which didn’t scale, or nobody did, which was the prototype that deleted the status. Both are bad. The useful setting sits between them, and it should be a setting.

open-refinery has five oversight levels, set per process. At L0, manual, every action needs a human to approve it. L1 and L2 loosen the grip: assisted, then supervised, where only the gated steps wait for a person. L3 is autonomous, no approval. L4 is dark, fully lights-out. You pick the level for a process based on how much you trust it — not for the whole system at once.

What stays the same across all five is the machinery. The gates run the same way at every level, and the records share one schema. What changes is how many approval events the trail carries, and how many actions wait for a person before they execute. Because the loop is deterministic, moving the dial doesn’t change what gets checked or how it’s recorded; only who has to say yes, and how often.

The boundary shows up again here. The dial sits on the platform, not in the harness. You can turn a process from supervised to dark without touching a line of agent code, because the approval step lives on the seam.

How to tell whether a policy is real

Once the platform could gate actions, I needed to know which policies were actually doing anything. A policy you wrote and a policy an agent respects are not the same policy.

open-refinery models this with claims. A governance claim is a statement about a repo: agents may not delete production data, all migrations get reviewed. Each claim can be backed two ways. An instruction states the intent. A gate enforces it. A claim is covered only when it has both.

That gives three states, and the middle one is the whole problem.

Instruction and gate: covered. The policy is written, and something stops violations.

Instruction, no gate: partial. You wrote the policy. Nothing enforces it. This is the governance gap, made countable. It was the most common state I found. It’s the one that feels safe and isn’t.

Neither instruction nor gate: an imitation surface. It reads as governed. It isn’t.

open-refinery scores a repo on this. How many claims have a gate, and not just an instruction? I won’t pretend I have clean fleet-wide numbers to wave at you. What the model buys me is the ability to ask the question per repo instead of assuming the answer. Partial coverage isn’t a failure of intent. It’s a gap you can name, which means you can close it.

A record you can’t quietly rewrite

Gating decides what happens. Provenance decides whether you can trust the record of what happened. These are different jobs, and I discovered that after ran them together at first.

An audit log you can edit isn’t evidence. If someone can delete the record of the delete, the record is worth nothing. So the records in open-refinery are immutable, and each one chains to the one before it with a hash. Change a past event and every hash after it breaks. One function walks the chain and checks it:

verify_chain(events) # raises if any link's hash doesn't match

This doesn’t stop someone from doing something bad. It stops them from hiding that they did. I want to be honest about which one this is: tamper-evidence, not tamper-proofing. It only holds if two things are true: someone actually checks the chain, and its head is anchored somewhere the writer can’t reach. Put the log and the verifier on the same box an attacker owns, and they can rewrite an event and recompute every hash after it. A local chain raises the cost of quiet tampering. It doesn’t make it impossible.

The gate isn’t one thing in one place, either. It shows up at three seams. The transition gate checks a work item moving from one stage to the next. The invoke gate checks an executor about to run a command. And there’s a call the harness makes before it acts at all:

POST /authorize { role, action, resource, intent: "deploy to prod" }

The harness asks before it deploys. It gets back an allow, or a PolicyDenied. The policy does nothing until it sits on one of these seams. A policy with no seam is the partial-coverage case from the last section, seen from the code side.

The policy itself needs governing

This is the part the governance-gap conversation skips. If governance lives on the boundary, then changing the boundary is itself an action. So who governs that?

In my early prototypes, the policy lived in a config file. Anyone who could edit the file could change it, and there was no record of who changed what, or why. I’d built a system that gated every agent action and left the policy itself wide open. That’s not a small hole. That’s the same gap, just moved up one level.

In open-refinery, a policy change is a governed action like any other. It walks an approval chain. It lands in the same audit trail. Every version of a policy records its author, its timestamp, and what changed. Ask “who changed this policy, and when, and why,” and there’s an answer.

A policy file anyone can edit with no record isn’t governance. It’s a suggestion with a version history, at best. Governance that can’t govern itself doesn’t hold. This is the second-order problem, and it’s where the boundary earns its keep: moving the seam is an action, so it passes a gate too.

Put one gate on one policy this week

You don’t need a platform to start closing your own governance gap. You need to stop treating your audit log as a control.

Pick one policy you’ve written for your agents. A real one: in a doc, a prompt, or a README. Find where it’s enforced. If the honest answer is “it isn’t,” you’ve found a partial-coverage surface on your own system. That’s the gap, and now you can see it.

Add one gate: a check that runs before the action and can refuse it. Start with the action you’d least want an agent to take without you watching. For me that was a destructive database change. Pick yours.

Make the policy change leave a trace. Even a commit with a reason in the message is a start on second-order governance. The goal is that “who changed this policy” always has an answer.

Do those three things and you’ve moved from governance you can read to governance that can say no. That’s the whole gap, and it closes one gate at a time.

Charter and Harness: What Each One Is

Ian Johnson — Tue, 30 Jun 2026 23:47:29 +0000

It seems that every time I think I know the shape of the harness, a new fact presents itself that I have to account for. I recently wrote a book Harness Engineering. But new information has caused me to rethink my model a bit. I plan to leave the term harness alone, and tighten up the definition. In addition, I add the term charter.

What follows is a chapter I’m working on for the book to bridge the gap in my model. I’m offering it completely free because I think it’s the best definition I’ve had so far. Your feedback is very welcome.

Clone a repository that someone has tuned for agentic coding and open the .claude folder at its root. What you find is mostly prose. A CLAUDE.md that explains the project. Rule files with names like testing.md and migrations.md. A few skills, each a short markdown file describing a workflow. Maybe a settings.json wiring up a hook. Almost none of it is code that runs on its own. It is a stack of instructions, written for a reader who happens to be a model.

Now start the agent: Claude Code, or whatever the team standardized on. It reads that folder, holds it in mind, and goes to work on your repo. The folder did nothing. The agent did. There, in a single clone, sit the two things this book is about: the material you author, and the engine that acts on it. This chapter names both and says what each is made of. The next chapter draws the line between them. Here the job is to fill the two boxes.

The charter: what you author

The charter is everything you author to specify how the agent should behave in your project. That .claude folder is most of it. The word is borrowed from its plainest sense: a charter is a founding document that says how a thing is to be governed. Your repo’s charter governs the agent that enters it. You write it, you review it in pull requests, you amend it when it turns out wrong, and it governs every agent that works the repo until you change it.

A charter comes in four kinds of material, and naming them now saves arguments later.

Guides instruct before the agent acts: the CLAUDE.md, the rule files, the slash commands, the prose half of a skill. A guide is a sentence the agent reads and weighs against the situation, like prefer composition over inheritance, and justify a base class in the PR description. No graph expresses that. It is judgment, written down.

Tool definitions declare what the agent can reach for and how it is wired: the mcp.json entry, a tool’s schema, a skill’s script, or a plugin’s config. One artifact can span families: a skill’s prose is a guide, but its script a tool definition. The definition is yours and lives in the repo. The thing it points at, the installed server or the plugin binary, is external, and the next chapter sorts that seam.

Sensors tell the agent whether the work it just did is acceptable: the test files, the lint config, the type-checker settings. You authored them, so like a hook they are charter even though they run. A sensor reports; it does not refuse.

Gates refuse: the pre-commit hook, the branch-protection rule, the blocking reviewer. A gate is a sensor wired to a stop.

Guides, tool definitions, sensors, gates. If something in your charter is none of those four, it is probably not earning its place.

Most of a charter is prose, and that is the point. A rule written as a sentence covers cases no control-flow graph would, because the agent weighs it instead of branching on it. Some of a charter is code you author, a hook or a gate, and that code is still charter: you wrote it to state a standard, so it is a specification the engine runs, never the engine itself. A charter is also a living thing. It is versioned in the repo, reviewed like any change, and amended as the project teaches you where it was wrong. The rule you write today because an agent ignored a convention is a line in a document you will keep editing for as long as the repo lives.

Here is the whole of it, as a definition you can hold: the charter is your repo’s .claude folder, all of it (the rules, the hooks, the agents, the skills, the slash commands, the settings), plus the dependencies you declare: plugins, policy packs, shared skills, tools such as MCP servers. Those are two questions about a thing, not one. The families above sort by role: is it a guide, a tool definition, a sensor, a gate? Origin is the second question: did you write it by hand in the folder, or adopt it as a declared dependency? A declared dependency keeps its declaration in your repo, which is charter, while its installed body lives outside, which is not. Declaration in, source out. Either way the test does not change: you authored or adopted it to specify behavior.

If you have been in this field a while, you may have called this folder “the harness.” It is the charter. The word slipped for an understandable reason, and the next chapter traces it; for now, hold the line that the folder you author is the charter, and the engine that reads it is the harness.

The harness: the engine that applies it

The harness is the engine that applies the charter against the model. The field has a shorthand worth borrowing: an agent is a model plus a harness. The model predicts text and nothing more. The harness is everything wrapped around it to turn prediction into work, and the charter is what the harness consults while it does. So the relationship runs in a line. You author a charter; the harness applies it to drive the model; the model produces a result, a diff or a document or an answer. Charter, harness, model, result. The charter is the input you control. The harness is the only stage that acts. The next chapter takes that line apart; here it is enough to see the shape.

An engine that does this has jobs, and Part B of this book takes them one at a time. It assembles the context the model sees each turn, choosing what to load and what to drop, and invokes the tools your charter defined. The permissions you set decide which files it may touch and which commands it may run. The gates fire, the sensors report, and the loop turns with bounds and a kill switch until the work is done or the budget is spent. Every one of those is the engine at work, not a standard you wrote down. The standard is the charter; the doing is the harness.

You get most of a harness rather than build it. Claude Code is one, Cursor is another, and a LangGraph orchestration your platform team stood up is a third. The vendor ships the engine, you point it at your repo, and it reads your charter on the way in. For most teams that is the entire relationship with the harness: choose one, configure it, move on. The teams that build their own, a custom orchestrator or a queue that runs agents across a fleet, are doing harness engineering proper, and Part B of this book is written for them. Part A is for everyone, because everyone writes a charter.

A harness can even ship its own charter-like pieces: Claude Code’s built-in slash commands, a default skill, baked-in rules. Those are harness, because the vendor authored them, not you, and they do not sit in your .claude folder. Your charter can override them. Drop a fix-bug skill into your folder and the agent runs your bug workflow instead of the vendor’s default; write a rule that bends a built-in command to your process, and your version wins. The built-in is harness, your override is charter, and the agent meets the charter.

One part of the charter couples to the harness more tightly than the rest: workflow. A rule or a hook is a primitive the harness simply reads, but how work gets defined, prioritized, and completed is a process the harness has to register and run, not merely read. That is one reason a team builds its own harness, when the workflow it needs is one a stock harness cannot express. Part A’s chapter on contracts is where you define the workflow; Part B’s on orchestration is where the harness runs it.

Two words, two boxes: the charter you author, the harness that applies it. They are not the same kind of thing, and it is a mistake to treat them as one. Where exactly the line falls, why a hook you wrote is charter even though it runs, why the orchestrator is harness even though you might own it, is the next chapter. For now, hold the two boxes apart and know which one takes most of your time. For most of this book, it is the charter.

What’s in each box

Run this against your own repo.

Your charter — everything you authored or adopted to specify behavior. Most of it is the whole .claude folder; the rest is declared dependencies. By role:

guides: CLAUDE.md, rules, skills, slash commands (what instructs)
tool and agent definitions: mcp.json, tool schemas, sub-agent and plugin config (what the agent reaches for)
sensors: test files, lint and type-check config (what reports)
gates: pre-commit hooks, branch protection, required reviews (what refuses)

By origin: written by hand in the folder, or a declared dependency (plugin, policy pack, shared skill, tool such as an MCP server) whose declaration is charter and whose installed body is external.

Your harness — the engine you run, mostly not yours to write:

the coding agent (Claude Code, Cursor, Aider)
any framework or orchestrator above it (LangGraph, a custom runner)
the loop that runs work on repeat
the runtime that fires your hooks, invokes your tools, and enforces your permissions

A thing on the first list, you maintain, and it is where your reliability comes from. A thing on the second, you mostly choose. Telling which list something belongs to is the whole skill the next chapter sharpens into a single test.

The charter is the half of the work with your name on it. The rest of this book is about writing it well.

Additional Resources

Harness Engineering: Building Reliable Workflows Around Non-Deterministic Agents

Ian Johnson — Fri, 26 Jun 2026 21:27:58 +0000

Hello everyone. I just released a book about harness engineering. It’s on LeanPub. Pay what you want, starts at $10.

https://leanpub.com/harness-engineering

What does it cover?

The discipline, the taxonomy, the behavioral system, the reliability layers, rules, guides, corpus, playbooks, rule cascades, architecture as communication, tools, skills, agents, plugins, MCP, evals, delivery orchestration, continuous delivery with agents, PR as checkpoints, policy libraries for team and org extension, patterns and anti-patterns, harness debt, post-mortem, harness curation, loop engineering. Everything I have learned over the past six months with hands-on experiments with a real production codebase.

Why so cheap?

We have a problem right now with junior and mid level engineers feeling the pressure of AI. I think we have an obligation to help them — and I think that means providing a path to senior. How could they get to senior if the agent does everything for them?

We must give juniors the support they need to become seniors.

One of the goals with this book is to support these folks in their growth by showing the big picture, and then anchoring it with cited works of software engineering canon. This book gives both and has an appendix specifically for juniors with how to operate and grow in the world of agentic coding.

If you are concerned about losing your job or not being able to find a job doing software engineering because of LLMs, then this book is for you!

There’s more

The book also covers scopes from single contributor to enterprise and provides actionable advice whether you read a particular part or the whole thing. It offers benefits to anyone looking to learn how to get better results out of their coding agents, regardless of experience, role, project, team, company, or organization.

I hope it serves you well.

This just rearranged my view on harness engineering

Ian Johnson — Thu, 25 Jun 2026 23:16:14 +0000

I just read this article — I recommend you do as well.

Loop Engineering Is Just Software Engineering. We Have a Name for That.

I’ve been doing a lot of work on harnesses, but I have been focusing at the project level. I’ve only recently started to delve into LangChain/LangGraph. I built my own daemon that implements this with a shell on my local machine. It works, but it could be better. I had always thought of these as two layers and they’ve become crystal clear to me now.

When I read this, I immediately want to try it, but I do not have access to an API key yet. Also, I don’t really want to clobber my organization with API pricing. So I asked claude to build a prototype in Python given the description. I tested it, and it worked pretty well.

But if you’re already at the API point (and you probably are if you are looping), here’s the tool: https://iii.dev/

This idea is important

The single most important takeaway from this article is that we’ve already solve the problems of loop engineering with distributed systems. Instead of getting stuck in loops or overpaying, you can take advantage of the queueing pattern — dead letter queues, circuit breakers, retries, and so on.

This idea is dynamite.

It had slipped past me because I was focused on the project level of the harness and loop engineering has been mainly focusing on creating loops with agents, not how to do it. So diagrams and explanations tend to lead to synchronous or daemon sorts of solutions. In hindsight, it seems so obvious.

It also removes the orchestrator problem: the fact that different subagents about bound to a main orchestrator thread replicates Brooks’ warning for organizations in the form of agents. But with the distributed system, there is no orchestrator — that’s been reduced to state. Brilliant!

Vocabulary

It also shifts my vocabulary. I’m currently writing a book about harness engineering: https://leanpub.com/harness-engineering

I was focusing on the repository level, but now the delivery harness (as I was calling it) builds one level up. This will probably cause me to have to re-arrange my definitions, but here’s where my head is at.

Layers of the Harness

I visualize the harness like an onion. But the harness I’ve been talking about has been code, and the delivery harness is a service. That means that personal still sits at the center, wrapped by the project (or repo) harness, and then wrapped with team and org harnesses that implement policies just like software libraries, and then wrapped in the delivery or orchestration harness. I’ve used some tools like this recently, like Hivemind (https://cohen-liel.github.io/hivemind/). It’s great, and there are plenty of others to try, but this has changed my entire outlook on the type of architecture the delivery harness should use. I think he’s right — it’s a distributed system.

Thank you Mike Piccolo!

Reading the Agent Log Like a Detective

Ian Johnson — Wed, 24 Jun 2026 16:21:53 +0000

The agent shipped a broken migration last month. The PR was clean, the tests passed locally, the reviewer signed off in twelve minutes, and the deploy to staging green-lit on the first try. Production started erroring within four minutes of the merge because the migration assumed a column existed that had been renamed two weeks earlier.

The first instinct was to blame the model. Bad agent, didn’t read the docs. The second instinct was to add a rule. Always check the current schema before writing migrations. Both instincts felt productive. Neither would have fixed it.

What fixed it was reading the transcript. The agent had, in fact, looked at the schema file. The schema file in the repo was correct. It had been updated when the column was renamed. But the agent had also loaded a database/CLAUDE.md rule that described the column under its old name as the canonical reference. The rule was stale. The agent followed the rule, but the rule was wrong.

You could call this a model problem — the agent trusted a stale rule over the live schema file sitting right next to it. But that’s not the lever you can pull. The fixable cause was in the harness (the rules, skills, hooks, and tools wrapped around the model) and here it was a rule that lied. It was visible in the log if you read it forensically.

That’s what this post is about. Reading agent transcripts as evidence, not as narrative. The five failure modes that show up over and over. And the habit of treating every wrong commit as a case to investigate before reaching for a fix.

Replay, not blame

The first move when an agent ships bad code is to resist the urge to fix and move on. The fix is downstream. The question worth asking is how did the agent decide to do that. The answer lives in the transcript.

Most modern agent harnesses save full transcripts somewhere. Claude Code stores them in ~/.claude/projects//, Cursor keeps recent ones in its UI, and most CI-driven agent runs log to a build artifact. The transcript is your scene of the incident. It records what the agent loaded, what it read, what it ran, what it produced, and in what order. Everything you need to reconstruct its reasoning is in that file.

The discipline is to treat the transcript the same way an SRE treats a stack trace: not as a list of complaints, but as the most reliable evidence of what happened. The agent’s text answers in the transcript are sometimes hand-wavy. The tool calls are not. The tool calls are facts.

Read the tool calls first.

The five failure modes

Across the agent transcripts I’ve reviewed, the same five failure modes keep producing wrong-code incidents. Each one has a cause in the model but each also has a cause in the harness, and the harness is the half you can change. All five are visible in the log.

Missing context

The agent never read the file that mattered. It wrote a migration without checking the schema. It wrote a test without checking the test conventions. It added a new module without checking how other modules in the same area were structured.

How to spot it in the log: scroll back to the start of the task and look at the Read and Glob tool calls. Note which files the agent loaded. Compare to the files the change touched. The pattern is: the agent edited file X, but never read file Y, where Y was the file that defined the convention X had to match.

The fix isn’t in the model. The fix is to make sure the relevant context loads automatically. Either by giving the rule a sharper path scope (so it loads when the agent touches the area), by adding a skill that walks the agent through the right files in order, or by adding a hook that injects the relevant context at the start of tasks of that shape.

The agent did what it was told to do. The instruction set didn’t include read Y first. That’s a harness gap.

Contradicted rules

The agent followed a rule that was wrong. The rule was outdated, or it referenced a pattern the codebase no longer uses, or two rules in the harness said different things and the agent picked the one that produced the bug.

How to spot it in the log: look at the rules the agent cited in its reasoning. If the rule cited matches the code change exactly, but the change is still wrong, the rule is the bug. Cross-check the rule’s claims against the current code. Rule says use UserRepository, code uses UserService is a smoking gun.

The fix is the rule-rot audit. Rules drift faster than people think. A rule written six months ago can be confidently citing a class that was renamed in April. The agent trusts the harness; if the harness lies, the agent ships lies.

The contradicted-rules failure mode is the one I find most embarrassing because the agent did exactly what the harness said to do. The agent isn’t wrong. The harness is wrong. And the harness was wrong because I didn’t audit it.

Hallucinated APIs

The agent called a function that doesn’t exist. Used an option that isn’t supported. Imported from a module that doesn’t export what it imported.

How to spot it in the log: search the agent’s output for function names and option names, then grep the actual codebase or library for them. The mismatches are obvious. The agent might have hallucinated findOrFailWithLock when the real method is findOrFail and there’s no lock option at all.

The fix has two parts. Short-term, tighten the harness with documentation links or type signatures pointing at the real reference is the cheapest cure. Longer-term, give the agent tools instead of documentation. An MCP server that exposes the library’s actual API (so the agent can call it instead of guessing) ends the hallucination class. A type checker run as a sensor (an automated check wired into the harness that fails the run when something’s wrong) catches the hallucinations that slip through.

The recurrence pattern worth noting: hallucinations cluster around APIs the model is confidently familiar with from training but where the project has chosen to use the library differently than the common convention. The model isn’t making up the function from nothing; it’s substituting the public-API version when the project actually wraps it.

Scope creep

The agent changed more than it was asked to. The task was fix a bug in the login flow. The diff also reformatted three unrelated files, renamed a helper for clarity, and updated four imports because the rename touched them. The bug fix is correct. The reformatting is harmless. But the PR is now many times the size it should be, and the reviewer’s eye is going to slide past the actual fix.

How to spot it in the log: look at the Edit and Write tool calls. Compare the files touched against the files the task description called out. The gap is the creep.

The fix is a rule and a sensor. The rule says touch only what you must, clean up only your own mess. The sensor is a pre-PR check that fails if the diff is wider than a budget; N files, M lines, whatever your numbers happen to be.

Scope creep is one of the most common failure modes in my experience and the most invisible. It doesn’t break anything. It just makes every PR worse to review and every regression harder to track down.

Premature commit

The agent committed before the work was done. The tests pass because the agent skipped writing the failing test first. The acceptance criteria look met because the agent declared them met without verifying. The PR description claims feature X is implemented when, under the hood, feature X is a stub.

How to spot it in the log: look for the moment the agent decided the work was done. Compare against the actual evidence of doneness. Did the agent run the test it claimed proves the fix? Did the test exist before the change, or did the agent write it after seeing the change pass? Did the agent verify the acceptance criteria, or did it skip the verification step and write the PR description from memory of what the task asked?

The fix is procedural. A skill called finishing-work that walks the agent through the verification steps before allowing a commit. Perhaps a pre-commit hook that fails if the changed files don’t have at least one paired test change. An agent that has to produce a completion report (see my earlier post on post-mortems for agent runs) before declaring done.

Premature commit is the failure mode where the agent looks most productive and is least useful. The PR ships, the work isn’t finished, and the next person to touch the code finds out.

Reading the log: a worked example

Here’s a transcript fragment from a real run I investigated. The task was to add a new field to a user profile.

> User: Add a `pronouns` field to the user profile. Optional string, nullable.
[Read .claude/CLAUDE.md]
[Read app/Models/User.php]
[Glob database/migrations/*_users_*.php]
[Read database/migrations/2024_05_12_add_email_to_users.php]
[Write database/migrations/2026_06_18_add_pronouns_to_users.php]
[Edit app/Models/User.php] // added 'pronouns' to $fillable
[Bash: php artisan migrate - pretend]
[Edit app/Http/Resources/UserResource.php] // added pronouns to toArray
[Bash: vendor/bin/pest tests/Feature/UserResourceTest.php]
[Write app/Http/Controllers/UserController.php] // added validation rule

The PR shipped. The reviewer signed off. Production threw an error the next morning, from the admin panel: Undefined array key 'pronouns'. The admin user view read $user['pronouns'] straight off the resource array, and the resource never put it there.

Reading the transcript: the agent loaded the project’s CLAUDE.md, looked at the User model, looked at a recent migration as a template, and wrote a similar migration. It also added the field to the User resource and the controller validation. It did not read app/Http/Resources/AdminUserResource.php, which the admin panel uses and which lists its fields explicitly, key by key. Unlike the model’s $fillable, the admin resource doesn’t auto-pick up new columns. A field exists there only if someone names it. The agent never opened the file, so pronouns was never added, and the admin view that assumed the key blew up.

The failure mode: missing context. The harness had no rule that said user-profile fields appear in three places: User, UserResource, AdminUserResource. The agent followed the convention from the two places it knew about and shipped a partial change.

The fix wasn’t be smarter, agent. The fix was a path-scoped rule in app/Http/Resources/CLAUDE.md that named the resource files that needed to stay in sync. The rule went in. The sensor (a quick AST check that any new $fillable field appears in both resource files) went in alongside. The next pronouns-shaped task didn’t repeat the bug.

That’s the loop. The transcript pointed at the gap and the harness change closed the gap. The model wasn’t the variable.

What forensic reading teaches you about your harness

The payoff of reading transcripts isn’t fixing the specific bug. It’s the patterns that show up after twenty or thirty of them.

You start noticing which files the agent rarely reads, and asking why. You notice which rules it cites a lot (they’re load-bearing) and which it never cites (they’re dead weight). You notice the directories where transcripts go wrong most often — usually the directories with the worst seams, the least context, or the most conventions that aren’t written down.

There’s a cost the failure modes hide from each other. The fix for missing context is almost always a new rule and every rule you add is a future candidate for contradicted-rules rot. Fix enough gaps by piling on instructions and you build the exact stale-harness problem that produced failure mode two. So the audit cuts both ways: you read transcripts to find the rule you’re missing, and you read them to find the rule that’s now lying. A harness isn’t a pile of rules that only grows. The transcripts tell you which rules to add and which to retire, and the second list matters as much as the first. The transcript is the highest-resolution feedback signal the harness produces.

Two practical reading habits help.

Read at least one failed transcript a week. Not all of them. One. Pick the one that produced the most surprising failure. Read it end to end. Note the failure mode. Ask whether the harness has a gap there. The discipline isn’t to fix everything. Rather, it’s to develop the muscle.

Keep a transcript notebook. A file in the repo. Each entry: link or paste of the transcript, the failure mode (pick from the five above), the gap in the harness, the change that fixed it. After ten entries, the notebook becomes a map of where your harness is weak.

The notebook doesn’t have to be pretty. It has to exist.

The thing the model isn’t responsible for

A theme runs through every failure mode above. You can argue each one back to “the model should have known better”. But arguing about whose fault it is doesn’t change the rate. Changing the harness does. Missing context is a closable harness gap. Contradicted rules are stale rules. Hallucinated APIs are weak documentation or missing tools. Scope creep is an unset constraint. Premature commit is a missing procedure.

This isn’t a defense of the model. The model is fallible, and the next generation of models will be more capable than this one. The point is that the cheap improvements live in the harness, not in the model. You can’t retrain the model from your laptop. You can fix a rule, scope a rule, add a sensor, sharpen a tool, and write a procedure. All five are at your fingertips. All five reduce the rate of the same bug shipping again.

Forensic reading of transcripts is the practice that tells you which one to fix.

Read one transcript this week

Find the last commit your agent made that you wish it hadn’t, or the last PR that came back with a “this isn’t quite right” review. Open the transcript that produced it.

Read the tool calls first. Note which files the agent loaded and which it should have loaded. Find the rule it cited or skipped. Cross-check the agent’s claims against the actual code. Pick which of the five failure modes you’re looking at.

Then write the harness change that would have stopped it. Whether a new rule, a path-scope, a sensor, a skill or a tool. Ship it the same day.

After three or four passes, you’ll start seeing the same failure mode keep recurring. That’s the signal. The pattern is the rule waiting to be written. The transcript is just the place the pattern lives until you spot it.

The model can’t read its own transcript. You can.

What Breaks When You Skip the Harness

Ian Johnson — Tue, 23 Jun 2026 14:47:49 +0000

The agent kept running the tests, watching them go red, scrolling up to find the failure, and then running the tests again because it had already lost the output. Third time in one session. The model wasn’t the problem. The session’s memory of its own output was. I had Claude tee the test command to a log file. After that it read the log instead of re-running.

That was one fix on one project. The pattern repeats. If the model keeps producing bad code, what is actually broken?

The model isn’t the problem

Output quality is the sum of two things: the model and the surroundings.

In the teams I’ve worked with, the obsession is always the model. They upgrade. Switch tools. New IDE plugin every month. They read every benchmark. The defects keep landing in the same places.

The surroundings get almost no attention. Those surroundings are the harness: the files, rules, tools, and feedback loops that wrap the model inside a real project. A CLAUDE.md with conventions. Skills — codified workflows the agent can run by name. MCP servers. Scoping rules. A feedback.md that records what the team has corrected. Pre-commit gates.

Without those, the model is guessing.

Five things teams blame on the model, each pointing at a missing piece of the harness, each with a first move you can ship this week. Some of the bigger fixes (codifying a team’s lore, writing a useful project-specific MCP, etc.) are quarter-of-work to do properly. The first move is week-sized. The first move gets you started.

APIs that don’t exist

The model writes a call to client.users.list({ since: lastSyncAt }). The argument doesn’t exist. The real API takes a Unix timestamp on a different endpoint. Code compiles. Test fails. The engineer reads the docs. The bug gets filed as “the model hallucinated.”

Except the model didn’t hallucinate from nothing. It wrote what it learned from training data and didn’t verify. Nothing in the session told it to.

The fix is grounding. Give the model a way to read the real docs in the moment.

Two grounding moves do the work.

First: an MCP server for library docs — Context7 is one. The model fetches current docs and writes against what’s there. When it fails, it’s almost always because the rule telling the agent to fetch first wasn’t in CLAUDE.md.

Second: a project-specific MCP for internal APIs. Every team I’ve worked with has one or two services nobody got around to documenting well. An MCP that serves the OpenAPI spec — or a search over the service’s source — puts real signatures in front of the model.

A single rule in CLAUDE.md ties it together. “Fetch docs before writing library code.”

Smallest first move: add Context7. Add the rule. Try it on the next library task.

Code that doesn’t match yours

The model writes a new service. The file lands in the wrong directory. Errors throw instead of returning. The naming is camelCase in a snake_case codebase. The diff looks foreign.

The reviewer sees it and rewrites it. The next PR has the same problems.

The model has no reference for “how we do it here.” The rules live in three engineers’ heads and one out-of-date README.

Part one is a CLAUDE.md that names the project’s conventions in plain words. Where files go. How errors propagate. Naming style. Testing framework. Logger. Time library. One or two sentences each. The bare minimum.

Part two matters more: worked examples.

When we moved a service to hexagonal architecture, the agent kept writing new code on the wrong side of the ports-and-adapters boundary. The old code was still everywhere and gave it permission to keep doing what it had always done.

The model copies texture from examples better than from prose. A skill or a feature doc with one full worked example , such as a real PR, a real file, or a real test, gives it something to imitate. Three examples beat a thousand words of rules.

The catch: if the codebase itself is inconsistent (legacy module is snake_case, new code is camelCase), the model picks up the contradiction and writes both. The worked example has to come from the side of the codebase you want the agent to copy.

Smallest first move: pick the workflow your team runs most often. Write one skill for it. Drop one full example into the file.

The same bug, in every session

You correct the agent on Monday. It used the wrong logger. You tell it to use the project’s logger. It does. Friday, new task, wrong logger again.

The agent didn’t learn. The session ended. The correction went with it. The fix is a feedback loop the agent can read. Not a chat log, but a file.

Create a feedback.md. When the team corrects the agent on something that’s a rule, not a typo, write it down in three lines. Context. Mistake. Rule going forward. Tell the agent to read it before every relevant task.

The first entry in mine was about flaky E2E tests. The team’s rule was fix them or delete them. Never live with them. The agent kept patching around them instead. After the third time I corrected it I stopped repeating myself and wrote the rule down.

Who writes to it: anyone on the team. The cost of adding an entry has to be smaller than the cost of fixing the same thing twice. Otherwise nobody writes.

The promotion rule: when the same correction shows up in feedback.md three times, lift it into CLAUDE.md or into a skill. Three is arbitrary, pick a threshold and stick with it. I chose three because of the rule of three. The point is that nothing useful stays buried in the feedback file forever. Every promoted entry is one more rule the model reads at the start of the next session.

Smallest first move: create the file. Write one entry today. Add one line to CLAUDE.md that says read it first.

The knowledge stays in heads

A senior engineer is on a call, explaining why the payment service has its own retry logic that is different from the rest of the system. The new engineer nods. The agent in everyone’s terminal doesn’t, because the agent wasn’t on the call.

Two weeks later the agent writes a new payment integration. It calls the shared retry helper. It has no idea there’s a reason not to. The PR ships. The on-call gets paged.

This is tribal knowledge. The lore of the system. It lives in Slack scrollback, in senior engineers’ heads, in a Notion or Confluence page.

The fix is codification. Three kinds of files do the work.

A glossary captures domain vocabulary. The word “session” can mean three different things in one system — auth, user, analytics. The glossary picks one definition per term and points to it. One paragraph each.

A feature doc holds the why of a part of the system. Why payments has its own retry logic. Why we picked Postgres over MySQL. Why the queue uses Redis Streams and not Kafka. Short notes a new reader can scan in a minute.

Skills hold workflows. The steps to ship a database migration. How we add a feature flag. The onboarding checklist for a new service joining the monitoring stack.

The hard part of this fix is not the format. The hard part is getting the senior engineer to write the feature doc instead of explaining it on calls. That’s a social problem, not a documentation problem. Pair on the first one. Make the doc the deliverable.

A billing module on one of my projects had quiet invariants about how transaction state combined. The agent made an arithmetic change that looked clean in the diff. It would have produced wrong totals in production. The reason the change mattered wasn’t in the code. It was in the team’s heads.

Smallest first move: pick the one piece of lore most often re-explained to new engineers. Write it down as a feature doc this week. Reference it in CLAUDE.md.

Reviewers as the harness

Every PR re-litigates the same things. Naming, test coverage, error handling, conventions, and so on. Same three reviewers, same six comments. PRs take two days to merge. The reviewers are tired.

The reviewers have become the harness. Humans are doing what a script should do.

The fix is gates the machine can run.

A pre-commit hook catches the same six review comments before the diff exists. Match the gate to the team’s actual repeated comments. It could be naming-convention nits that needs a linter rule, missing-test-file nits that needs a coverage gate, or risky-file-changed nits that needs a CODEOWNERS-style script. The point is to read the team’s review history and turn the top three repeat offenders into machine checks.

A review skill or review agent reads the diff and flags the things a linter can’t see.

A code-health check (CodeScene is what I use) gives a numeric score for maintainability and blocks regressions. The score settles the argument over whether a diff makes the code worse in seconds.

The worst case I’ve watched was naming inconsistencies during a migration. Two valid styles in the codebase, the model picking both, sometimes in the same PR. We added sensors that flagged the old style in new files and ran them in CI. The drift stopped showing up in review.

The work reviewers should be doing (design, intent, what the PR is actually trying to do) gets crowded out by everything machines could catch and don’t.

Smallest first move: pick the review comment your team has left most often this month. Write one pre-commit hook for it. Push it in a PR this week.

Write three rules this week

The model isn’t the problem. The surroundings are the work.

Three things to do this week.

Open CLAUDE.md. Write three rules: the three things you’ve corrected an agent on this week. That’s the start of a harness.
Create feedback.md. Write one entry. Tell the agent to read it. The feedback loop now exists.
Pick one workflow you’ve run through an agent more than three times this month. Make it a skill. The third time you reach for it, it’s already written down.

The harness rots too. CLAUDE.md drifts as the project changes. Skills go stale. feedback.md swells to four hundred entries and turns into noise. Two skills contradict each other and the agent picks one at random. Most teams who tried a harness and quit do so at the rot stage, not the setup. Treat it like the code it wraps: audit on a schedule, delete what no longer applies, promote what has earned its place.

This isn’t the only thing that matters. A great model still beats a mediocre one on hard problems. I don’t have a clean benchmark for the bigger claim. What I have is a preference: given the choice between a great model in a bad harness and a mediocre model in a good one, I take the mediocre model every time. The model’s mistakes I can debug. The drift I cannot.

Watch for two weeks. The defects don’t vanish. Instead they show up earlier at pre-commit, in the feedback file, or in the review skill instead of in CI or production. That’s the win.

Tools Are Harness Too

Ian Johnson — Mon, 22 Jun 2026 15:30:29 +0000

I had a rule in my project CLAUDE.md for months that said always check Code Health before committing. The agent followed it about 70% of the time. The other 30%, it forgot; usually on the longer tasks, when its context window was filling up and the rules at the start of the prompt were losing weight against the code at the end. I added emphasis. The 70% became 80%. I added a section break. 85%. I rewrote the rule in capital letters with three exclamation points. Same 85%.

Then I gave the agent a Code Health tool. Not a rule. A tool — an MCP server entry point called code_health_safeguard that returned a structured analysis of the staged diff. The next day, the agent ran it before every commit. The day after that. The day after that. The compliance went to 100% and stayed there.

The difference between a rule and a tool is the difference between telling the agent what to do and making the right thing easier than the wrong thing. Both belong in a working harness. Tools are the half nobody talks about.

Rules tell, tools enforce

A rule is a paragraph. The agent reads it, weighs it against the other things in its context, and produces output. If the rule wins the weighing, the agent follows it. If something else wins (the code’s existing pattern, a more recently mentioned constraint, the user’s apparent intent in the prompt) the rule loses.

The probability the rule wins is not 100%. It is some number less than 100%, and the number drops as the context grows. Rules in long-running agent sessions are like introductory remarks at a long meeting: present, recorded, mostly forgotten by hour three.

A tool is different. A tool is an affordance: a thing the agent can do, exposed at the same level as everything else the agent does. The agent’s working surface is I can read files, edit files, run shell commands, and call these specific functions. A tool slots into that list. The agent doesn’t have to weigh a paragraph against another paragraph; it sees a function and decides whether to call it.

The decision shifts from should I obey this rule to should I use this capability. The shift matters because the decision happens at a different layer in the agent’s reasoning. Capability decisions are cheaper than rule-weighing decisions. They survive the long-context drift.

When a tool beats a rule

Not every rule should become a tool. The translation has a cost. Tools take time to write, they introduce new surface area to maintain, and they only earn their place when they replace a rule that fails often enough to be worth replacing.

A rule belongs as a tool when:

The rule produces a check the agent can run. Always verify the schema before writing a migration is a checkable property. Be thoughtful about naming is not.
The check has a deterministic output. Run the linter and pass is a yes-or-no answer. Match the team’s style is a vibes call.
The check fails silently when skipped. A rule the agent can ignore without anyone noticing is a rule that begs to become a tool. The tool is the gate that closes the silent skip.
The check is run often. A check the agent should run on every commit is a tool. A check the agent should run on the rare cross-project migration is a skill or a rule.

For everything else, rules are still the right unit. Tone, naming conventions, design preferences, when-to-use-X-vs-Y judgment calls: those are paragraphs, not function calls. Trying to encode them as tools is the path to a brittle harness with thirty checks and no taste.

The Code Health example, end to end

To make the trade-off concrete, here’s the rule-to-tool conversion that started this post.

The rule, in CLAUDE.md:

## Code Health is authoritative

Before suggesting a commit, run a Code Health check on the changed files.
If any file regresses below 9.0, refactor before committing. Do not commit
code with a Code Health below the project floor.

The rule is fine. It’s specific, it names the threshold, it tells the agent what to do. It just doesn’t fire reliably under load.

The tool is the CodeScene MCP server’s pre_commit_code_health_safeguard. It exposes itself to the agent as a callable function. The function takes the staged diff and returns a structured response — files checked, scores before and after, regressions flagged, suggested refactorings. The agent doesn’t have to remember to do the right thing; the right thing is one tool call away, and not calling it now looks conspicuous next to the call to git diff --cached that the agent makes anyway.

The rule still lives in CLAUDE.md, with one important change. It now says call pre_commit_code_health_safeguard before suggesting a commit. The rule names the tool. The tool does the work. The rule is shorter and the agent’s compliance is higher.

That’s the conversion shape: a rule that names a tool, and a tool that does the check the rule used to describe. Both layers are present. Each is cheaper than it would be alone.

Tools as constraint design

The other reason tools matter is that they let you design constraints that rules can’t express.

Consider the constraint the agent should never delete production data. As a rule, it’s a sentence: don’t run destructive operations against production. The rule is correct and entirely advisory. As a tool, it’s the inverse. The absence of any production-database-write capability in the tool list at all. The agent literally cannot call what isn’t exposed.

This is the difference between advisory constraint and constraint by construction. Rules are advisory. Tools are construction. The harness designer chooses where on that spectrum to put each constraint.

The rule of thumb I use: constraints that are critical to safety belong as constraints by construction. Tools the agent shouldn’t use shouldn’t be in the tool list. Permissions the agent shouldn’t have shouldn’t be granted. The advisory layer (rules) is for guidance; the construction layer (tools, permissions, capabilities) is for safety.

When I’m reading a harness, the first place I look isn’t the rules. It’s the tool list and the permission allowlist. The tool list tells me what the agent can do. The rules tell me what it should do. The safety properties live in the first answer, not the second.

What an MCP server gives you

MCP (the Model Context Protocol) is the standard most modern agents use to expose tools. The mechanics are simple: a server runs somewhere (often as a subprocess of the editor), declares a set of capabilities to the agent, and handles requests when the agent calls them.

The capabilities are arbitrary. They can wrap a CLI, an HTTP API, a database, a local file system, a third-party service, or pure computation. Whatever can be wrapped becomes a tool.

A few of the MCP servers I find load-bearing in my own work:

CodeScene. Code Health checks, refactoring guidance, hotspot detection. Replaces a long rule about complexity with a tool that gives concrete numbers.
Context7. Real-time library documentation. Replaces the failure mode where the agent hallucinates an API by giving it a way to look up the actual one.
The project’s own MCP server. Custom checks the team cares about. A “show me the current schema” tool. A “validate this against the API spec” tool. A “list the existing repositories” tool.

The third one is the one most teams skip and shouldn’t. The project’s own MCP server is where the team’s domain constraints get encoded as capabilities. A validate_contract tool, a verify_acceptance_criteria tool, a check_naming_convention tool — each takes a rule that used to live in CLAUDE.md and turns it into something callable.

The tradeoff: tools cost more

The honest cost of tools is that they’re harder to write than rules. A rule is a paragraph in a markdown file. A tool is a function with a typed schema, error handling, an installer, and a test surface. Writing a good tool takes an afternoon. Writing the rule it replaces takes ten minutes.

The afternoon pays back, but only for the right tools. The rules to convert are the ones with high cost (the rule has been violated multiple times, each violation has a real cost) and high check-ability (the check is mechanical). The rules to leave alone are the ones with low cost or low check-ability. They’re cheaper to keep as rules and live with the occasional miss.

There’s a second tradeoff. Tools accumulate. Every tool added to the agent’s surface is another option it considers. A tool list that grows to fifty capabilities slows the agent’s decision-making and pushes the relevant tools further out of immediate reach. Tools, like rules, deserve their own audit cadence in the same way rule-rot eats at rules, tool sprawl eats at agents.

Keep the load-bearing tools. Delete the ones the agent never calls. Watch the same pattern as the rules: which tools earn their slot, which sit unused.

Tools as the third layer of the harness

If rules are the first layer and sensors are the second, tools are the third. Each layer does work the others can’t.

Rules communicate intent and shape the agent’s reasoning. They’re cheap. They’re advisory.
Sensors detect when intent isn’t followed. They’re checks at the workflow’s edges. They’re load-bearing.
Tools make right things easy and wrong things impossible. They’re capabilities. They’re construction.

A working harness uses all three. The rule says do X. The tool makes X a callable capability. The sensor verifies X happened. Each layer has a different cost and a different power. The mistake is to load everything into one layer because that layer is comfortable.

The teams I see with rules-only harnesses ship the same bugs over and over with great-sounding documentation. The teams I see with tools-only harnesses ship code that compiles but doesn’t match the team’s conventions. The teams I see thriving have a balance — rules for the guidance, tools for the capability, sensors for the gate.

The seams between rules and tools

The most powerful arrangement is when a rule references a tool by name. The rule loads into context, communicates the intent, and tells the agent which tool to reach for. The tool, when called, produces the deterministic answer the rule was hand-waving toward.

## Verifying schema changes

Before writing a migration that adds, drops, or alters a column, call
`db_schema_dump` to get the current schema. Confirm the columns you reference
exist and are typed as you expect. Cite the schema dump in your reasoning so
the reviewer can see you verified.

That’s a rule paired with a tool. The rule does what rules do: it explains the intent, names the situations it applies to, and tells the agent how to handle them. The tool does what tools do: it provides the actual capability — fetching the current schema, in a structured form, deterministically.

The pair is stronger than either piece alone. The rule alone would be ignored on long contexts. The tool alone wouldn’t know when to be called. Together, they cover the case.

Convert one rule to a tool this week

Pick a rule from your CLAUDE.md that names a check. Always run the linter. Verify the schema first. Check the test coverage. Anything that says always X where X is mechanical.

Look at whether there’s an MCP server that already does X. If so, install it, expose the tool to the agent, and rewrite the rule to name the tool. If not, look at the size of X. If it’s small (a shell command, a script), write a one-tool MCP server that exposes the script. Most agent harnesses make this surprisingly easy now.

Ship the tool. Update the rule to name the tool. Watch the next ten agent runs. The rule that was complied with 80% of the time becomes 100% almost overnight, because the agent isn’t being asked to remember anymore. The capability is right there.

Rules tell. Tools enforce. The harness with both is the harness the agent doesn’t drift out of.

Harness Engineering — Building Reliable Workflows Around Non-Deterministic Agents

Ian Johnson — Sun, 21 Jun 2026 22:55:28 +0000

Six months ago this was a blog post. Then a second blog post. Then Bridle, Sellier, intent-driven-delivery, and eventually Keystone.

Now it’s a book: Harness Engineering — Building Reliable Workflows Around Non-Deterministic Agents.

What surprised me writing it: the experiments earned their place. Bridle taught me the flywheel. Sellier taught me defaults beat flexibility. IDD taught me the team layer can be reified. Keystone is the synthesis they pointed at.

It’s on LeanPub now. Pay what you want, starts at $10.

https://leanpub.com/harness-engineering

Sensors: The Other Half of the Harness

Ian Johnson — Sat, 20 Jun 2026 15:15:50 +0000

The pre-commit hook caught a migration last Tuesday that would have shipped to staging green and broken production on the next deploy. The migration dropped a foreign key constraint without naming a backfill plan, and the rule that said every constraint change needs a backfill plan had been sitting in migrations.md for four months. The rule didn’t stop the agent. The rule didn’t stop the human reviewer either. The diff was 38 files and the constraint drop was a single line. What stopped it was a four-line shell script wired to pre-commit that grepped the staged migration for DROP CONSTRAINT and exited non-zero if no--backfill: comment followed it.

That four-line script is what I want to talk about. Most of what gets written about agent harnesses is about rules — what CLAUDE.md should say, where to put it, how to scope it, when to version it. Rules are half the harness. The other half is the set of checks that fire when a rule gets broken. I call them sensors, and they get talked about a lot less than they should.

The asymmetry in the conversation

Read any of the well-circulated posts on agent harnesses and count the words spent on each side. Rules are the centerpiece. Sensors are glanced over, as if they’re so obvious they don’t deserve their own treatment. They aren’t obvious, and the assumption that they’re already there is the most expensive assumption in the field.

The asymmetry has a reason. Rules are easier to write. A rule is a paragraph. A sensor is a script, a hook, a CI step, a pre-commit config, a custom check. Rules sit at the cognitive level a writer naturally operates at; sensors sit at the level of plumbing. The first one is a lot more fun to write than the second.

The asymmetry also has a cost. Rules without sensors are vibes. The agent reads them, claims to follow them, and gets graded on whether the resulting code looks like it followed them. Looks like is the failure mode. The agent is a probabilistic system. It will obey a rule most of the time and skip it some of the time, and the times it skips are the times the rule mattered most: the awkward branch, the migration nobody likes to think about, or the edge case the rule was written to catch.

A sensor changes the contract. The rule is no longer a hope. It’s a falsifiable check.

What a sensor actually is

A sensor is anything that can detect whether a rule was followed and signal that detection in a way the workflow respects. The shape is narrow:

It runs deterministically.
It returns a clear pass or fail.
It fires at a point in the workflow where its result still matters.
It’s cheap enough that nobody is tempted to skip it.

Most of the sensors in a working harness aren’t fancy. A grep with set -e. A line in a linter config. A pytest fixture that asserts the database fixture hasn’t drifted. A pre-commit hook that runs mypy on changed files. A GitHub Actions job that fails the PR if the migration directory has a file without a paired rollback. None of these is impressive on its own. The point isn’t the individual sensor; it’s the discipline of having one for every rule that matters.

Five places a sensor fires

There are five workflow positions where sensors earn their pay. Each catches a different class of failure. A working harness has sensors at most of them.

Pre-edit

The earliest sensor is the one the agent encounters before it writes code. The rule is loaded into context; the sensor is something the agent can run to check its own work before producing output. A type stub generator, a schema dump, a --dry-run command that simulates the change. The agent doesn’t need approval; it just needs to see what its proposed change would do.

Pre-edit sensors are the rarest of the five, because they require giving the agent tools, not just rules. The pay-off is that they catch errors before any bytes hit disk. The agent that can run prisma validate on a proposed schema change is going to make fewer broken commits than the agent that can only read the rule that says make sure your schema is valid.

Pre-commit

Pre-commit is where most teams put their first sensor, because Git makes it easy. A .pre-commit.config.yaml or a husky hook fires the moment the agent (or human) tries to land a change. The check has the staged diff to look at, the rest of the repo for context, and a hard exit code that the workflow respects.

This is the sensor layer for rules that constrain the artifact — what the code looks like, not what it does. Format, lint, type, dead code, banned imports, naming patterns, the migration backfill check above. Cheap, fast, local. The agent that breaks a pre-commit hook either fixes the violation or doesn’t commit. Either outcome beats the agent commits and nobody notices.

Pre-PR

A pre-PR sensor is one that runs after the commits exist but before the diff goes up for review. The check has more to work with: a full branch, a base ref, a set of changed files that’s no longer one commit at a time. It’s the right layer for cross-file checks. Did you change this API without updating its consumers? Did you add a new migration without bumping the schema version? Does the test coverage diff drop more than two points?

Pre-PR runs cheap on CI and slow locally. Most teams run it on the branch push, which catches the agent’s work right as it’s offering the work for human attention. That timing matters. A check that fires after the human has started reviewing is a check that’s been skipped.

Post-merge

Post-merge sensors are the ones that run against main. They aren’t strictly part of the agent’s loop, but they’re part of the harness because they detect when the loop produced something that broke once integrated. Smoke tests, end-to-end suites, schema-diff jobs that compare staging to production, query-plan monitors that fire when a slow query joins the rotation.

The instinct is to think of post-merge as CI and not as part of the harness. That’s a mistake. The agent’s behavior is shaped by what fails after it shipped. A post-merge sensor that catches a regression and gets traced back to a missing rule is the most valuable kind of sensor in the whole stack. It tells you something the inner sensors missed, and it tells you what to add upstream.

Drift

The fifth and most-skipped sensor is the one that checks whether the rules themselves are still true. The codebase moves. The framework version bumps. The pattern the rule used to describe gets replaced by a new pattern. The rule is now wrong, and nobody notices because nothing fires.

A drift sensor is the check that wakes up periodically and asks: does this rule still match reality? Sometimes that’s a literal grep for the pattern the rule references. Sometimes it’s a count of how many files in the codebase still match the rule’s example. Sometimes it’s a last-modified audit that flags any rule older than six months for review. The cadence is monthly or quarterly; the goal is to keep the harness honest as the code beneath it changes.

I’ll write a whole post about rule-rot soon. The short version: a drift sensor is the difference between a harness that’s six months old and helpful and a harness that’s six months old and lying.

The pairing rule

If a rule matters, it has a sensor. That’s the principle. It sounds rigid, and it is, on purpose.

The discipline works because it forces a real question every time a rule gets written: how would I know if this got violated? If the answer is I’d see it in code review, the rule is on probation. Code review is slow, expensive, and inconsistent. It is also the wrong layer for rules that produce mechanical violations.

If the answer is the linter would catch it, the rule probably doesn’t need to be a rule. The linter is the rule. Just configure the linter and skip the prose.

The interesting cases are the rules where the answer is nothing currently checks for this, but I could write a check. Those are the rules that earn a sensor. The sensor doesn’t have to ship the same day the rule ships, but it should be on the list and have a date.

The rules that aren’t worth a sensor are the rules that aren’t worth writing down. Be thoughtful is not a rule. Bullet points should be sentence fragments unless they form a list of full sentences is also not a rule, because nothing catches a violation and the cost of writing it down is more than the cost of fixing the rare slip in editing.

A rule with no sensor is either too vague to matter or important enough that someone should pair it within the month. Either way, the absence of a sensor is information.

What good sensors share

The sensors that survive in production look alike. They share five traits.

They run in seconds. A pre-commit hook that takes 90 seconds gets bypassed within two weeks. A pre-PR check that takes 12 minutes gets the --skip-checks shortcut added to the team Slack. The check has to be fast enough that running it is cheaper than figuring out how to avoid running it.

They produce clear messages. A failed sensor that prints EXIT 1 is a failed sensor that’s about to get ignored. A failed sensor that prints migrations/20260620_drop_users_index.sql dropped an index without a paired backfill comment. Add a comment starting with --backfill: explaining the plan, or annotate the migrations with --no-backfill-needed: is a sensor that teaches the agent (or the human) how to fix the violation.

They run locally. A check that only exists in CI is a check that the agent finds out about ten minutes after it pushed. Running the same check locally before push closes the loop. The pre-commit framework and npm run check-style scripts both make this easy; the discipline is to make sure CI runs the same scripts the developer can run.

They cite the rule they enforce. Every sensor message that earns its weight names the rule. *Failed: rule MIGRATIONS-04 (foreign-key changes require a backfill plan).* The agent reads the message; if the rule is wrong, the agent can find it and propose a change. The sensor is the gate, but the rule is the explanation.

They fail loud and pass quiet. A sensor that prints OK checked 47 migration files in 1.2s every commit is a sensor whose output everyone learns to ignore. Pass silently. Fail with a paragraph. Save the human’s attention for the cases that need it.

Where sensors live in the project

A working sensor layout has four physical homes. Each holds the sensors for a different workflow position.

.husky/ # or .pre-commit-config.yaml
  pre-commit # fast, staged-files only
  commit-msg # commit message linting
.github/workflows/
  pr-checks.yml # pre-PR, runs on push
  post-merge.yml # post-merge, runs on main
.claude/
  hooks/ # agent-facing pre-edit checks
  sensors/ # one shell script per drift check
scripts/check-*.sh # the actual checks, invoked from above

The shape isn’t sacred. What matters is that each sensor has a single home and the workflow knows where to call it. The anti-pattern is the same check running in three places with slightly different logic and the team unable to remember which one is authoritative.

The sensors themselves go in scripts/ (or wherever your project keeps tooling) and get called by whichever runner needs them. That separation means the same check can fire from pre-commit, from CI, and from a drift-audit cron without three copies. One script, three invocations.

Personally, I prefer to wrap these scripts in make tasks. The benefit that I receive is that the agent always knows to use make. And that makes everything easier to find and more consistent. I use make in CI too. Usually, I work with docker and so this is a natural pair for me. Isolate the environment with docker; isolate the commands with make. That makes it easier for the human and the agent to work with the repository.

Two sensors worth writing this month

The two highest-value sensors for a team starting from zero are both small.

The migration safeguard. Whatever your migration tool is, the team’s hardest-to-catch bugs ship through it. A pre-commit script that knows the patterns your migrations are supposed to avoid — DROP COLUMN without paired backfill, ADD COLUMN NOT NULL without a default, schema-breaking renames without a two-phase plan — catches more production issues than any rule on its own. The agent will write migrations that look right and miss the constraint. The sensor sees what the human reviewer doesn’t.

The diff-budget gate. A pre-PR check that fails if the diff exceeds N files or M lines. The number is yours to pick. I use 15 files and 400 lines, which roughly maps to a PR a human can review in one sitting. The rule that says keep PRs small is exactly the kind of rule that gets nodded at and then drowned in the next month. The sensor that fails the build at file 16 is the rule with teeth.

Both are under fifty lines of shell. Both pay for themselves the first week.

The anti-patterns to watch

A few sensor failure modes recur often enough to name.

The unrunnable sensor. It lives in CI, takes 14 minutes, and depends on a secret only the deploy bot has. Nobody on the team can run it locally. By the time it fires, the human reviewer has already mentally signed off. The fix: factor out the local-runnable piece and run that on every commit.

The sensor that always passes. Someone wrote it months ago, the inputs changed, the assertions still hold, the failure case is no longer reachable. The sensor is a green light glued to the dashboard. The fix: every sensor needs at least one known-bad fixture that the test suite runs against it to confirm it still detects the violation it was written for.

The flaky sensor. It fails sometimes and passes the rest. The team learns to rerun. Within a month, the rerun habit has spread to the other checks, and a real failure gets bypassed because it looked flaky. The fix: a flaky sensor is broken. Pull it from the gate, fix it, or delete it. Do not leave it firing.

The rule with no sensor. The whole point of this post. A rule that lives in CLAUDE.md and has no check is a rule that gets followed when the agent feels like it. The fix is the pairing rule above: if the rule matters, pair it within a month.

The sensor with no rule. Less common but worth naming. A check fires, a developer doesn’t know why, the rule it’s enforcing isn’t written down. The check is correct on its own terms but unteachable. The fix: cite the rule in the failure message. If you can’t, write the rule.

Pair one rule to a sensor this week

Pick one rule from your CLAUDE.md — the one you find yourself reminding the agent about most. Open a file. Write a check. Wire it to pre-commit. Ship it.

The right rule for the exercise is one with a clear violation pattern. Always use the Result type for fallible operations is a fine candidate; you can grep for throw in the changed files and warn. Match the existing code style is a poor candidate; the violation pattern is too diffuse for a small sensor to catch.

When the sensor catches its first violation, two things happen. The rule becomes load-bearing. The agent now has to satisfy it instead of just acknowledge it. Additionally, you find out whether the rule was right. Sensors that fail a lot on changes that look reasonable in review are sensors enforcing rules that need to change. The sensor is the feedback loop; the rule is the hypothesis.

Add a second one next month. By the end of the quarter your harness will feel different. Not because the rules are smarter, but because the rules can be checked.

The rules are the part everyone writes. The sensors are the part that makes them true.

The Accidental Framework

Ian Johnson — Fri, 19 Jun 2026 15:23:25 +0000

The thing I built was a Go binary that copied markdown files into a repository. I called it an installer. The README called it an installer. The CLI verb was init. For the several months in early 2026, that framing held. Then I sat down to plan the 1.0 release, started typing, and could not write the next paragraph without using the word framework. Not in the marketing-blurb sense. In the load-bearing sense: a small runtime, a set of named extension points, conventional file layouts, a resolver that picked a winner among competing sources, a lockfile, a migration runner, per-backend adapters. The thing I had been calling an installer for four months had been doing framework work the whole time, and I just had not noticed.

This post is about that moment of recognition and what came after. It is about the refactor that turned an installer-shaped framework into a framework-shaped framework. The most surprising part was not the rewrite. It was how little of the concepts had to change. The names were already right. The directories needed to move. The runtime needed a physical line down the middle between framework code and content. But the abstractions I had been using (guides, corpus, sensors, actions, playbooks, adapters) held up under the new framing without revision. The work I had done naming things over the previous months turned out to be the work that made the refactor cheap.

I want to walk through how that happened, what tipped me off, and the one wrong turn the refactor almost made before backing out of it. The story has a tidy ending; the lesson does not, and I will get to that.

The shape was there before the word

Keystone started as keystone init — a binary you ran in a fresh repo and it laid down a harness/ directory full of markdown. Rules for the agent. A small lifecycle. A few sensors that ran your existing lint and test commands. Nothing especially framework-ish. The 0.1 README said installer four times.

Then I started adding things. Each addition felt like a feature, not an architectural shift. The commit log reads like a feature log:

0.2: install-time options. Pick your agent. Add a target.
0.3: restructured the harness into corpus, guides, sensors, flywheels. Named taxonomy for the first time.
0.4: a kind taxonomy on guides and sensors. Different kinds, different load behavior.
0.5: a migrate command for forward-compatible upgrades. The harness had a schema now, and the schema had a version.
0.10: policy plugins. Org-level rules that projects pulled in.
0.11: the policy cascade. Org → Team → Project. Strict and non-strict layers.
0.12: playbooks. Ordered chains of actions.
0.13: sensors as a tier-aware policy kind.

Read that list once and it looks like product growth. Read it twice and the shape gets harder to miss. By 0.13 there was a runtime with conventions, plugins, a cascade, a lockfile, migrations, and per-agent rendering. The README still said installer.

The thing about a shape forming under a name is that the name is the last thing to change. Each individual commit looks like a small feature; the architectural mass piles up underneath and nobody is watching the mass. I was not, anyway. I was writing the next feature.

When I opened a fresh document called PLAN-10.md and tried to summarize where things stood, the first sentence I wrote was: Convert Keystone from a harness installer with org policy plugins into a harness framework. I stared at that for a while. The conversion was not a conversion. It was an admission.

What an installer was pretending not to be

Here is what makes the installer label dishonest in retrospect. An installer drops files. It is done. The user takes it from there. The framework that Keystone had become did all of this on top:

It defined named extension points. Guides, corpus, sensors, actions, playbooks, adapters per agent. Each one had a path convention, a frontmatter contract, and a load behavior. That is an API, not a directory listing.
It resolved conflicts across sources. The same /.md could exist in the project, in a team plugin, in an org plugin. Exactly one won. The resolution order mattered, and the ordering rules were stable across versions. That is a runtime, not a copy step.
It tracked drift. A lockfile pinned per-source SHAs and per-file hashes. The binary would notice when plugin files had been edited under it. That is integrity enforcement, not file-laying.
It migrated old layouts forward. keystone migrate would walk an installed harness, apply numbered transforms, and bring it to the current schema. That is a schema migrator, with all the implications of having a schema.
It rendered the same harness into multiple targets. Claude Code’s CLAUDE.md shape, Codex’s AGENTS.md, Cursor’s .cursor/rules/. The adapter layer translated one source into many. That is a code generator, not a paste-once-and-done.

Take any one of those in isolation and you can argue it was a feature an installer happened to include. Take them together and the argument falls over. Installers do not carry adapter layers. They do not have lockfiles. They do not have migration runners. They certainly do not have a cascade resolver with strict and non-strict overrides.

The give-away, when I went looking, was the layout of the source tree. Go files at the repo root next to a harness/ directory full of markdown that the binary embedded. No physical boundary. A single Go module mixing the runtime with the content the runtime shipped. If you asked me to point at “the framework” and “the content” in the 0.x tree, I would have had to do it with words, not directories. That is a tell.

The vocabulary did the heavy lifting

Here is the part of the story that surprised me most. I was bracing for a rewrite — the kind where you sit down with a blank internal/framework/ directory and start naming concepts from scratch, then translate the old code into them, then translate again because the first names were wrong. That is the usual cost of recognizing a shape late.

It did not happen. The abstractions held.

When I sat down to write the ports-and-adapters layer of the 1.0 plan, I had a table to draft. One row per port. For each port, a name, a path convention, an activation rule. I wrote it expecting to discover gaps. Here is what I found instead.

Every port already had a name. Every name was already in use. Every path was already conventional. Guide meant a rule loaded on every turn at guides//.md. Corpus meant reasoning loaded on demand at corpus//.md. Sensor meant an automated check at sensors/.md. Action meant a single unit of lifecycle work. Playbook meant an ordered chain of actions. Adapter meant the per-agent binding at adapters//…. The table I sat down to invent had been the truth of the code since half-way to 1.0. I was writing it down, not designing it.

Writing it down still mattered. The contract was implicit until that table existed. New ports would have slipped in by accident the same way old ones had. But the design work had already happened, one commit at a time, while I thought I was just shipping features.

That is the angle worth chewing on. The names I had given things while building features carried more architectural weight than any single commit they appeared in. Guide and corpus drew the line between rules-loaded-every-turn and reasoning-loaded-on-demand. Sensor drew the line between an automated check and an aspirational rule. Action and playbook drew the line between an atomic unit of work and an ordered chain of them. Every one of those lines was a load-bearing distinction in the runtime, and I had named them all before I knew they were load-bearing.

The corollary is the part to hold onto: when the refactor came, I did not have to invent a single new concept. I had to relocate the code that implemented the concepts. The concepts themselves stayed put. Naming had front-loaded the design work. The refactor was a directory move on top of a vocabulary that was already correct.

This is the underrated payoff of taking names seriously while building. If you have named the abstractions well in version 0.3, you can refactor the runtime in version 1.0 without renaming anything. If you have named them badly, version 1.0 starts with a six-month tax to fix the words before you can fix the code. The cheap version of a hard refactor is the one where the user-facing vocabulary is already correct and only the implementation moves.

It also changes the social cost of the refactor. A user who learned guide and corpus in 0.3 still knows what those words mean in 1.0. The blog post they wrote about the harness in 0.5 is still accurate. The wiki page in their team’s Notion has not gone stale. Refactor-without-renames is the kind a user does not feel.

Writing it down made it real

Once I admitted what was happening, the next move was to make the framing physical. Not “we say it is a framework now.” That is marketing. The framework had to look like a framework when you opened the source tree, and behave like one when you read the code.

The plan went through six phases. Each had a small handful of commits, none too clever, each one a small structural improvement. Order mattered, because some moves enabled others.

The phased refactor, in order. Each arrow is a phase the next one depends on.

A few things about the order are worth pulling out.

Phase 0 was the cheapest and the most important. Before any code moved, I wrote eight architecture decision records and one port contract per abstraction. Each ADR was a single page: context, decision, consequences, alternatives considered. Each port contract was a single page: path convention, required frontmatter, cascade behavior, an example, the command that scaffolds it. The total page count was around twenty. Writing them took two days. They turned out to be the most useful pages in the whole project, because every subsequent phase referenced them. When a phase started, I re-read the relevant ADR; when a phase finished, I re-read it again to check we had stayed inside the lines.

Phase 1 was the physical line down the middle. Every Go file under the repo root got moved into internal/framework/. The CLI entrypoint moved to cmd/keystone/. The template tree relocated to internal/framework/scaffold/templates/. After Phase 1, you could point at the framework in the directory tree without using words. That is a small thing that turns out to be a big thing. Two months later, when a contributor asks “is this framework behavior or content?”, the answer is a path, not a paragraph.

Phases 2 through 5 were small structural improvements stacking on top of that line. JSON-only config (Phase 2), because YAML had drifted across half a dozen schemas during 0.x. Vendored read-only plugins (Phase 3), because the plugin model had been “edit the markdown in harness/policies/” and that turned out to be the wrong default. Projects would silently diverge from upstream and nobody would notice. Conventions, generators, and a doctor command (Phase 4), because once you have named ports you can write generators that scaffold an adapter for each port. Per-port token budgets (Phase 5), because once you have named ports you can also count tokens per port and tell the user when one is bloated.

Phase 6 was the documentation phase. An upgrade guide for 0.x users. A compatibility doc spelling out what 1.x promised to keep stable. Nothing structural changed in Phase 6. The point was to make a contract that the next year’s worth of changes had to respect.

Each phase landed in a separate set of commits. Each commit was small enough to read in one sitting. None of them were a “big refactor commit.” That is the texture of a refactor that respects the work already there: many small surgical moves, each one preserving behavior, with the architecture emerging from the sum.

The plugin draft we threw away

I want to talk about a wrong turn, because every refactor has one, and the ones that do not get talked about tend to be the ones that bite later.

The first draft of the 1.0 plan made the built-in defaults (universal guides, lifecycle playbook, default sensors, per-agent adapters) into first-class plugins shipped embedded in the binary. The plan read:

The universal engineering corpus/guides, the lifecycle actions, the task playbook, and the default sensors all become first-class policy plugins — same shape as user-installed plugins, loaded by the same engine.

The promise was symmetry. Built-ins and external policy would travel the same pipeline. One loader, one cascade, no special cases. It was clean. It was elegant. It also had a giant cost buried in the elegance: the moment defaults are loaded as plugins, editing a default stops being “edit a markdown file in your repo” and becomes “fork the embedded plugin or override it from a higher layer.”

That breaks the Rails-style ergonomics I wanted at the project layer. The whole point of conventions over configuration is that the conventional file is just sitting there in the repo, your repo, your git, ready to be edited like any other file. If the default lives inside the binary and only appears in the consumer’s repo as a shadow that overrides it, you have turned a one-line edit into a four-step debugging session. Where is this rule actually coming from? Is the override winning? Why is my edit not taking effect? I have seen that pattern in tools that load defaults from inside their distribution, and the answer is always the same: it is a constant low-level tax on every user, paid forever.

I caught it because I sat with the plan for a day before starting any code. Two of the ADRs (number five, Conventions, not plugins, and number two, Framework / client boundary) ended up being the place where the wrong turn got walked back. The decision was: defaults are scaffolded into the consumer’s harness// on keystone init, from embedded templates. From that moment on, defaults are project content. The user edits them as markdown files in their own git. There is no override mechanism for defaults, because there is nothing to override. There is just one file sitting on their disk.

Plugins still exist. They do one job: share policy across projects. Read-only, vendored, hash-verified, drift-reset on the next run. They are not the mechanism for shipping defaults, and the mechanism for shipping defaults is not the mechanism for sharing policy. Two concerns, two mechanisms. The earlier symmetry was buying elegance with the user’s debugging time.

The lesson I want to draw from this is narrower than “avoid premature symmetry.” It is: when two mechanisms produce a surface that looks identical to the user, you should still ask whether the editing model is identical. If the user edits both the same way, symmetry is paying its rent. If the user edits one of them and merely reads the other, you have two concerns that happen to share a shape, and unifying them costs you the editing UX of the one the user actually edits.

I had to throw away two days of plan-writing to walk that back. It was worth it. The walkback is what made the 1.0 surface usable.

Code moved, concepts did not

When 1.0 shipped, I went back through the changelog to count. The framework had moved every Go file into a new location. It had dropped YAML. It had rewritten the cascade resolver. It had added a vendored plugin model, a doctor command, a budget command, a port-level scaffold generator, and per-agent adapter regeneration. The repo layout looked nothing like 0.x.

The user-facing vocabulary had not changed.

A 0.x user who never read the 1.0 plan could open the new docs and find every word they already knew. Guide still meant rules loaded every turn. Corpus still meant reasoning loaded on demand. Sensor still meant an automated check. Action and playbook still meant what they meant. The only new word at 1.0 was port, and port was a name for something the user already understood without the name. Namely, the named slot a piece of content lives in.

This is the payoff I underestimated when I was naming things in 0.2 and 0.3. The cost of naming a thing well at the start is that you have to actually sit with it for a few minutes and ask whether the name says what the thing does. The benefit is that later, when you refactor the runtime, you do not have to renegotiate the contract with every user who learned the old word. The user’s mental model does not change. Only the implementation does.

A refactor that does not rename anything is the cheapest kind for a user. They keep their docs, their wiki, their training onboarding. They keep the words they say out loud when describing the product to a teammate. The internals can move freely as long as the names stay fixed.

How shapes form under names

Step back from Keystone for a second. The pattern is broader than one project.

Tools grow into frameworks. They almost always do, if they live long enough. The path looks like this: a small utility solves a small problem. It picks up a configuration file. The config file grows extension points. The extension points need naming conventions. The naming conventions need a resolver. The resolver needs ordering rules. The ordering rules need a way to opt out, then a way to opt back in. Suddenly there is a runtime, and the runtime has a contract with everything plugged into it, and the contract is a framework whether you call it one or not.

The interesting question is when to notice. Too early and you are over-designing: naming abstractions before you have three concrete uses for them, building cascade rules for a single tier, writing port contracts for one port. Too late and the runtime has accumulated implicit contracts that nobody wrote down, and the cost of writing them down is paid in surprise regressions when you try to clean up.

The way I think about it now: the moment you find yourself adding a resolver (anything that picks a winner among multiple sources of the same concept) you are in framework territory. Lockfiles and migrations are also strong signals. Per-target rendering (the adapter layer) is a hard signal. Any one of those, taken alone, is not enough. Two of them together is. Three of them together and the question is not if you are building a framework, it is whether the framework is going to be honest about itself in its README.

If you only catch it at three, the way I did, you are not in trouble. You are in the spot where naming the shape costs you a few days of writing and a few weeks of refactoring, and the user-facing surface comes out untouched on the other side. If you catch it at six or seven signals deep, the cost is higher, because by then there are real users with real assumptions and the cleanup runs into compatibility costs.

Three things to do this week

If you are reading this and recognizing your own tool in the description, here is what I would do this week, in order.

Write the port table. Open a new document. List every named extension point in the tool. For each one, write the path convention, the activation rule, and the shape of the file (or config) that goes there. Do not stop until the table is exhaustive. The table is the easiest possible draft of the framework contract. If it takes an hour, you were already a framework and your vocabulary was holding the design up; if it takes a week and you keep discovering anonymous concepts, you have found the work that needs naming first. Either way, this is the highest-value hour you can spend on the project this week.

Draw the physical boundary. Look at your source tree. Can you point at “framework code” and “content the framework happens to ship” using a path, or do you have to use words? If it is words, move directories until it is a path. This is the single most clarifying refactor for a maturing tool, and the tests will catch you if you break anything along the way. One day’s work for a small project, a week for a medium one. From the moment it lands, every conversation about “where does this go?” gets shorter.

Write ADRs for the decisions you have already made. Not the future decisions, but the ones already locked in by the code. Three to ten ADRs of one page each. Context, decision, consequences, alternatives considered. Future you will thank present you the next time someone asks why X is the way it is. The act of writing them often surfaces a wrong turn you can still back out of cheaply, exactly like the plugin draft I walked back. Better to find that during ADR-writing than during a refactor PR.

After those three, you will have a clear picture of what you have, where it lives, and why. The work that comes next is the work that actually changes things — the renaming, if any, the directory moves, the new scaffolders, the migration path for existing users. That work is real and it takes time. But you can scope it, plan it, and ship it in phases that do not break users, because the contract you wrote down in the first three steps is the contract every subsequent phase has to honor.

The thing I keep coming back to from the Keystone refactor is how much of the work had been done already by past versions of me when I named guide and corpus and sensor and action in different commits weeks apart. They were not designing a framework. They were each adding one feature and giving it a name. The framework emerged from the sum of those names. The 1.0 refactor was the moment I looked up and noticed.

If you have been adding features and naming them well, your framework is already there. It is just waiting for you to write it down.

Keystone

Keytsone is now 2.1.1! It is open source and MIT-licensed. I am very open to feedback, so if you have any, please create a discussion or an issue on GitHub. It has a real chance of landing.

A lot landed between the 1.0 cut and 2.1.1, and all of it sits on top of the framework/client split 1.0 introduced. The shape of the project didn’t change. The surfaces around it did.

2.0 was the big one, shipped June 17. A new primitive taxonomy ran through the harness end to end. An in-binary MCP server moved keystone onto the same wire its agents already speak, so there’s no separate process to babysit. A localhost dashboard surfaced live state for the first time. An eval suite came with it. The disk layout moved alongside all of that, and the upgrade was still one command: keystone migrate.

2.1.0 followed the next day. The old patches mechanism retired in favor of a versioned migrations subsystem with paired Up/Down transforms under migrations//, run with keystone migrate up | down | status. The plugin→policy rename finished across Go source, JSON schemas, and docs in the same release. Installs that hadn’t migrated yet started warning and continuing instead of breaking, so the upgrade path stays soft.

2.1.1 reshaped the dashboard. It’s now an HTMX SPA with a single

swap target, real back/forward navigation, and fragment responses keyed off HX-Request. The 14 single-purpose nav links collapsed into five sections: Observability, Harness, Sources, Flywheels, Quality. SSE topic narrowing means a widget only refreshes when its own path changes. A per-session audit log lands at .keystone/state/audit/session--.jsonl, opened with O_CREATE|O_EXCL so nothing ever overwrites a prior run. Cmd+K (Ctrl+K on Windows and Linux) opens a search popover from any page.

If you’re still on 1.x, the upgrade is the same as the install (except Brew, which has an upgrade subcommand. It carries you the whole way to 2.1.1. To bring a project harness up-to-date with the new core framework structure, run keystone migrate up. This will patch core files only and will never touch user-edited files.