DEV Community: Todd Doane

A Claude Code hook that blocks pip install requests and tells the agent why

Todd Doane — Sat, 07 Nov 2026 13:19:56 +0000

A coding agent that hits ModuleNotFoundError can run pip install requests and carry on. The script works on that machine. The package never reaches requirements.txt, so CI or the next server breaks on the same import.

A rule in CLAUDE.md asking the agent not to do that is a request. A PreToolUse hook is a check that runs before the command does. Below is a 35-line one, the contract it depends on, and the case where it fails.

The contract

Claude Code runs a PreToolUse hook before a tool call and sends it the call as JSON on stdin, including tool_name and, for Bash, tool_input.command. Exit code 2 blocks the call, and the hook's stderr becomes the message Claude sees. Exit 0 with nothing on stdout approves nothing; the call goes on to the normal permission flow. A hook can also print JSON with a permissionDecision of allow, which skips the prompt. [src: https://code.claude.com/docs/en/hooks]

The hook below only exits 0 or 2 and prints nothing to stdout, so it cannot widen what the agent may do; your permission settings stay in charge.

The hook

.claude/hooks/block_pip.py, standard library only:

#!/usr/bin/env python3
"""PreToolUse hook: block `pip install <pkg>` that skips the requirements file."""
import json, shlex, sys

ALLOWED = {"-r", "--requirement", "-e", "--editable", "-c", "--constraint"}

def reason(command: str) -> str | None:
    try:
        tokens = shlex.split(command)
    except ValueError:
        tokens = command.split()
    if tokens[:3] in (["python", "-m", "pip"], ["python3", "-m", "pip"]):
        tokens = tokens[2:]
    if tokens[:2] not in (["pip", "install"], ["pip3", "install"]):
        return None
    args = tokens[2:]
    if any(a in ALLOWED for a in args):
        return None
    pkgs = [a for a in args if not a.startswith("-")]
    if not pkgs or all(p.startswith((".", "/")) for p in pkgs):
        return None
    return (f"`pip install {' '.join(pkgs)}` blocked: add it to requirements.txt "
            "with a version pin, then run `pip install -r requirements.txt`.")

def main() -> int:
    event = json.load(sys.stdin)
    command = event.get("tool_input", {}).get("command", "")
    why = reason(command)
    if why:
        print(why, file=sys.stderr)  # Claude Code shows stderr to the model
        return 2                     # exit 2 = block this tool call
    return 0                         # exit 0 = normal permission flow

if __name__ == "__main__":
    sys.exit(main())

The message is written for the agent, not for you. It says what was blocked and what to do instead, so the agent has a next step that is not another way around the block.

Wire it in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": "python3 \"$CLAUDE_PROJECT_DIR\"/.claude/hooks/block_pip.py" }
        ]
      }
    ]
  }
}

Test it without Claude Code

The hook is a function from a string to a reason, so most of it is a plain unit test:

from block_pip import reason

def test_blocks_adhoc():
    assert reason("pip install requests")
    assert reason("python3 -m pip install 'httpx==0.27.0'")

def test_allows_lockfile_and_local():
    for cmd in ("pip install -r requirements.txt", "pip install -e .",
                "pip install .", "pip list", "pytest -q"):
        assert reason(cmd) is None, cmd

def test_misses_chained_command():
    # one-line split only: the second command is not seen
    assert reason("cd app && pip install requests") is None

And the whole hook from the shell, with a fake payload:

$ echo '{"tool_name":"Bash","tool_input":{"command":"pip install requests"}}' | python3 .claude/hooks/block_pip.py
`pip install requests` blocked: add it to requirements.txt with a version pin, then run `pip install -r requirements.txt`.
$ echo $?
2

Then once in Claude Code itself. We ran it headless (claude -p, Claude Code 2.1.312), told to run pip install requests and not to try another install method. It was blocked, and the agent's reply quoted the hook's message word for word.

The case it misses

The third test is there on purpose. cd app && pip install requests is one string to this hook: the first token is cd, so it passes.

The fix is to split the line on &&, ||, ;, | and newlines before checking each part, and to skip leading FOO=1 assignments and sudo. Our paid kit's guard_bash.py does that. Neither version sees uv pip install, pipx, or a command hidden in a script file or eval. A hook that reads command text is a guard against the common mistake, not a sandbox.

Where this came from

The same pattern, extended to five command types (commit while a secrets linter finds a key, --no-verify, force-push with --force-with-lease allowed, rm -r on /, ~ or the project root, and ad-hoc pip install), is the guard hook in our Agent Ops Kit ($29 one-time). The storefront shows a capture of it blocking real commands, unedited except for line wrapping. The free, MIT-licensed agent-ops-kit-lite plugin has the CLAUDE.md scaffold, the secrets linter and a /verify-claims skill.

Written by Calyx AI, a product line of Quality Tester Solutions LLC. Drafted with AI assistance; the code and outputs were run on Python 3.14.8 and Claude Code 2.1.312.

Math.round(x * 100) / 100 rounds 1.005 to 1: the rounding bug in our unit tables

Todd Doane — Thu, 05 Nov 2026 13:24:59 +0000

We build free calculators for people who pull-test anchors and fasteners. One of them converts displacement between mm and inches and prints two reference tables. On 2026-11-04 our publish gate failed the page: the 1/16 in row said 1.587 mm.

1/16 in is 0.0625 in, and 1 in is exactly 25.4 mm, so the exact value is 1.5875 mm. Rounded half up to three decimals that is 1.588. The table was off by one in the last digit, on a page whose whole job is to be right about the last digit.

Where 1.587 came from

The static tables were generated in Python with an f-string:

>>> f"{0.0625 * 25.4:.3f}"
'1.587'
>>> from decimal import Decimal
>>> Decimal(1.5875)
Decimal('1.587499999999999911182158029987476766109466552734375')

1.5875 has no exact binary form. The nearest double is a little below it, and formatting rounds that stored value correctly: it is below the halfway point, so it goes down. Python's own docs give the same example with round(2.675, 2), which returns 2.67. [src: https://docs.python.org/3/library/functions.html#round]

The fix for the tables was to stop using floats for them: build each value as a Decimal from strings and round with ROUND_HALF_UP.

from decimal import Decimal as D, ROUND_HALF_UP
f"{(D('0.0625') * D('25.4')).quantize(D('0.001'), ROUND_HALF_UP)}"   # '1.588'

The same class in the browser

The live calculators round in JavaScript, so we checked those next. They used the two common idioms, and both have the same problem:

Math.round(1.005 * 100) / 100   // 1, not 1.01
1.005 * 100                     // 100.49999999999999
(1.005).toFixed(2)              // "1.00"
(2.55).toFixed(1)               // "2.5"

1.005 * 100 lands just below 100.5, so Math.round goes down. toFixed rounds the stored binary value, which for 1.005 and 2.55 is just below the halfway point; MDN documents the 2.55 case. [src: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toFixed]

There is a second, smaller trap: Math.round rounds halves toward positive infinity, so Math.round(-2.5) is -2, not -3, and Math.round(-0.4) returns -0. [src: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/round]

All 19 calculator pages we had at the time rounded with Math.round(x * 10^d) / 10^d or toFixed.

The fix: round the decimal string, not the binary value

JavaScript prints a number using the shortest decimal string that maps back to the same double. For the double nearest 1.005 that string is "1.005". If you shift the decimal point by editing the exponent in that string, instead of multiplying, the number you round is the one you see:

function rd(x, d) {
  if (!isFinite(x)) return x;
  var s = x < 0 ? -1 : 1, p = String(Math.abs(x)).split("e");
  var r = Math.round(Number(p[0] + "e" + ((+p[1] || 0) + d)));
  if (r === 0) return 0;
  p = String(r).split("e");
  return s * Number(p[0] + "e" + ((+p[1] || 0) - d));
}

rd(1.005, 2)    // 1.01
rd(-1.005, 2)   // -1.01   (half away from zero, like ROUND_HALF_UP)
rd(-2.5, 0)     // -3
rd(-0.004, 2)   // 0, not -0

Number("1.005e2") is exactly 100.5, so Math.round sees the true halfway value. The sign is handled separately, which gives half away from zero for negatives and matches Python's ROUND_HALF_UP on the table side. Exponents in the string (1e-7) are handled by adding to the existing exponent. The result goes to toLocaleString for display; the calculation itself keeps full precision.

Making it stay fixed

A fix that depends on everyone remembering it does not last, so the repo now has two tests:

A lint test that reads every calculator page and fails on Math.round(x * 10...) / Math.round(x * Math.pow(10, d)) or a bare .toFixed( (one on a value rd already rounded is allowed). It flagged all 19 old pages before the change and none after.
A headless-browser check that loads each live page, pulls out the rd that page actually ships, and runs 14 fixed cases through it (1.005, 1.5875, 0.0625 * 25.4, 2.675, -1.005, 1e-7, 0.1 + 0.2 and others), plus one check in the displacement converter's UI. It ran 267 checks when we made the change and 295 now, after two more pages, all passing.

We also rechecked the 245 static table cells on six pages against exact Decimal values after the 1/16 in fix: none were wrong.

What it does not fix

rd rounds the number it is given. If the arithmetic before it already drifted, rd rounds the drifted value: in our stress converter, 0.00100005 ksi to psi computes 1.0000499999999999, not 1.00005, so rd shows 1. We left that case out of the test on purpose and wrote down why. It fixes halfway rounding, not every floating-point error. For values that must be exact (prices, published tables), compute in decimal from the start.

The part we did not expect

The bug was not found by a unit test. It was found because the page went through our publish gate, which checks each claim and number in a draft against a facts file before it goes live, and the gate's claim-by-claim check flagged the row. A generalized version of that gate is the Publish Gate we sell ($19 one-time). A free claim-check skill for Claude Code, /verify-claims, is in our MIT-licensed agent-ops-kit-lite repo. The calculators, including the fixed displacement unit converter, are free at calyx-ai.net/qa.

Written by Calyx AI, a product line of Quality Tester Solutions LLC. Drafted with AI assistance and checked by the gate it describes; the code and the numbers are from our own repo and were run on Node v26.7.0 and Python 3.14.8.

A zero-dependency MCP server in one file: what the protocol actually needs

Todd Doane — Sat, 31 Oct 2026 12:23:14 +0000

We wanted two checks from our publish gate available inside any MCP client: "is every number in this draft in my facts file" and "does every link in it load". The official SDKs are fine, but for two tools we did not want a dependency tree, so we wrote the server against the protocol directly. It is one file, 125 lines, and it imports only node:fs and node:readline. We also got the error split wrong, twice, which makes it the most useful section below.

What a stdio MCP server has to do

For a tools-only server on the stdio transport, the job is small:

Read JSON-RPC 2.0 messages from stdin, one per line.
Write one JSON reply per line to stdout. The spec says the server must not write anything to stdout that is not a valid MCP message; logs go to stderr. [src: https://modelcontextprotocol.io/specification/2025-06-18/basic/transports]
Answer four methods: initialize, tools/list, tools/call and ping.
Stay silent on notifications (messages without an id), such as notifications/initialized.

The read loop is readline over stdin. This is the same logic as server.js, restructured for reading:

const rl = createInterface({ input: process.stdin });
rl.on("line", async (line) => {
  let msg;
  try { msg = JSON.parse(line); } catch { return; }   // blank or bad line: ignore
  const reply = (body) => {
    if (msg.id === undefined) return;                 // notification: never reply
    process.stdout.write(JSON.stringify({ jsonrpc: "2.0", id: msg.id, ...body }) + "\n");
  };
  try {
    const result = await handle(msg);
    if (result === undefined) return;
    reply({ result });
  } catch (e) {
    reply({ error: { code: e.code ?? -32603, message: e.message } });
  }
});

And handle is a switch:

switch (method) {
  case "initialize":
    return { protocolVersion: PROTOCOL_VERSIONS.includes(params?.protocolVersion)
               ? params.protocolVersion : PROTOCOL_VERSIONS[0],   // ["2025-11-25", "2025-06-18"]
             capabilities: { tools: {} },
             serverInfo: { name: "publish-gate", version: VERSION } };
  case "tools/list": return { tools: TOOLS };
  case "tools/call": /* run the tool, see below */
  case "ping": return {};
  default:
    if (id === undefined) return undefined;            // notification: no reply
    throw Object.assign(new Error(`method not found: ${method}`), { code: -32601 });
}

initialize answers with the client's protocol version only if it is one of the revisions we checked the server against; otherwise it offers the newest of those. The lifecycle spec asks for exactly that: same version if supported, otherwise another version the server supports. [src: https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle] The first version echoed whatever the client sent, which told a client asking for a newer revision that the server followed rules it had never been checked against.

Two kinds of error, and the one we got wrong

Update 2026-11-01: a reader in the comments pointed out that this section followed the 2025-06-18 revision, and the 2025-11-25 revision moved input validation errors to the other bucket (SEP-1303). [src: https://modelcontextprotocol.io/specification/2025-11-25/changelog] They were right; the code and this section are corrected below.

The MCP spec splits tool errors in two. Protocol errors are ordinary JSON-RPC errors, for an unknown tool or a request that does not fit the call schema. Tool execution errors are a normal result with isError: true, for input validation errors and failures while the tool runs, so the model sees the message and can retry. Clients should pass tool execution errors to the model and may pass protocol errors. [src: https://modelcontextprotocol.io/specification/2025-11-25/server/tools]

Our first version put everything in the second bucket. On 2026-10-31 we moved both an unknown tool and a call with no draft to protocol errors, reading the older revision. The second is an input validation error, so a model might never have learned which argument was missing. Now only an unknown tool gets a coded error, rethrown so the read loop turns it into a JSON-RPC error:

const invalidParams = (message) => Object.assign(new Error(message), { code: -32602 });

// in callTool: reject bad calls before doing any work
if (TOOLS.some((t) => t.name === name) === false) throw invalidParams(`unknown tool: ${name}`);
if (args.draft_text == null && args.draft_path == null) throw new Error("give draft_text or draft_path");

// in handle, case "tools/call":
try { return await callTool(params?.name, params?.arguments); }
catch (e) {
  if (e.code === -32602) throw e;                       // protocol error
  return { content: [{ type: "text", text: `error: ${e.message}` }], isError: true };
}

Now an unknown tool gets -32602, an unknown method gets -32601, a call without a draft and an unreadable draft_path file both come back as an isError result starting with error:, and anything else thrown becomes -32603. A stdio test sends an unknown tool, a call without a draft, an unreadable draft_path and an unknown method, and checks each reply; another checks the version answer for a known, an unknown and a missing protocolVersion.

Each tool returns plain text: PASS, or FAIL (n) followed by one line per finding. That is a design choice for a model reading the result, not something we measured.

Replies can arrive out of order

Handlers are async, so a slow call can finish after a fast one. In a test session on 2026-10-31, request 4 (an unknown method) was answered before request 3 (a tools/call to check_numbers). That is allowed: clients match replies by id, never by order. Do not "fix" it with a queue unless a tool really needs ordering.

Test it without a client

You can drive the server from a shell, which is a quick way to see what a client sees:

printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18"}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
  | node server.js

For unit tests, the server exports its check functions (checkNumbers, checkLinks) and injects what touches the outside world: checkLinks takes a fetchFn, so node --test runs without the network. The server has 6 tests and they pass on Node v26.7.0.

Install it

The server is not on npm yet. From a clone, in Claude Code:

git clone https://github.com/skibuff22/agent-ops-kit-lite
claude mcp add publish-gate -- node "$PWD/agent-ops-kit-lite/mcp/publish-gate/server.js"

We have tested it in Claude Code (claude mcp list shows Connected); other stdio MCP clients should take the same setup (command node, the path to server.js as the argument), but we have not tested them. check_numbers takes the draft and your facts.md as text or paths; check_links takes the draft. The source is in agent-ops-kit-lite (MIT), next to the free Claude Code plugin it belongs to.

These are two of the deterministic checks in our publish gate. The paid Publish Gate adds the rest: style and voice checks, an LLM claim-by-claim check that fails closed, a SHA-256 lock, and publishers that refuse a draft that changed after it passed. $19 one-time.

Written by Calyx AI, a product line of Quality Tester Solutions LLC. Drafted with AI assistance and checked by the gate it describes; the code and the test session are from our own repo.

A publish gate for AI-assisted posts: check every claim before it goes out

Todd Doane — Mon, 26 Oct 2026 12:23:11 +0000

If an AI agent drafts what you publish, the risk is not bad grammar. It is a confident number nobody measured, a feature that does not exist, or a paragraph of stock phrases with your name on it. We let our store's daily agent post under a real person's name, so we built a gate it cannot skip.

What goes wrong without one

A language model writes what is plausible. Plausible is fine for a first draft and wrong for a public claim. The failures we wrote the gate against are ordinary ones: a round number that sounds right, a feature described the way it will work next month, a link to a checkout page with no tracking, and the familiar vocabulary that tells a reader nobody read the draft.

Asking the agent to "be accurate" does not fix this. The agent that wrote the claim is the one judging it, in the same context, with the same blind spots. The check has to sit outside the writing step and it has to be able to say no.

The gate, in three layers

A facts file is the only source of truth

Every claim and every number a post may use lives in facts.md, each with its evidence: a file path, a command and its output, or a URL. If it is not in the file, it is not claimable. The file also has a "not claimable" section: customer counts, benchmarks, anything nobody has measured. Writing that section down is what keeps later posts honest.

Deterministic checks are the floor

Numbers: every number in the body must appear in facts.md, or the line carries a source link. Years, times, versions and list numbering are exempt.
Phrases: a built-in list of stock AI phrases plus your own "Banned phrases" list from voice.md.
Structure: no emoji, at most one exclamation mark, no rhetorical-question openers, a cap on em-dash density.
Links: every link must resolve, and checkout links must carry utm_source so you can see which post sent the buyer.

These cost nothing to run and they never get tired. They catch the cheap mistakes before a model is involved.

An LLM checks each claim, and fails closed

The last check sends the draft and the facts file to a model and asks for a verdict per claim: SUPPORTED, UNSUPPORTED or CONTRADICTED. Anything other than SUPPORTED fails the post. If the model cannot be reached, the gate fails too. A check that passes when it cannot run is not a check.

A PASS writes a small sidecar file with the SHA-256 of the post body. Every publisher script calls one function before it posts:

side = require_verified(path, body)   # refuses if the sidecar is missing,
                                      # the hash differs, or the LLM step was skipped

Edit one word after the PASS and the publisher refuses until the gate runs again. That rule stops an agent from tweaking a post after it passed. It does not stop an agent with write access from changing the publisher itself; that part is still code review.

What it caught, and what it did not

The gate checks content. Today it showed us the edge of that. Our Show HN post passed the gate, the publisher submitted it and printed PUBLISHED, and the account's submissions page was empty. Hacker News had answered with a notice that it is temporarily restricting Show HNs from new accounts. The publisher had trusted the redirect after submitting and, when it could not find the item, reported a generic URL as success.

That is a publisher bug, not a content problem, and we handled it the way we handle any failure: find the cause, fix it, add the test, then recover. The publisher now reports success only when the item appears under the account's own submissions, saves HN's response page otherwise, and reports the restriction as declined. We did not retitle the post to get around the rule; the draft is parked and the rule is logged. A gate tells you a post matches your facts file and your voice guide. It cannot tell you a community will take it, and your publishers need their own proof that the post actually exists.

Try the pattern without buying anything

Start with the cheapest part: a facts.md with evidence per line and a "not claimable" section, plus a rule in your CLAUDE.md or AGENTS.md that nothing is published unless every number in it is in that file. Then add the hash lock: write the body's hash on approval and refuse to publish if it changed. Those two steps are the core of the pattern.

The free Agent Ops Kit Lite (MIT) has the CLAUDE.md structure this runs inside.

Disclosure: I sell the finished version. Publish Gate v1 is verify_post.py with configurable facts and voice files, any model CLI through PUBLISH_GATE_LLM (default claude -p), the require_verified helper, an example dev.to publisher, templates, a directive for your agent and 8 tests: $19, or $29 bundled with the Agent Ops Kit, at calyx-ai.net. This article was drafted with AI assistance and passed the gate it describes. Originally published at calyx-ai.net.

The self-annealing loop: make every AI agent failure leave a test behind

Todd Doane — Wed, 21 Oct 2026 12:26:37 +0000

When an AI coding agent hits an error, the easiest move is to get past it: retry, reset, reprocess. That makes today's run green and guarantees the same failure next week. The self-annealing loop is a short rule that changes what the agent does after something breaks, so the system gets harder to break each time.

Why agents default to recovery

Ask a coding agent to finish a job and you have told it, implicitly, that a finished job is the goal. When a step fails, the shortest path back to "finished" is recovery: run it again, clear the stuck queue, restart the service, add a retry. Each of those is reasonable in isolation. Together they hide the cause, and because the next session starts with a fresh context, nothing about the failure survives except the workaround.

A human engineer does the same thing under pressure. The difference is that the human usually remembers. The agent does not, unless you give it somewhere to write the lesson down and a rule that says it must.

The loop, in order

Annealing is the metallurgy term for heating and cooling a metal so its internal stresses relax and it ends up more ductile and less likely to crack. In an agent-run system the "heat" is a failure. The loop is five steps, and the order is the whole point:

Diagnose the root cause. Keep asking "why" until you reach something that, once changed, prevents the failure from recurring. "The request timed out" is a symptom. "The health check treats any timeout as a store outage" is a cause.
Fix it in code, config or infrastructure, not in the data.
Add the test that would have caught it, so the fix cannot silently regress.
Write the standard down. Add a line to the directive's Edge cases & learnings section, and a lint if the pattern can be detected mechanically.
Then recover: reprocess the stuck items, restart, re-run.

Recovery still happens. It just happens last, after the cause can no longer produce the same failure. A rule worth adding: if the agent cannot finish the first four steps in the session, it says so and leaves the item open instead of quietly recovering.

Put it where the agent reads it first

The loop only works if it is part of the agent's standing instructions, not a prompt you remember to type. In a D.O.E. repo (Directive, Orchestration, Execution, described in the first article) it lives in CLAUDE.md. A condensed example:

## When something fails (self-annealing)
1. Diagnose the root cause. A symptom is not a cause.
2. Fix it in code/config/infra.
3. Add the test that would have caught it.
4. Add the standard to the directive's "Edge cases & learnings"
   (and a lint if it is mechanically detectable).
5. Only then recover. Never recover first and call it fixed.
Record each pass in the anneal log: date, symptom, cause, fix, test.

The anneal log matters more than it looks. It is how you see that the same area keeps failing, which is usually the signal that the first fix treated a symptom.

A real example: three anneal passes in four days

We run our store with a daily agent job on a small Linux box. Each day a script checks the storefront and the checkout endpoint, and if either returns nothing (HTTP status 000), it raises an ATTENTION note: the agent must diagnose and fix the cause before doing anything else.

Pass 1. ATTENTION: storefront unreachable. Root cause: the box had lost its own uplink (gateway and DNS failing); the store was fine. Left behind: a "host offline" check before raising ATTENTION, curl timeouts and retries, and a fix for a git pull that failed on a dirty log file.
Pass 2. The metrics job failed with a name-resolution error. Root cause: flaky DNS on the same box. Left behind: retries with backoff on transport errors only.
Pass 3. ATTENTION again, storefront unreachable. Root cause: the LAN was clean but the WAN was dropping most packets, and the single offline probe happened to succeed, so the pass-1 check did not fire. Left behind: six probes instead of one; two or more failures is classified as "host network degraded", not a store outage. Plus a standard in the store-operations directive, and a note that this is a hardware problem for a human.

Pass 1 was a real fix and still not the whole answer, which is normal. What the log gives you is the history: by pass 3 it showed the same alarm with the same kind of cause, which pointed at the check itself rather than at the store.

The last pass also shows the loop's honest exit: some root causes are not code. When the cause is a flaky network link, the right standard is "classify it correctly and tell a human", not another retry.

What to check before you call a pass done

Could you state the cause in one sentence that does not contain the words "sometimes" or "flaky"? If not, keep asking why.
Does a test fail on the old code and pass on the new? If you cannot write one, write the lint or the directive line and say why.
Is the learning in the file the next session reads first, not in a commit message or a chat transcript?
Did recovery happen after the fix, and was it verified (an HTTP OK, a matching hash, a row count)?

Start with one directive

You do not need a framework to try this. Add the block above to your CLAUDE.md, .cursorrules or AGENTS.md, and give your most failure-prone job a directive with an Edge cases & learnings section. After a week, read that section. It is a list of failures that now have a test, a rule or a lint against them.

The free Agent Ops Kit Lite (MIT) has the CLAUDE.md with this loop, the directive template and a secrets linter. It also installs as a Claude Code plugin: /plugin marketplace add skibuff22/agent-ops-kit-lite.

Originally published at calyx-ai.net. Disclosure: I sell the full Agent Ops Kit ($29), which adds the self-annealing, bug-fix and agent-governance directives, three example directives, seven execution scripts and multi-agent templates. The free lite kit above stands on its own. This article was drafted with AI assistance; the incident list is from our own run log.

Why your AI coding agent keeps breaking prod (and the 3-layer fix)

Todd Doane — Sun, 18 Oct 2026 12:08:23 +0000

Claude Code, Cursor and Codex are excellent at single steps. Production work is not a single step. Here is the arithmetic of why agents fall apart on real tasks, and the structure we use to run a small company's platform with agents without babysitting them.

The symptom: brilliant in the demo, flaky on Tuesday

You ask the agent to deploy a fix. It edits the right file, writes a plausible commit message, runs the deploy, and then something small goes wrong: a CRLF line ending breaks a shell script on the server, an environment variable is read from the wrong file, a webhook key rotates on redeploy. The agent notices the failure, "fixes" it by retrying or resetting state, and reports success. Next week, in a fresh session, it does exactly the same thing.

None of this is the model being dumb. It is the shape of the work.

The arithmetic: errors compound

Suppose your agent gets each step right 90% of the time, which is generous for a step like "parse this API response and write the right rows". Chain steps together and the probabilities multiply:

Steps the model executes	Success rate
3 decisions, scripts do the rest	0.9³ = 73%
5 steps	0.9⁵ = 59%
10 steps	0.9¹⁰ = 35%

Real tasks (fetch, parse, transform, validate, write, deploy, verify) are ten to fifty steps. If the model executes every one of them by improvising code or commands in the moment, the success rate of the whole job collapses, and you end up watching it.

The fix is not a better prompt. It is moving the steps that have a right answer out of the model entirely.

The 3-layer fix: Directive, Orchestration, Execution

We call it D.O.E. Each layer does one kind of work, and the boundaries between them are the point.

Layer	Nature	Lives in	Changed by
Directive	Intent, in prose: goal, inputs, tools, outputs, edge cases, how to verify	`directives/*.md`	Humans write; agents append learnings
Orchestration	Judgment: which directive applies, which inputs, did it work	The agent; `CLAUDE.md` is its job description	You, rarely
Execution	Determinism: fetch, parse, transform, deploy	`executions/*.py`	Agents write; tests guard

A directive never contains code. A script never contains policy ("ask the user if…"). The agent never contains either: it reads one and runs the other. Now the model makes three decisions instead of fifty executions, and decisions are the part a model is actually good at. A script that parses a response correctly once parses it correctly forever.

What a directive looks like

# Directive: Deploy the API

## Goal
Ship the current branch to the API server and prove it's live.

## Execution tools
executions/deploy_service.py --target api   # normalizes CRLF/BOM, uploads, restarts

## Edge cases & learnings
- Shell scripts committed from Windows had CRLF endings and
  failed with "bad interpreter". deploy_service.py now normalizes them.
- The webhook key rotates on redeploy; the script refreshes it.

## Verification
GET /health returns 200 and the commit hash matches HEAD.

The Edge cases & learnings section is where the return on investment lives. Every gotcha the agent meets is written down where the next session reads it before acting, instead of being rediscovered at 2 a.m.

The part that makes it get better: self-annealing

Structure alone stops the bleeding. What makes month three dramatically better than week one is a rule about failure: recovery is not a fix. When something breaks, the agent runs a fixed loop, in this order:

Diagnose the root cause: keep asking "why" until you reach something that, once fixed, prevents recurrence.
Fix it in code, config or infrastructure.
Add the test that would have caught it.
Add a standard to the directive (and a lint, if the pattern is mechanically detectable).
Then recover: reprocess the stuck items, restart the service.

Most agent sessions do step 5 first and stop. "Let's just reprocess the stuck items" and "I'll add a cron to retry failures" feel like progress and leave the cause in place. Making recovery the last step means every failure leaves behind a test, a rule and sometimes a lint, so whole classes of mistakes stop being possible.

A real example from building our own store

While building the checkout for our store, a secrets linter found a live Stripe key hard-coded in an old helper script in our own repo. Removing that one key would have been recovery. The fix was making lint_secrets.py run before every commit and adding a line to CLAUDE.md requiring it. That kind of mistake can no longer reach git in that repo, whoever (or whatever) is typing.

How to start today, for free

Put a CLAUDE.md (or .cursorrules / AGENTS.md) at your repo root that states the three layers and the self-annealing loop.
Pick the one job your agent fails at most often. Write a directive for it with the six sections above, and point it at a script.
Move the deterministic parts of that job into the script. Give the script a test.
Tell the agent: "Read CLAUDE.md, then run the deploy directive." Watch what it appends to Edge cases & learnings.

We publish a free starter, Agent Ops Kit Lite on GitHub (MIT), with the CLAUDE.md, the operating-principles directive, the directive template and the secrets linter. It is enough to try the approach on one job this afternoon.

Originally published at calyx-ai.net. Disclosure: I sell the full Agent Ops Kit ($29) — the extra directives, tested execution scripts and multi-agent templates. The free lite kit above stands on its own. This article was drafted with AI assistance and reviewed against the framework we run in production.