DEV Community: Tatsuya Shimomoto

What Humans Should Approve Is Intent, Not the Diff — A Decision Table for Agent Approval Gates

Tatsuya Shimomoto — Wed, 29 Jul 2026 03:41:20 +0000

What this article covers: How to catch drift from your intent while it's still cheap to undo (just before commit or publish) without slowing your agent's autonomous execution down. You get a decision table that mechanically determines, from the kind of thing being changed, whether the gate should show a human the full diff text or just an intent summary — plus the one required item that keeps the summary honest: a three-valued Divergence from plan declaration. It ports directly into your own workflow or your team's code review conventions.

As you extend how far an agent runs on its own, you eventually hit this fork.

Lean toward not stopping, and drift from intent only becomes visible after the fact. The thing works, but it's pointed the wrong way. By the time you notice, it has piled up and you can't afford to unwind it
Lean toward stopping, and the human becomes the bottleneck. A review queue forms, and the point of autonomous execution evaporates
So you compromise: "let's at least show the diff." This is the worst of the three. You're paying the cost of stopping, but the volume means nobody reads it, and intent still isn't protected

I suspect the third one is the most widespread. The gate survives as a formality, and in substance you're back to the first case.

Here's what to re-examine. Who is that automation for? If going fast is itself the goal, humans are in the way — but what you actually want is for the thing you intended to get built fast. If so, what to cut is not human involvement itself, but what the human is involved in.

So this article keeps the number of times a human is stopped the same, and moves only the layer they judge at. Whether the artifact is correct becomes the machine's primary responsibility, and the human holds the layer above it: what this is aiming at, and what changes as a result. That changes what the gate shows. That routing is the decision table below.

Assumptions

The examples come from Claude Code 2.1.220 plus my own harness (the set of rules and skills under ~/.claude)
But the deliverable is a way of writing conventions, so it isn't tool-dependent. It works the same in Cursor, in Devin, or in your company's code review policy
Background assumed: you have an AI agent implement things, and you have some approval step before commit

Gates have two axes

When you design an approval gate, there are actually two independent questions.

Axis	Question	What decides it
First axis	When do you stop	Reversibility. An edit you can undo with `git checkout` doesn't stop; publishing externally, minting a DOI, or committing does
Second axis	When you've stopped, what does the human judge	The kind of target (the subject of this article)

Usually only the first axis is settled. "Stop before commit." "Confirm before publishing." — the stop condition is written down, but what to show when you stop isn't. My own harness was like that.

That blank doesn't stay blank. Every time you write a gate, it gets filled in by whatever interpretation is handy at that moment, and the default it fills in with is almost always "show the diff." From the writer's side that looks safest. Show everything and nothing gets missed. The result is the third case from the opening, mass-produced.

The decision table: what you show is decided by the target

This is the deliverable. When a gate stops, what you put in front of the human is determined by the kind of thing being changed. (Below, "deterministic gate" means checks a machine can answer Yes/No on: lint, type checking, tests, secret scan.)

Target	What to show	Why
Behavior-shaping artifacts (`CLAUDE.md` / `AGENTS.md` / rules / skill and agent definitions / public documentation)	Full text	The text itself is the intent. Reading it is already work at the intent layer
Control plane (hooks / permission settings / permission definitions like `--allowedTools` / scheduled task definitions) and artifacts that produce the evidence the checks run on (tests / fixtures / lint config / coverage thresholds / CI definitions / review agent prompts / dependencies)	Full text	These move the gate itself, and the evidence the gate rests on. Fold them into a summary and "a change that loosens the checks" disappears
Implementation code and generated output	Intent summary (what it aims at, and what changes as a result)	Correctness that mechanized checks can decide belongs to deterministic gates and review agents. The diff text and the PASS list don't go on the approval screen
When a deterministic gate FAILs	The detection line itself (mask only the secret's actual value)	This is a state absent from the approved plan, so there's nothing to fold a summary into (the cross-check mechanism is described later). Bypassing it is a decision to disable one check, which puts it in the same class as the control plane

Escalation rule: irreversibility overrides the target category

The table is a principle, and it doesn't decide everything on its own. Even within "implementation code," padding tweaks in a UI and changes to authorization logic, DB migrations, billing, data deletion, or key rotation call for different things to hand a human. So add a one-line override.

For irreversible, high-impact changes, show the full text (or the relevant diff) regardless of target category.

Which is to say the first axis (reversibility) affects not only "when you stop" but also "what you show." The principle is decided by the kind of target, and irreversibility escalates it toward showing full text — that two-stage form is the accurate statement. No third axis is needed.

In one line

Correctness that mechanized checks can decide belongs to the machine; the human holds intent. The one exception is targets where the text is the intent — there, reading the text is itself the intent judgment, so you show the text. Showing full text isn't a backslide into the old way; it's the same principle showing up differently.

There's one more principle you need alongside the decision table.

A review agent is an inspector, not an approver. If the side proposing and the side inspecting come from the same lineage, the inspection inherits the proposer's blind spots wholesale (this article calls that the generator–verifier gap). So approval is composed of two things, "the deterministic gate PASSed" plus "the human's intent judgment," and you never build a path where approval closes on an LLM alone. Making review heavier does not mean approval can be delegated to an LLM.

Written into a conventions file, it looks like this (excerpted from the real thing; internal links and some references are omitted).

## Artifacts are the machine's, intent is the human's

Correctness that mechanized checks can decide belongs to deterministic gates and review agents;
**do not put humans back on artifact inspection** (an assignment of primary responsibility, not a
guarantee — residual risk is caught by the escalation rule). A review agent is **an inspector, not
an approver** (generator–verifier gap). Approval is composed of "the deterministic gate PASSed"
plus "the human's intent judgment"; **never create an approval path that closes on an LLM alone.**

Applying it: what was actually happening in five places

Before I made the decision table, my harness had the second axis blank. What follows is my own environment, but any setup with the same blank will produce the same shape.

The same gate had two competing sources of truth

For the intervention point just before commit, two files said different things. Nowhere was it written which one was canonical (i.e. which one you ultimately follow).

File	What it said
`rules/common/planning.md`	"Verify results check — just before commit" (what to show was unspecified)
`skills/implementation-chain/SKILL.md`	"diff approval just before publish / deposit / commit"

The origin was a single sentence I'd written earlier in a record of a different design decision.

I just do a final check before commit

It doesn't say what the "final check" is of. Downstream, that sentence got read as "diff approval," and settled in wearing the face of a canonical rule.

"A human inspects the artifact" was scattered across five places

Skill	Wording before	Result of applying the decision table
`implementation-chain`	"diff approval just before commit"	Implementation code → changed to intent summary
`readme-writer`	"a human approves the diff and applies it"	Public documentation → rewritten as show full text (same target, different reason)
`release-doi`	"visually confirm the latest commit is newer than the previous release"	A predicate a machine can check → hand it to the machine
`paper-deposit`	Open the generated PDF's first page and eyeball it (paraphrased)	Same → replaced with machine verification of font embedding
`harness-sync`	"`git diff` is the review gate"	What's being synced is rules / skills → leave it as is

Those five were written on different days, for different purposes. All of them landed on the same default with no coordination, so this isn't individual carelessness. Wherever there's a blank, "show the human everything" fills it. Which is why patching the individual wording wouldn't stop it — I needed one convention covering the second axis itself.

And that rightmost column shows where the decision table earns its keep. Grep and all five look identical, but wiping them out uniformly would take out gates you need. harness-sync did end up staying: what it syncs is rules and skills, so reading the text is itself the intent judgment.

Reinforcement 1: the control plane and "artifacts that produce the evidence"

In the first version of the decision table, I put hooks and permissions on the "implementation code and config" side. A review from an external model (Codex CLI) hit exactly there as its top-priority finding.

Codex's finding

> The central target boundary permits executable harness controls to be hidden behind proposer-authored summaries, and the new rule conflicts with the existing security false-positive workflow.
> - [P1] Classify executable harness controls as behavior-shaping — …

Suppose the agent disabled the pre-commit secret scan hook. It can follow the convention to the letter and still summarize it like this.

Intent summary: Tidied up the pre-commit check configuration and strengthened the checks

The control plane "moves the gate itself," so folding it into a summary turns a change that loosens the checks into a well-meaning sentence.

And hooks and permissions aren't the only way to weaken things. Rewriting the tests so they pass against your implementation is the quietest path of all. Here the agent doesn't even need to lie.

Intent summary: Fixed the implementation; all tests PASS

That's exactly true. The one thing that never surfaces in the summary is that what got rewritten was the tests. Not just "the tests passed" but "what counts as passing" is under review. Fixtures, coverage thresholds, lint config, CI definitions, review agent prompts, and dependencies all have the same property.

So how should you present it

The answer is: don't choose summary-or-text per commit. If one commit contains both implementation and evidence, show the summary, and alongside it show the full text only for the parts that are evidence.

Intent summary: Changed the login-failure retry limit from 5 to 3 (plan step 2)

⚠ This change touches the evidence the checks rest on — review the text below

  tests/test_auth.py:41
  -    assert client.retry_limit == 5
  +    assert client.retry_limit == 3

The human reads only those last two lines. The implementation-side diff isn't shown. All you have to look at is whether the tests were loosened to match the implementation, so volume can't beat you. The example above corresponds to a change decided in the plan, so it goes through. If the plan said nothing about retries and only the test threshold dropped, that's where you stop.

Which files are on the evidence side can usually be picked up from the path.

# Extract candidates for "produces the evidence" from changed files, to show as full text
git diff --cached --name-only | grep -E \
  '^(tests?/|\.github/workflows/|\.claude/|hooks/)|(_test\.|\.test\.|pyproject\.toml|package\.json|ruff\.toml|codecov\.yml)'

This is not a complete classifier; it's a conservative candidate extractor. It misses tests embedded in ordinary code, custom directories, lockfiles, and inline snapshots. Add the evidence-side files you know about in your own repo and grow it.

In my environment, this extraction has moved out of the convention's prose and into a PreToolUse hook (evidence-file-notice.sh). When it detects a git commit, it scans the staged files against a pattern like the one above (the real list is a bit broader — it also covers spec/, conftest.py, pre-commit config, and so on), and if anything matches, it injects the instruction itself into the approval flow: show the human the diff of these files. The canonical home of the evidence-file enumeration is no longer the convention's text but that one regex in the hook; the convention keeps a single pointer line saying detection is canonical there.

There are two reasons for this restructuring.

A convention that exists only in a document is followed only probabilistically. Write "evidence-side files get their text shown" into a convention and there's no guarantee the agent recalls it every time. "Which files are evidence-side," though, is a structural property decided purely by the shape of the path — a machine fires on it 100% of the time
Keep the enumeration in two places — convention and hook — and they will drift apart. Centralize it on the side that fires, and let the convention hold only the principle

Semantic judgments like the decision table stay in the document; anything mechanically decidable gets lowered into a gate as soon as you spot it — this convention itself has been growing by exactly that process.

The control plane (hooks / permissions / scheduled tasks) works the same way: summary plus the full text of the relevant files. Anything producing the evidence a judgment rests on gets treated the same as the judgment itself.

Reinforcement 2: on FAIL, show the detection line

The second thing that same review found. Push "no machine-check PASS list and no diff text goes to the human" all the way, and you also delete the path where a human judges a secret scan false positive.

In my harness, when the pre-commit secret scan detects something, there's a workflow to let it through with SECRET_SCAN_BYPASS=1. The judgment is "this is a dummy key in a test, so it's fine."

The reason this goes to a human isn't that a machine can't tell the difference. It's that even if the machine or the review agent happens to be right, this is the kind of decision whose owner must be a human. A bypass is the act of disabling one check on the spot. That has the same nature as moving the control plane, so for the same reason as Reinforcement 1, you need the full text (i.e. the detection line).

There's a second reason: a FAIL is a state absent from the approved plan. An intent summary is meant to be read against the plan (see below), so a state that isn't in the plan has nothing to fold into.

**FAIL is the exception** — a deterministic gate's FAIL presents **the detection line itself**.
Mask the secret's actual value; the owner of the false-positive call (`*_BYPASS`) is the human

PASS needs care. PASS only means "the checks you configured accepted this." Whether it was implemented according to the plan is not something PASS can tell you. That's exactly why you need a separate intent summary — you leave out the PASS list because you're looking at the summary instead.

"Not shown" is not "not kept"

This gets misread easily, so let me be explicit. Leaving out the PASS list means moving it off the human's approval screen, not throwing check results away. Delete the trail and you lose any way to trace "what was passing back then" later. The idea in this article isn't to delete information; it's to move unneeded information off the human's judgment surface.

I originally had this distinction written into the convention as its own clause; the current version drops it. Claude Code already persists the transcript and tool results in machine-readable form at all times, so writing it down merely restated a default the runtime already guarantees. If your execution environment doesn't guarantee log retention, keep it as an explicit clause. "Don't write into the convention what the substrate already guarantees" is itself one of the judgment calls you'll make when porting this.

One more thing: on a true positive, dumping the detection line as-is duplicates the secret's actual value into the conversation, the approval screen, and the logs. That would spread the very leak the gate exists to prevent, so show the file, the rule name, and the surrounding context, and mask only the value. What a human needs to judge a false positive is not the value itself, but where it was and which rule it hit.

Reinforcement 3: don't let the intent summary be free-form

This is a prerequisite for putting the decision table into practice.

If you stop at "implementation code gets an intent summary," the human ends up judging by reading a self-report written by the proposer itself. That just relocates the gap that was at the artifact layer. What they read changed from a diff to an essay; the author is the same.

The countermeasure is to fix what the summary is cross-checked against to something human-originated. My harness has two intervention points.

Plan check — when what to do has been settled (this is where the human approves)
Intent check — just before commit

And the summary in 2 is presented cross-checked against the plan the human approved in 1. Because the referent is a human-approved object, the loop can't close on self-reporting alone.

Only one item is required: the `Divergence from plan` declaration

But "write it against the plan" still leaves room to write it conveniently. In particular, when something new is discovered during implementation and the work naturally diverges from the plan, that divergence quietly disappears from the summary.

My first countermeasure was to make the intent summary itself a fixed form: five mandatory headings — approved intent, what changed, divergence from plan, impact on users and operations, evidence-side changes. But the day after that form went into the convention, an audit re-examining the whole convention clause by clause forced me to re-sort those five. Only one of the fields was actually preventing the worst case — the silent disappearance of divergence. This is all that's required now.

Intent summary: Changed the login-failure retry limit from 5 to 3 (plan step 2)
Divergence from plan: None

Divergence from plan takes one of three values: None / Yes / Needs re-approval. The plan changing because of something you discovered mid-implementation isn't bad in itself. What's dangerous is the fact that it changed disappearing from the summary.

And making this declaration required turns omission into falsehood. In free-form text, a summary that doesn't mention the divergence isn't lying — it just didn't bring it up. With the declaration required, writing None when there is divergence is a falsehood, and writing Yes points the human's eyes exactly there. It closes the omission escape route structurally, without leaning on the writer's honesty. This is the one thing only formal enforcement can protect.

So why drop the other four fields? Because they were the summary's shape, not its defense. What was intended, what changed, what's affected — summarizing without dropping those is exactly the calibration a model should bring to any summary: lead with the outcome, report results faithfully, match the response to the question. Impose a fixed form on the parts calibration already covers, and you're forcing five headings onto a one-line config change — the approval screen turns into template skimming.

That would reproduce, on the summary side, the third case I called the worst at the start: a gate that survives as a form nobody reads.

(As for the "evidence-side changes" field — the hook from Reinforcement 1 covers it structurally, so there was never a need to make a human write it.)

Anthropic's context-engineering guidance for the Claude 5 generation (The new rules of context engineering for Claude 5 models) states this as a general rule: shift from binding with rules to delegating to judgment — except in the regions where the worst case is unacceptable, which stay explicitly bound. The 5→1 reduction is that rule applied. Delegate the summary's shape to the model's calibration; bind exactly one point with form — the disappearance of divergence, the worst case you can't accept.

The whole point of this gate convention was to concentrate the human's cognitive budget on judging intent. If the convention itself burns the writer's and reader's attention on enforcing a form, that defeats the purpose.

One more practical upside: the three values are an enumeration, machine-readable, so "does the declaration exist at all" can itself be lowered into a hook check later. Same as Reinforcement 1 — build the convention so it has an exit ramp down into a deterministic gate.

Cap the number of stops too: one gate per unit of work

Paired with what to show, fix one rule about how often. The intent check happens once, at the completion point of a unit of work. If a commit, a push, a publish, and the accompanying doc updates all arrive together, bundle them into one decision with the count and scope explicitly enumerated. Ask for approval at every intermediate phase or intermediate commit and the approvals themselves become noise — the gate degrades into formality. There are exactly two exceptions: a deterministic gate FAILing, and Divergence from plan: Needs re-approval.

But "bundling is allowed" is not "implicit is allowed." If one decision covers N items, say N at approval time. The moment an approval granted for one item gets silently reused for N, it has stopped being a gate.

Raise the granularity and the bottleneck moves

This cross-check bites hardest when the agent is good. Things going well for a long stretch is exactly the condition under which the intent summary becomes a formality. If you find yourself skimming past the same "no divergence" every time, raise the granularity of the plan you're checking against.

But that isn't free. The finer the plan, the further the bottleneck moves from just-before-commit to plan approval. Reserve the finer granularity for the things you can't take back when they drift (external publishing, data migration, permission changes). You don't get to erase the fork entirely — the accurate statement is that you get to choose where you pay.

On the conventions side, I also renamed the second intervention point from "Verify results check" to "intent check." Leave the name as "Verify results" and it drags you back into making humans read a list of machine-check results.

Porting it into your own conventions: three steps

Enumerate your existing approval steps and check what each one says to present. Grep for wording like "approve the diff," "visually confirm," "review and apply"
Put the decision table and the escalation rule into a single conventions file, and have each place you found point at it. If you only patch the individual wording, the next skill or document you write will fill the blank the same way again
Build the Divergence from plan declaration (None / Yes / Needs re-approval) into the summary as a required item. The summary as a whole doesn't need a fixed form. Without this, you can install the decision table and still be verifying the proposer's own self-report

In step 1's grep, watch out for missing word stems. I searched for eyeballed and missed eyeball (the uninflected stem), so one place survived all the way to the external model's review. Search verbs by stem rather than inflected form, or run grep -i with several patterns.
For reference, in my environment this change (one new conventions file plus re-pointing references across 13 related files) came to 14 files and 217 added lines. The decision table itself is short; what made the difference was re-wiring everything that referenced it.

The convention's body has since been cut in half (the text auto-loaded into every session's context went from 265 to 125 words). What remains is only the principles — the decision table, the escalation rule, the divergence declaration. The "why" moved into the decision record (ADR), and the evidence-file enumeration into the hook. Left alone, a convention fattens on rationale and examples until nobody reads it. Principles in the document, why in the record, anything machine-decidable in a gate — splitting the storage that way is what keeps it maintainable.

The limits of "the machine holds it"

Let me be straight about this. Build, types, lint, tests, and secret scan tell you only about the properties they cover. Holes in authorization logic, concurrency races, irreversible side effects of a migration, requirements that were never implemented at all — all of that sails through as PASS. And the review agent, being from the same lineage as the proposer, shares its blind spots (the generator–verifier gap above).

So the third row of the decision table isn't "the machine guarantees correctness" — it's an operational tradeoff that puts primary responsibility for artifact inspection on the machine. Residual risk does not go to zero.

Which is exactly why you need the escalation rule. Keep the human on the artifact side only in the regions you can't undo — data migration, permissions and billing, external publishing, deletion. Put another way: if a human is reading diffs of reversible implementation code, that isn't a response to residual risk, it's just inertia.

You also can't reduce the whole convention to a deterministic lint. Extracting the evidence-side files (the hook in Reinforcement 1) and checking that the divergence declaration exists can be mechanized, but deciding "is this a behavior-shaping artifact" is semantic, and for edge cases like generated documents and config files you have to think each time. The decision table illustrates by enumeration; it doesn't exhaust the space.

Back to the opening question

The further you extend autonomous execution, the more drift from intent becomes visible only after the fact. But put the human back on review and the point of autonomy disappears. This looks like a fork only while you're counting human involvement by number of stops.

Count by content instead and it stops being a fork. Keep the number of stops the same, and move what they read at the stop from the artifact to the intent. What you cut isn't human involvement — it's the time a human spends re-checking the correctness of artifacts. The machine was always better at that, and when a human does it, volume wins and it becomes a formality.

To the question of who the automation is for, this is the answer. Not to go fast, but so that what you intended gets built fast. If so, there's only one thing the human should hold to the end. Intent. The decision table was a tool for taking everything else away from them.

When you're talking about reducing human review, what's left isn't the work you forgot to cut. It's the work you can't cut.

Opus 5 Changed How Rules Should Be Written — Audit Yours

Tatsuya Shimomoto — Tue, 28 Jul 2026 11:38:58 +0000

What this article covers: Anthropic's official guidance for the Claude 5 generation (Opus 5 / Fable 5) is "fewer rules, trust the model's judgment." This is a procedure for auditing the custom rules (CLAUDE.md / rules files) you accumulated for older model generations against what the product itself loads, and deciding for each rule: keep, fix, or retire.

If you've been using Claude Code for a while, you probably share these worries:

You wrote CLAUDE.md files and rules to compensate for older models' weaknesses, and you have no idea whether they still help with Claude 5
The official guidance now says "cut your rules," but you can't tell which of your rules are the ones to cut
Deleting everything at once and watching behavior degrade is scary, so you don't touch any of it

On 2026-07-25 I ran this audit on my own ~/.claude harness (my set of custom configuration: CLAUDE.md, rules files, agent definitions) and cut the resident rules — the part auto-loaded into every session — from 5,789 down to 2,463 words (measurement details at the end). This article documents the cross-checking procedure I used, in a reproducible form. The takeaway is not the reduction itself but the decision procedure: on what evidence, which rules, and how to dispose of them.

Background: the official guidance on rule-writing changed

On 2026-07-24 (the day Opus 5 launched), Anthropic published The new rules of context engineering for Claude 5 generation models. Two points matter here.

First, Anthropic itself deleted more than 80% of Claude Code's system prompt:

We removed over 80% of Claude Code's system prompt for models like Claude Opus 5 and Claude Fable 5 with no measurable loss on our coding evaluations.

Second, the post spells out the shift in context engineering as six Then → Now pairs.

Then (older generations)	Now (Claude 5 generation)
Give rules	Trust judgment
Give examples	Design interfaces
Front-load everything	Progressive disclosure
Repeat yourself	Simple tool descriptions
Write memory into CLAUDE.md	Auto-memory
Sparse specs	Rich reference material

The post names a concrete harm of over-instruction: conflicting instructions. System prompt, skills, and user instructions collide, so that "leave documentation as appropriate" and "DO NOT add comments" coexist in the same request.

In other words, the reason to cut rules is not just token savings. The problem is that your custom rules conflict with the product's own instructions, and you are forcing the model to resolve the contradiction. From here, the procedure is about finding which of your rules are the conflicting ones.

Premise: the ground truth is the system prompt and tool descriptions

What actually reaches the model in the same context as your custom rules is the system prompt and the tool descriptions. That is the ground truth to audit against, and the first step is simply to read it. Official docs and blog posts state recommendations, but they are not loaded at inference time.

The meaning of a mismatch changes with this distinction, so let's name the two layers.

Layer	Contents	Loaded at inference time?	Name of the mismatch
Runtime layer	System prompt + tool descriptions	Yes (loaded into context alongside your rules)	Conflict (the model is forced to resolve a contradiction)
Guidance layer	Official docs and blog posts	No	Drift (you've merely diverged from official recommendations)

A mismatch only qualifies as a conflict when both sides are in the same context (a tool description is only loaded in sessions where that tool is available).

All examples below come from my own ~/.claude harness (Claude Code 2.1.220, audited 2026-07-25 to 07-26).

Step 1: capture the current runtime layer from a live session

The first task is to know exactly what your custom rules are sharing context with. There is a trap here: the runtime layer is injected from outside your config repository too, so no amount of reading ~/.claude shows you the whole picture.

A concrete example. My session's system prompt contained this resident line:

"Do not call the AgentTool unless the user requested it"

Yet grepping ~/.claude finds this string in no file at all (I haven't identified the source; I suspect the harness itself or a plugin, but couldn't confirm). If you only audit config files, you miss this class of instruction.

So capture from a live session instead. There are two techniques.

For tool descriptions, get the real thing. Ask Claude Code to print the target tool's description:

Quote the description of the EnterPlanMode tool verbatim. Do not summarize.

For the system prompt, have the session quote itself. This is self-reported, so say "verbatim" explicitly to prevent summarizing:

From the system prompt currently loaded, quote verbatim the instructions
about plan mode / commit messages / scope handling.

The trick is to ask per theme. "Print everything" is too long and summaries creep in. Build the theme list from your own rules: open every custom rules file (rules / CLAUDE.md) first, assign each instruction to a theme (planning, commits, review, scope, ...), then query the runtime layer theme by theme. That way you at least eliminate audit gaps on your side of the comparison.

Two caveats. First, gaps in the other direction — runtime instructions you don't know exist — can't be fully caught by this procedure, so zero detections means "this question found nothing," not "no conflicts." Second, both capture techniques are model-mediated self-reports, with no guarantee of exactly matching the actual input. Treat the captures as a screening pass; before acting on a rule's disposition (retire / invert), confirm the same wording reproduces in a separate session.

Step 2: cross-check against your rules and sort into 3 categories

Cross-check the captured runtime layer against your custom rules and sort each mismatch into three categories.

Category	Meaning	Detected in my harness
Conflict	An instruction or claim that contradicts the runtime layer is loaded at the same time	2
Redundant	The runtime layer already says nearly the same thing	1
Drift	Diverged from guidance-layer (official docs) recommendations	3

Conflict example 1: a no-plan-mode rule

My planning.md contained this, written as a countermeasure to older generations' habit of not planning:

On direct implementation instructions, execute immediately. Do not enter plan mode.

Meanwhile, the EnterPlanMode tool description loaded in the same context (Claude Code 2.1.220) says:

"Prefer using EnterPlanMode for implementation tasks unless they're simple."
"If unsure whether to use it, err on the side of planning"

Two instructions pointing in exactly opposite directions were loaded into the same context, and the model was silently picking one every time.

Conflict example 2: a rule referencing a ghost setting

The second one was a nastier pattern. Background: by default, Claude Code appends a Co-Authored-By: Claude ... trailer to commit messages — the attribution that shows Claude as a co-author on GitHub. On whether to emit that attribution, my git-workflow.md said:

Note: Attribution disabled globally via ~/.claude/settings.json.

But checking the actual settings.json, the relevant key (includeCoAuthoredBy) did not exist:

python3 -c "import json; print('\n'.join(sorted(json.load(open('settings.json')).keys())))"
# includeCoAuthoredBy is not in the output

Whether I deleted the setting at some point or the note was wrong from day one, there's no way to tell anymore. What I can say for certain: a rule cannot detect that its own justification has vanished. Code errors out when its referent disappears; a rule keeps asserting "disabled" while sitting in context, contradicting the runtime layer's instruction to append the Co-Authored-By trailer to commit messages.

Strictly speaking, this line is not an instruction ("don't add the trailer") but a (false) statement of fact ("already disabled"). But since it kept riding along in the same context as information contradicting a runtime-layer instruction, I counted it on the conflict side.

How the unresolved conflict was actually being resolved (a model-dependent observation, with limits)

Digging through session logs from the period when this ghost-setting conflict sat unresolved, I found an interesting split. Same Claude Code version (2.1.220), same rules: two Opus 5 sessions added the trailer to commits, one Fable 5 session did not. Within each session, behavior was 100% consistent.

So when you leave contradictory instructions in place, how they get resolved is up to the model — and when the model changes, the behavior can change too.

The limits, stated plainly: this is a natural observation, not a controlled comparison (5 commits with differing tasks and user instructions). The most I can claim is "likely model-dependent." Also, the observed difference — attribution — is harmless; whether conflicts with real consequences split the same way is unverified.

Redundancy example: a scope-adherence rule

Separate from conflicts, some rules had simply become unnecessary to say. My custom rule "when a scope is specified, adhere to it strictly" exists almost verbatim in the Claude 5 generation system prompt:

"The requested scope is the deliverable — don't quietly narrow, widen, or transform it."

No collision, so the harm is small — but I was paying resident tokens to repeat what the product already says. It was exactly the official post's "Repeat yourself → Simple tool descriptions" shift.

Drift example: a confidence threshold for review

I detected three drift items. I'll cover one representative case here; the other two — the handling of pre-commit verification steps and the rule splitting review into a separate agent — get different verdicts, so they're covered in Step 3 and the pitfalls section respectively.

The representative case: my review-agent definition said "only report findings you are at least 80% confident in" (a number unrelated to the 80% system prompt reduction above). The official Opus 5 prompting guide warns against exactly this kind of suppression instruction, using it as an example (it does not mention the number 80% specifically):

If your review prompt says "only report high-severity issues" or "be conservative," the model may follow that instruction literally and report less; ask it to report everything and filter in a separate pass instead.

This is not a load-time conflict, but with the official docs explicitly saying the model may follow this kind of instruction literally and under-report, leaving it in place means dropped findings. I haven't verified how much damage it did on older generations, but since it's the exact pattern the current generation's docs warn against by example, I classified it as drift.

Step 3: don't treat "conflict = bad" — run each item through a decision frame

Once the three-way sort is done, decide dispositions. The important part: do not mechanically mark conflicts and drift for deletion. Some of your custom rules were written precisely to override product defaults on purpose. "Points the opposite way" alone cannot distinguish an accident from a deliberate choice.

I judged each item on four axes.

Axis	Question
Intent	Did you write it to override the product default, or was it simply not conflicting at the time?
Evidence	Is there a record of why it was written — an ADR, an incident note?
Freshness	Does the product behavior it assumed (an old generation's weakness, etc.) still hold?
Expiry condition	Have you defined what would trigger revisiting this rule?

Three example dispositions.

Retire (freshness expired): The no-plan-mode rule (conflict example 1) was a countermeasure to older generations' "start running without planning" weakness. In the Claude 5 generation the product itself now recommends planning; the premise is gone. There was no reason to keep it as an intentional override, so I retired it.

Retire (conflict + redundancy): Beyond the ghost-setting line (conflict example 2), git-workflow.md contained nothing but content the model and the runtime layer already cover — commit message format, how to write PRs. Fix the conflicting line and the rest is still redundant. So the whole file had no reason to exist, and I retired it.

Keep (intent and evidence alive): On the other hand, I kept my rule "run review in a separate agent process from the implementer." The official guide contains the line "do not use subagents to verify or double-check your own work," which at first glance collides with this rule.

But read in context, this is not a rejection of division-of-labor review. That line is one phrase in an example prompt developers can add to control subagent spawning, in a section about cost containment. The official premise is that Opus 5 verifies its own work without being told — explicit verification instructions just cause over-verification and should be cut — and that the new generation, which also delegates more, would otherwise reflexively spawn "just in case" double-check subagents and burn cost. That's the brake being illustrated. The same doc also positively cites the writer–verifier pattern — separating the writer from the verifier — as a strength of Opus 5.

My rule has two tiers. The base: never let review happen in the same context as the implementation (same-context review inherits the implementer's blind spots). On top of that, for important changes, a different model (in my case, Codex CLI) does the review, decorrelating the blind spots. The latter was an easy keep — cross-model review is a capability Claude Code itself structurally cannot provide no matter how much it evolves, so there is nothing for the product to absorb it into. The design decision is recorded in an ADR.

The former (separate-process review on the same model) may still overlap with the "added verification passes" the official guidance wants to suppress. This is the interpretive part: if Opus 5's self-verification turns out to solve even the blind-spot-inheritance problem, the base tier may become unnecessary. It stays — as a keep with an expiry condition: revisit when the premise changes.

Running this decision frame taught me one thing: rules with recorded evidence (ADRs, incident notes) are fast to judge, and the judgments are more confident. Having a record doesn't prevent retirement (the plan-mode rule had a known rationale and was retired anyway, on expired freshness). What the record changes is not the outcome but the cost of deciding. For rules with no recorded rationale, even the question "is it safe to delete?" turns into git log archaeology.

Pitfall: mechanically applying the official advice misdiagnoses

The official guide says verification steps added by legacy harness scaffolding are also candidates for removal. Applying that mechanically will misdiagnose.

My custom rules had an 8-item pre-commit Verify step (build / type check / lint / tests / secret scan / dependency audit / doc sync check / git status check). Taken literally, "cut verification steps" makes all eight look like deletion targets.

But what the official guidance objects to is instructions that increase the model's self-verification. Most of the eight items are deterministic command runs — builds, tests — that add zero model judgment. The axis is:

The verification is done by a machine (command, hook) → fine to keep
The verification is done by the model, on its own judgment → the official guidance's target; consider cutting

The other pitfall: cases where deletion isn't enough and you need an inversion. If you simply delete the "only report at ≥80% confidence" line, the section it lived in (a noise-suppression framework) survives and keeps pushing in the suppressive direction. I rewrote the section to: "Do not suppress by confidence. Report everything; the caller filters in a separate pass." An instruction whose correct direction has flipped must be rewritten in the opposite direction, not merely removed.

Results and numbers

The center of gravity is the procedure, but for reference, my harness before and after (measured with wc -w, resident total of CLAUDE.md + rules):

Point in time	Resident words	Files
Before the audit	5,789	20
After (as of 2026-07-26)	2,463	14

Note that I am not claiming "resolving the conflicts improved performance." My environment has no instrument for measuring the cost of contradiction-resolution; the quantitative case rests on Anthropic's measurement quoted at the top (80% deletion, no degradation). What this article demonstrates is not the effect but real examples of config drift, and a procedure for finding and disposing of it.

Summary: what you can do next

Capture the runtime layer (system prompt + tool descriptions) from a live session. Reading your config repository alone doesn't show the whole picture
Cross-check against your custom rules; sort into conflict / redundant / drift
Don't treat "conflict = bad" — judge keep / invert / retire on the four axes of intent, evidence, freshness, and expiry condition
Apply the official reduction advice with the distinction between model self-verification and machine verification

And one lesson to add from this round: when you write a rule, record its rationale and its expiry condition. I'll confess I wasn't doing this systematically either. Rules with an ADR (design decision record) took minutes to judge; rules without one turned into git log archaeology; and for the ghost setting, I couldn't even pin down when it disappeared. A rule cannot detect that its own justification has vanished — so whether you'll be able to judge your rules at the next generation change will be decided by the notes you leave now.

Postscript (2026-07-26)

The lessons from this article have already flowed back into my harness.

Every resident rule file now carries rationale: and review-when: metadata, and my rules-audit skill rules-stocktake reads these fields during audits
The procedure in this article (collect the runtime layer → cross-check → judge with the decision frame) has been generalized into a skill called generation-audit, published as a standalone repo, so it can be re-run as-is at the next model generation change

claude-security Beta: Cost, Output, Accuracy — 189 Agents, 2 Hours, Measured

Tatsuya Shimomoto — Mon, 27 Jul 2026 13:00:04 +0000

What this article covers: A measured run of Anthropic's official security-scanning plugin claude-security (beta). What the tool does, how long it takes, what output it returns, and how far you can trust it — backed by the raw data from the run's own artifacts.

You may know the official plugin exists and still not have tried it, for reasons like these:

The cost is unreadable. It warns you at startup that it "may take a while and use a significant number of tokens," but never says how much
The output quality is unknown. How is this different from existing static analysis, and does an LLM that only reads code produce findings you can act on?
It's unclear what to do afterward. If dozens of findings come back, where do you start?

On 2026-07-25 I installed v0.10.0 and scanned my entire ~/.claude configuration set (hooks, skills, agents, permission settings). 189 subagents ran for two hours and returned 20 findings. This article is that measurement.

This plugin is in beta as of 2026-07 (the version number is 0.10.0). Every number and behavior below is measured on v0.10.0 and may change.

Prerequisites

Item	Requirement
Claude Code	Measured on 2.1.220
Plugin	`claude-security` v0.10.0 (by Anthropic, marketplace `claude-plugins-official`)
Python	python3 3.9+ (used by the report-generation scripts)
git	Change scans require a git checkout. Whole-repository scans work without git
Permission mode	auto mode recommended (the plugin says so itself)

This plugin is the "runs inside your session" version of the hosted Claude Security product. It starts no separate process and no daemon; it runs entirely inside your Claude Code session.

What the tool is, and what happens when you run it

From install to launch

# Run these inside a Claude Code session
/plugin install claude-security@claude-plugins-official
/reload-plugins

If it reports that the marketplace was not found, run /plugin marketplace add anthropics/claude-plugins-official first and retry.

After installing, /claude-security opens a menu offering three jobs.

Job	Target	Duration
Scan codebase	The whole repository, or a scoped part of it	Depends on size (2 hours 1 minute in my run)
Scan changes	A branch diff, a pull request diff, or a single commit	Minutes for a small diff
Suggest patches	Turns an existing report's findings into patch files	Depends on the finding count

You can also skip the menu and name the job in the arguments.

/claude-security scan my branch's changes
/claude-security --base main
/claude-security 3cb30d2          # a hex string of 7+ chars is read as a commit SHA

Get the confirmation prompt out of the way

Choosing a whole-repository scan always triggers this confirmation.

This scan may take a while and may use a significant number of tokens. You will need to leave Claude Code open while the scan completes. Are you sure you want to continue?

Accept the cost up front and you go straight in. The check is not a literal string match — it asks whether your request already reads as accepting the time and token cost — so include a sentence that conveys that.

/claude-security scan this repository — whole codebase.
I understand it may take a while and use a significant number of tokens.

You do need to leave Claude Code open while the scan runs. No interaction is required, so you can walk away.

Effort changes how much it searches, not how hard it verifies

--effort has four tiers. The distinctive design choice here: the verification panel is fixed at three voters at every tier.

effort	What it does
`low`	One researcher over the whole repository. No inventory, threat model, or breadth sweep
`medium` (default)	The full workflow: inventory → threat model → one researcher per component × category → one sweep → three-voter panel
`high`	Like `medium`, but the component cap rises to 24, two researchers per cell, two sweeps
`max`	Like `high`, plus an adversarial phase: marginal keeps are re-panelled and every survivor faces a red team

So dropping the tier does not mean "wave things through" — it means "search a narrower area." The documentation says as much, noting that the report's confidence figures are calibrated against that three-voter panel. I ran medium.

A six-stage pipeline runs

flowchart LR
    A["Inventory<br/>partition the tree"] --> B["Threat model<br/>one per component"]
    B --> C["Research<br/>component × category"]
    C --> D["Sweep<br/>what the matrix missed"]
    D --> E["Panel<br/>3-lens adversarial check"]
    E -. max only .-> F["Adversarial<br/>repanel + red team"]

Here is what that came to against ~/.claude.

Stage	Agents	Detail
Inventory	1	Partitioned the tree into 11 components
Threat model	11	One per component
Research + Sweep	42	One per component × category cell; the sweep is counted in this same pool
Panel	135	45 candidates × 3 votes
Total	189	2 hours 1 minute

This breakdown and the elapsed time are measured, not estimated. You can reproduce the same tally on your own run: the workflow's journal.jsonl records each agent's return value, so you can classify agents by the shape of what they returned.

# Run inside ~/.claude/projects/<project>/<session-id>/subagents/workflows/wf_*/
python3 - <<'PY'
import json, collections
sig = collections.Counter()
with open("journal.jsonl", encoding="utf-8", errors="replace") as f:
    for line in f:
        try: d = json.loads(line)
        except ValueError: continue
        if d.get("type") != "result": continue
        r = d.get("result")
        sig[tuple(sorted(r))[:5] if isinstance(r, dict) else ("?",)] += 1
for k, v in sig.most_common(): print(v, k)
PY

135 ('reasoning', 'verdict')                                          <- panel votes
 42 ('findings',)                                                     <- researchers
 11 ('assumptions', 'entryPoints', 'hotFiles', 'sinks', 'trustBoundaries')  <- threat models
  1 ('components', 'securityScanSkippedComponents')                    <- inventory

The elapsed time comes from the same logs. The first record is 02:28:12Z and the last is 04:29:16Z. The largest gap in between is 47 seconds, so something is running for the entire two hours.

But this does not mean you are tied up for two hours. You must leave the Claude Code window open; you do not have to do anything. The plugin's internal documentation has a line that assumes exactly this.

Users desire to leave the session unattended very soon after kicking off a scan, around a minute of wall-clock time.

That matched my experience — I spent the interval in a separate session doing unrelated work.

Progress is visible stage by stage in /workflows while it runs.

What the output looked like

When it finishes, three artifacts land in a timestamped directory.

CLAUDE-SECURITY-20260725-022756/
├── CLAUDE-SECURITY-RESULTS.md        # the human-readable report
├── CLAUDE-SECURITY-RESULTS.jsonl     # machine-readable (one finding per line)
└── CLAUDE-SECURITY-REVISION-*.json   # the revision and settings stamp for the run

The result was 20 findings (5 HIGH / 15 MEDIUM). 111 raw candidates became 83 after dedup; of the 45 that reached the panel, 25 were rejected, leaving these 20.

A rejection rate above half can be read as evidence that the panel is working, or as evidence that the research stage over-generates candidates and the panel cleans up after it. The report contains nothing that decides between those readings.

What does decide something is the number on the way out. I acted on all 20 findings, and rejected none as a false positive. I confirmed each one reproduced before fixing it (three were addressed only partially — I did not add OS-level isolation). This is one repository and one run, but at least in this case, a tool that only reads and reasons did not produce a pile of off-target noise.

Three things about the output quality stood out.

1. It reads configuration and natural-language instruction files

The inventory partitioned the tree like this.

hooks / scripts / scheduled-tasks / skills-executable-scripts /
skills-instructions / agents / rules / templates / docs /
notes-and-metrics / tests

rules and skills-instructions — meaning the instruction files written in plain natural language were treated as their own audit components.

The 20 findings broke down by location like this.

hooks: 10 / skills: 5 / settings.json: 3 / scheduled-tasks: 1 / agents: 1

The one under agents was about the trust-ordering instruction itself, written in an agent definition .md. One of my agents treated raw session logs as "machine records, hard to alter" and placed them at the top of its trust order.

But those logs hold the verbatim bodies of external pages fetched in the past. The file may be hard to tamper with; its contents are not trustworthy.

The one under scheduled-tasks was similar in character: a README said a job was "not wired up," when it was in fact registered and running weekly. It flags documentation that contradicts reality as something that makes reviewers misjudge risk.

A tool that searches for patterns with regular expressions produces neither of these.

2. It folds the count down instead of inflating it

The report opens with this.

Read the count with one caveat: the 20 are not 20 distinct defects.

It then names three clusters itself.

Cluster	Findings	What it actually is
Permission allowlist	3	The same line of `settings.json` seen through three lenses
Log-guard leaks	8	Three hooks protecting the same asset, all failing open because of one shared constant
Child-agent execution	4	Two paths where a skill launching a child Claude executed generated strings directly

Those three clusters account for 15; the remaining 5 stood alone. The report describes the whole as "closer to eight problems" when counted by root cause.

It goes further and names a single repair as the highest-leverage one in the eight-finding cluster — fix that one spot and seven close at once. In a field where finding count tends to be treated as a performance metric, the report folds its own count down for you.

3. The machine-readable side lets you sort by what you care about

The report body is long, so start with the jsonl for the shape of it.

# Run in the parent directory of CLAUDE-SECURITY-<timestamp>/
python3 - <<'PY'
import json, collections, glob
path = glob.glob("CLAUDE-SECURITY-*/CLAUDE-SECURITY-RESULTS.jsonl")[0]
with open(path) as f:
    rows = [json.loads(line) for line in f]
print("count:", len(rows), collections.Counter(r["severity"] for r in rows))
print("where:", collections.Counter(r["file"].split("/")[0] for r in rows))
PY

count: 20 Counter({'MEDIUM': 15, 'HIGH': 5})
where: Counter({'hooks': 10, 'skills': 5, 'settings.json': 3, 'scheduled-tasks': 1, 'agents': 1})

Each line carries fields including id, title, severity, confidence, file, line, exploit_scenario, preconditions, recommendation, and cwe_id.

confidence is worth reading alongside severity, because that value is set by the panel's vote.

Confidence in this report is clamped by that vote — only unanimous panels claim high.

How far you can trust it

The tool discloses its own limits. Three things that static-analysis reports tend to omit were stated outright.

Limits the report writes down itself

Some candidates go unreviewed. In my run, 38 candidate sites never reached the panel after dedup and capping. The earlier "45 candidates × 3 votes" counts only what reached the panel; these 38 are not in it. The report states plainly that absence from the report is not evidence of absence in the tree
It names what it excluded. In my run, three vendored Python virtualenvs and __pycache__ — with the note that if a dependency inside those virtualenvs is itself compromised, this scan did not see it
It accounts for coverage. Every top-level directory must be either scanned or set aside with a stated reason, and that reconciliation runs before the search begins. All 11 of my directories were accounted for

It executes no code

No tests run, no exploit fires, no proof of concept is validated. Everything is derived from reading. The heaviest finding in my run — an arbitrary code execution path — was confirmed not by planting an attack file but by reading it against a similar neighboring hook (which turned out to already carry the same defense).

That is the safe design, but the flip side is that reproducible confirmation becomes your job. I wrote a failing test first for each fix before implementing it.

Anthropic states that it is nondeterministic

From the README:

Scans are nondeterministic. Two scans of the same code can surface different findings.

It goes on to say that because it reasons about code the way a human security researcher does, it complements rather than replaces SAST, dependency scanning, and code review. This is not a drop-in substitute for static analysis.

It writes nothing

A scan only reads. Even the patch-generation job leaves your working tree untouched: nothing is committed, pushed, or opened as a pull request. Patch files land on disk and you decide whether to apply them.

What to do afterward

How to order the fixes

I ranked them in four tiers.

Priority	Condition	Why
1	Independently sufficient for arbitrary code execution	It bypasses every other defense, so fixing anything else while this stands is pointless
2	A cluster that reduces to one constant or anchor	One fix closes several findings at once
3	A definition with the trust direction inverted	The code may be correct while the premise of the judgment is broken
4	Documentation that contradicts reality	The harm is indirect, but it gives reviewers false reassurance

Priority 2 is handed to you by the report, so look for wording to the effect of "highest-leverage" in the body before planning anything.

You do not have to take the recommendation as given

The recommendation field is accurate, but it is not the only answer. One example.

My settings.json carried entries like Bash(bash:*) and Bash(python:*) that grant a whole interpreter. With those present, the other ~80 narrowly scoped grants are moot, because bash -c '<anything>' matches by prefix.

The report recommended moving these into permissions.deny. I chose removal from the allowlist instead.

deny is a refusal, so it makes legitimate execution permanently impossible
Dropping an entry from the allowlist merely returns it to a confirmation prompt in auto mode, which hands the judgment back to a human

The replacement shape is a shift from "grant by interpreter name" to "grant by the script you actually run."

-      "Bash(python:*)",
-      "Bash(python3:*)",
-      "Bash(sh:*)",
-      "Bash(bash:*)",
-      "Bash(node:*)",
-      (export / source / command removed too — 8 entries in all)
+      "Bash(bash ~/.claude/hooks/:*)",
+      "Bash(bash ~/.claude/scripts/:*)",
+      "Bash(bash ~/.claude/tests/:*)",
+      "Bash(python3 ~/.claude/skills/:*)",
+      "Bash(python3 -m pytest:*)",
+      "Bash(python3 -m scripts.:*)",

The finding is right; the recommendation is one option — that is the distance to keep.

Day to day, put it on Scan changes

Running the whole-repository scan every time is not realistic. For routine use, Scan changes is the one.

It targets a branch diff, a pull request diff, or a single commit
At medium, a diff of at most 5 files and 300 changed lines runs the lighter single-researcher shape instead of the full component matrix (the panel verification is unchanged)
Only committed changes are in scope. Uncommitted work in the tree is not part of any diff, so commit or stash first

If you want the patch-generation job, scan with a clean working tree. When the run's stamp carries revision.dirty: true, the job stops before drafting anything. Mine ran dirty, so that path was unavailable.

The scanner tripped one of my own guardrails

My setup includes a homegrown mechanism that detects attempts to abuse the auto-mode confirmation skip. During the scan, one panel voter systematically searched the Claude Code binary for permission-bypass symbols and set it off.

That candidate was rejected 0-3 by the panel, but the report records the behavior rather than burying it. The original phrasing was precise about why — a scanner probing its own guardrails is exactly the thing a reader should be told about.

Summary

As a measurement of the beta, here is what to take away.

Cost — one configuration directory (11 components) at medium came to 189 subagents over 2 hours. Put routine use on Scan changes instead
Character of the output — it produces findings regular expressions do not. Configuration and natural-language instruction files enter the audit scope, and documentation that contradicts reality gets flagged
Precision, anecdotally — of the 20 findings delivered, I rejected none as a false positive. One repository and one run, but this is not a tool built on the assumption that you skim and discard
How to read the count — it does not inflate the number; it folds the findings into clusters and names the highest-leverage repair. Look there first
Bounds of trust — unreviewed candidate counts, excluded areas, code never executed, nondeterminism: the report discloses all of it. A complement to static analysis, not a replacement

To be straight about it: I have not re-scanned after the fixes. I have not asked the tool itself to confirm that the findings are closed, and since it is nondeterministic there is no guarantee the same result would come back anyway. Taking that into account, treating it as "one more reviewer on the team" seems like the right weight for now.

herdr, a tmux for AI Agents — Until the Editor Disappeared

Tatsuya Shimomoto — Mon, 20 Jul 2026 13:00:06 +0000

What this article covers: how to build a terminal environment where you can monitor multiple Claude Code sessions with live status, come back to the same sessions after stepping away or over SSH, and — the interesting part — let the agents reorganize their own screen layout. It's a field log from installation to the point where the editor's (Zed's) role shrank from "writing" to "sign-off" (reviewing and approving) to "gone."

Intro — the "where do I put them" problem of parallel agents

Once you start running multiple Claude Code or Codex sessions in parallel, you hit these walls:

Terminal tabs pile up, and remembering "which tab was doing what" costs you time every single round
Quitting your editor or terminal app kills the agent sessions running inside it
You can't check on your running agents from your iPhone or another machine while away from your desk
Agents can't touch their own execution environment (pane layout, workspaces) at all. Rearranging things is always manual human work

This article solves these with herdr (an agent multiplexer — a terminal multiplexer built for agents). I'm a Zed user, and right after installing it my verdict was "isn't this redundant with what Zed already does?" — until the moment I let an agent manipulate the layout itself, which flipped my evaluation.

And the flip didn't stop there. Once herdr became my main battleground, the editor's (Zed's) role shrank from "a tool for writing" to "a tool for signing off on agent output" — and eventually it lost even that role and I stopped opening it. This is the whole story, from installation to the editor becoming unnecessary, with the configs and commands that actually worked.

Prerequisites

macOS + Homebrew (herdr also runs on Linux)
You're already using CLI agents such as Claude Code
This article is based on herdr v0.7.4 (verified 2026-07-18). The tool is only about three and a half months old, so the command surface may change
The editor is Zed (a recent build with the markdown_preview_* settings), and the terminal I end up choosing is Ghostty (the how and why comes later)
Pane IDs in this article (w5:p8 etc.) are real values from my environment. Yours will differ

What herdr is — only two real differences from tmux

herdr is an "agent multiplexer that lives in your terminal." It's a single Rust binary, and like tmux it persists sessions in a server process (the prefix key is even tmux-compatible: ctrl+b). GitHub is ogulcancelik/herdr, dual-licensed AGPL-3.0-or-later + commercial (stated in the LICENSE file).

The essential differences from tmux come down to two things:

Semantic agent state tracking — it auto-detects agents inside panes and lists their state — working / blocked / done / idle / unknown — in a sidebar. You can see "which one is waiting on me" without visually patrolling every pane
A socket API — pane splitting, command execution, output reading, and layout changes are all controllable from external processes (CLI commands like herdr pane run are wrappers around a Unix-socket API). Which means an agent can operate its own execution environment

The screen has three levels: workspace → tab → pane. From the CLI you address them as w1 (workspace 1), w1:t1 (tab 1 inside it), and w1:p1 (pane 1). Read the command examples below with this notation in mind.

Installing — one brew command

It's bottled in homebrew-core, so brew it is. I skipped the official site's curl | sh because piped install scripts are hard to audit. Including config generation, it's four lines:

brew install herdr                                    # v0.7.4
mkdir -p ~/.config/herdr
herdr --default-config > ~/.config/herdr/config.toml  # 305-line baseline config
herdr config check                                    # → config: ok

Running herdr starts the TUI, and the server (herdr server) comes up automatically. No login-item daemon (brew services) was needed.

As a sanity check, run a smoke test of the socket API. This doubles as the minimal example of "letting an agent operate the environment" covered later:

# Create a workspace, run a command in a pane, wait for output, read it back
herdr workspace create --cwd ~ --label smoke-test --no-focus
# → Returns workspace_id and pane_id as JSON. The lines below assume it returned
#   "w1:p1" — substitute whatever IDs you actually got (if you already have
#   workspaces, you'll get w2 or later)
herdr pane run "w1:p1" "echo herdr-smoke-ok"
herdr wait output "w1:p1" --match "herdr-smoke-ok" --timeout 5000
herdr pane read "w1:p1" --lines 10
# → If the output contains herdr-smoke-ok, you're wired up

If all four pass, layout, execution, and reading are controllable from an external process.

Where Zed is enough, and where it isn't

Let me be honest here: right after installing, my evaluation was "this is barely different from Zed."

Zed announced Parallel Agents on 2026-04-22. The Threads sidebar runs multiple agents in parallel, with per-thread git worktree isolation. Its terminal already does tabs (cmd+N) and splits (cmd+D). If you're at your desk reviewing in a GUI while agents run, Zed's experience is better. herdr's workspace switching looked about the same as switching Zed windows.

In fact, I almost shelved it with a tidy division of "at the desk = Zed, away = herdr."

What changed my evaluation was using the layers that structurally don't exist in Zed:

Layer	Zed	herdr
GUI review experience	◎ Parallel Agents + editor integration	— (terminal only)
Execution persistence	Quitting the app stops execution (thread history survives)	Server-resident. Execution continues with every app closed
External reattach to a running session	Not possible (SSH remote dev exists, but it's a different thing)	`ssh → herdr` from an iPhone etc. drops you back into the same screen
Agents operating their own environment	Not possible	Full control via the socket API

The first row is Zed's win; the bottom three are territory only herdr has (and the first row's verdict changes later, once plugins enter the picture). So it's not competition — it's a complement: herdr fills the "managing many agents" layer where Zed is weak.

One more trap that matters in practice: sessions launched from Zed's agent panel can't be grabbed from outside — not by the official Claude Code app (Remote Control), not by Termius (verified myself). They're sealed inside the editor process. A Claude Code launched as a CLI from a terminal shows up in the official app's list and is reachable via SSH → herdr. If there's any chance you'll want to look at a session from outside, launch it as a CLI.

The next section is the clincher for the complement story.

Letting the agent operate the layout itself

After installing, I wanted my agents — scattered across tabs — on one screen, and on a whim asked Claude Code itself: "consolidate the tabs into one."

Claude Code executed these three commands:

# Move panes from other tabs into tab t2 as splits (executed by Claude Code itself)
herdr pane move w5:p5 --tab w5:t2 --split right --no-focus
herdr pane move w5:p7 --tab w5:t2 --split down --target-pane w5:p4 --no-focus
herdr pane move w5:p6 --tab w5:t2 --split down --target-pane w5:p5 --no-focus
# → Panes scattered across tabs become a 2×2 grid on one tab. Emptied tabs auto-close

Two agents were in working state at the time, and the layout changed without stopping a single one. All I did was say one sentence. No worrying about reopening panes and interrupting processes, no assembling the sequence of move commands by hand.

Here's what it looked like. Starting from one-tab-per-agent (Before):

Before: four tabs in the bar. Only one session visible at a time

I ask the Claude Code in another pane to "consolidate them":

The instruction is a single sentence. Claude Code surveys the current tab structure via the socket API, assembles the pane moves, and runs them

And everything lands in a 2×2 grid on one tab:

After: every agent on one screen. Running processes never stopped

Agents know where they are through environment variables:

$ env | grep -i herdr    # home directory in the output replaced with ~
HERDR_ENV=1
HERDR_PANE_ID=w5:p8
HERDR_SOCKET_PATH=~/.config/herdr/herdr.sock
HERDR_TAB_ID=w5:t2
HERDR_WORKSPACE_ID=w5

So an agent knows "which pane am I in," and from there it can split a pane next door, run a command, and read the result. The shape of the execution environment itself becomes one of the agent's tools. Incidentally, the session writing this very article is a Claude Code inside a herdr pane — herdr agent list shows it as working on this draft.

git worktree integration was also a one-liner (verified myself):

# Create a worktree + branch + new workspace in one shot
herdr worktree create --workspace w6 --branch feature-x
# → Creates a worktree at ~/.herdr/worktrees/<repo-name>/feature-x/ and
#    opens a workspace labeled "feature-x" with that as its cwd

The same "one room per branch, work in parallel" structure as Zed Parallel Agents — except an agent can assemble it from the CLI.

Gotcha: a zoomed tab refuses layout changes

pane move can return changed: false (reason: "zoomed_tab") and do nothing. If the target tab is in zoomed view, layout changes are rejected by design. Run herdr pane zoom --off to unzoom first.

Gotcha: an omitted target means the focused pane, not the caller

When I had Claude Code run herdr pane split, the pane split in a different workspace — the one I happened to be looking at — not the intended one. That's because when you omit the pane target, it resolves to "the pane focused in the TUI," not "the pane the command was invoked from" (stated in the official CLI reference; passing --current resolves to the calling pane).

When agents drive the CLI, don't let them omit the target; have them pass their own position explicitly via --current or HERDR_PANE_ID.

Gotcha: server lifetime and startup directory

The server belongs to the parent process that started it. If herdr server gets started from Claude Code's shell, it can die with that session. After my experiments I stopped the server once and restarted herdr myself so the server ownership sat with my own process
The startup cwd only matters once, at first session creation. cd-ing into a repo and running herdr just reattaches to the existing session from the second time on. Add repos from inside herdr with herdr workspace create --cwd --label. It took a mental-model move from "cd to navigate" to "switch workspaces to navigate"

Does the sidebar show state, or history?

Running herdr inside Zed's terminal gives you this nesting:

The spaces / agents sidebar on the left. Each agent gets one line with its state and remaining context

What I noticed in use: herdr's sidebar only shows rows that map 1:1 to living processes. When an agent finishes, its row disappears. It structurally cannot get cluttered.

Zed's Threads sidebar, by contrast, is a history list. Sessions that ended days or months ago sit alongside active ones, projects interleaved. Finished work keeps occupying your screen as "attention inventory."

Zed's Threads sidebar. Sessions from 4 days, 2 weeks, and 2 months ago mixed into the same list as active ones

Does a list show current state, or past history? That felt like a litmus test for UIs in the parallel-agent era.

This realization changed how I use the whole screen, too. I used to keep a permanent two-way split of editor and CLI — but I only actually looked at the editor "when reading code," and the rest of the time it was dead space.

Before: permanent two-way split. The editor side spends most of its time unwatched

After: herdr (the CLI side) is the main screen; when I read code I flip the editor to full screen with cmd+shift+backtick

In agent-driven development, the primary screen flips from the editor to the agent CLI, and the editor becomes the on-demand side. Same root idea as the sidebar: align "what's in view" with "what's in use right now," and you stop leaking attention.

For a while, the answer was "Zed at the desk, herdr everywhere else"

Even after the evaluation flipped, my placement answer briefly returned to that first instinct (the "shelve it" plan above). Only now it wasn't a passive shelving but an active division of labor: GUI review at the desk goes to Zed; being away, persistence, and delegation to agents go to herdr. Monitoring and reattaching from the iPhone is measured and working on the herdr side.

Field notes on away-from-desk operation (iPhone mirroring, sleep behavior) - The iPhone connection path (Tailscale + Termius) is identical to [the one I wrote up in the tmux days](https://dev.to/shimo4228/running-claude-code-from-iphone-via-ssh-tmux-4c10) — just swap tmux for herdr. If conversational UX is your main goal, [my write-up on the official Claude Code app](https://dev.to/shimo4228/claude-code-from-iphone-plugging-3-holes-in-remote-control-17cf) is a better fit. The split is: "conversation = official app / fleet monitoring = Termius + herdr" - Attach to the same session from the Mac's terminal and from Termius on the iPhone, and the two become **mirrors of the same screen**. Workspace switches and focus moves on one side show up on the other with barely any lag ![Termius on iPhone mirroring the same herdr session as the Mac](https://raw.githubusercontent.com/shimo4228/zenn-content/main/images/herdr-termius-iphone-sync.png) *The iPhone side. It syncs with the Mac's screen with barely any lag* - The display size **syncs to the smaller client**. While you're driving from the iPhone, the Mac's terminal also rewraps to the iPhone's screen width ![The Mac side while the iPhone is in control. Pane contents rendered at the iPhone's screen width](https://raw.githubusercontent.com/shimo4228/zenn-content/main/images/herdr-termius-mac-sync.png) *The Mac at the same moment. While the iPhone is driving, the Mac's display syncs to iPhone size* - **Agents make no progress while the Mac sleeps** (sleep suspends process execution, launchd-managed or not). Sleep doesn't kill the server — sessions are intact on wake (measured myself: sleep → wake, session survived). If you want agents running while you're away, configure the Mac not to sleep

But this division of labor didn't last. A few days of using herdr as the main battleground ran me into the next question: Claude Code writes the code — so what am I actually doing in the editor? Here begins the second half.

The three jobs left in Zed — from "writing tool" to "sign-off tool"

Once you hand all code-writing to the CLI (Claude Code), the editor's "write" function has long since retired. I was still keeping Zed around for three things:

Reading — following multiple files side by side on one screen (multibuffer)
Diff review — eyeballing the changes
Checking Markdown — polishing how articles and design notes look

In other words, everything left was on the "reading" side. So I flipped the framing: optimize Zed as a tool for reading and signing off.

The first thing that paid off was the zed CLI. Append :line to a file path and it opens with the cursor on that line:

# Open a specific line (line:column also works)
zed src/contemplative_agent/cli.py:715

# Open several sign-off points at once
zed src/core/metrics.py:88 src/cli.py:715 tests/test_cli.py:350

# --wait: don't proceed to the next command until the human closes the file
zed --wait changed_file.py

This reverses the direction of the sign-off flow:

Before: the agent tells you path:line as text → the human hunts for it in the editor
Now: Claude Code pinpoints the line with grep, runs zed <file>:<line> → and hands the human an editor already open at the spot

The human stops searching and just reads the line being pointed at. Add --wait and you get a structural pause — "don't proceed until the human has laid eyes on this file" — in a single command. One caveat: closing the file only means "seen." It can't distinguish approve from reject, so don't chain irreversible operations like commits onto --wait; run those explicitly, human-side, after confirming.

I didn't uninstall Zed at this point. Its resident cost while unused is zero, and the cost of rebuilding a "reading" replacement on the terminal side is real. Let actual usage make the call — that was the judgment.

The Japanese-text rendering wall, and a host swap — to Ghostty

Once I started optimizing for "reading and signing off," I hit a rendering-quality wall next. Two problems, both around Japanese text.

First: line spacing in Zed's Markdown preview. Preview a Japanese Markdown document and the lines are cramped and hard to read. Digging in, the preview's paragraph line height is hard-coded at 1.3 with no user setting (zed#56111 has a request for Japanese-friendly settings). The preview-only font settings (markdown_preview_font_family etc.) let you raise the size, but the 1.3 ratio itself can't be changed, so it's not a real fix. I moved final article checks to npx zenn preview (the same rendering as production) and demoted Zed's preview to "glances while writing." Zed can't finish the reading experience — the first crack.

Second, and nastier: Markdown tables emitted by Claude Code came out misaligned in the terminal, with broken rules.

My suspect was characters whose East Asian Width (Unicode's character-width classification) is "ambiguous" — box-drawing characters and the like. That width can be interpreted differently by each layer: Claude Code computing character widths, herdr laying out the grid, and the terminal actually drawing. When the layers disagree about width, tables break. (Other rendering factors, like font fallback, can produce similar symptoms — so this is a leading suspect, not a confirmed cause.)

I couldn't pin down the exact mechanism, but which layer was at fault could be isolated with a controlled swap: replace only the host (the terminal doing the rendering). Piping the same Claude Code + herdr output into Terminal.app, the tables lined up cleanly. The misbehaving layer was confirmed to be Zed's terminal rendering. For display bugs in a multi-layer stack, this "swap exactly one layer" move is the fastest isolation you can do.

Host	Table correctness	Color (True Color)
Zed's terminal	Breaks	Accurate
Terminal.app	Accurate	Not supported
Ghostty	Accurate	Supported

I ended up on Ghostty. It draws box-drawing and block characters itself instead of relying on the font, so tables don't break, and colors come out accurate.

"After the fix." Same Claude Code + herdr output — on Ghostty, tables with box-drawing rules line up.

With the host settled, unify the look across the nesting too. If the host (Ghostty) and the TUI (herdr) disagree on color scheme, every glance between them adds a small snag. herdr can auto-follow the host's light/dark appearance:

# ~/.config/herdr/config.toml
[theme]
name = "tokyo-night"
auto_switch = true          # follow the host's light/dark
dark_name = "tokyo-night"
light_name = "tokyo-night-day"

herdr server reload-config applies it without complaint, and the three tiers — macOS appearance → Ghostty → herdr — switch together.

The same two-pane screen in light and dark. It follows the host's (Ghostty's) appearance, TUI and all.

Why herdr gets details like Japanese input right — a solo dev and his bots

What surprised me in use: herdr has engineering effort in places as practical as Japanese input (IME) handling. The settings dialog has entries like these:

For example, with a Japanese IME active, pressing a key sequence like ctrl+b v can get the v swallowed by the IME. herdr's countermeasure: switch the input source to ASCII only while accepting a key sequence, then switch back. There's also a feature that persists pane scrollback across server restarts.

How does a tool get this level of polish? The repository's raw data hints at the reason:

gh api repos/ogulcancelik/herdr \
  --jq '{created: .created_at, stars: .stargazers_count, license: .license.spdx_id}'
# → {"created":"2026-03-27...","stars":17794,"license":"NOASSERTION"}
#   (license shows NOASSERTION because GitHub can't auto-detect
#    the AGPL + commercial dual license)

gh api "repos/ogulcancelik/herdr/contributors?per_page=6" \
  --jq '.[] | "\(.login): \(.contributions)"'
# → ogulcancelik: 979 / kangal-bot: 54 / github-actions[bot]: 43
#   akbash-bot: 16 / human contributors are at 4 commits or fewer each

At 113 days since creation (as of 2026-07-18) it's past 17k stars, and it hit Hacker News on 2026-06-29 (166 points, 110 comments; all figures as of writing). Meanwhile, roughly 979 of the commits are the author's own — it's effectively a solo project. The fun part: the #2 and #4 contributors are bots (kangal-bot and akbash-bot). Presumably these are agents the author operates, committing under their own accounts.

An aside: kangal and akbash are both real breeds of Turkish livestock guardian dogs. Perhaps a deliberate match with the name herdr (one who herds) — though that's speculation about naming intent. What the API establishes for certain is only that nearly all commits are attributed to the author himself and to bot-named accounts; that the author operates those bots as agents is itself an inference from circumstance.
The tool's defects bounce back into the developer's own velocity every day. So requirement discovery gets fast. Tools of the agent era fit the hand best when they're built together with agents — and that structure was right there in the contributor list.

The ending — Zed didn't even get to be the sign-off viewer

Everything so far was about "optimizing Zed as a sign-off viewer." Then even that role got taken by the herdr side.

The trigger was two herdr plugins:

file-viewer — read-only, git-aware file browsing
reviewr — shows diffs in a sidebar and sends line comments back to the agent

file-viewer covers "reading"; reviewr covers "review the diff and send feedback" — both now inside herdr.

The agent on the left and the file it opened on the right, on the same screen. "Reading" moved to the terminal side.

The reviewr screen. The right sidebar shows the uncommitted diff and the changed-file list. What's on screen is the diff of this very article's consolidation work.

Here the flow comes full circle. Against the "agent → human" bridge from the earlier section (zed file:line), reviewr builds the reverse "human → agent" bridge (comment send-back). The sign-off loop closed inside herdr.

This plugin marketplace just auto-indexes a GitHub topic — there's no review process. Installing a plugin is the same as granting code execution, so I read the manifest and install script before installing (for file-viewer: SHA-256 verification present, no automatic hooks — confirmed).
The result: the zed file:line bridge stopped needing to be crossed within hours of being built. The third job, checking Markdown, was covered too — glances-while-writing by file-viewer's rendered view (the v key cycles diff ⇄ rendered ⇄ syntax), and final checks already consolidated onto npx zenn preview — leaving no reason for Zed to stay. Zed lost even the sign-off viewer seat, and within a day I'd stopped opening it.

The final stack: Ghostty + herdr + plugins + Claude Code. There is no editor.

Closing — the shape of the execution environment becomes the agent's tool

Seen as "a tmux successor," herdr looks redundant next to Zed. My evaluation flipped when the socket API let an agent operate its own environment. Splitting panes, opening rooms, cutting worktrees — the "workspace housekeeping" humans used to do becomes delegable, wholesale.

And the flip kept going, all the way to a change of tools. From "isn't Zed enough?" to "sign-off only" to "gone." While every layer — editor, terminal, harness (Claude Code's own subagent machinery) — is absorbing "agent orchestration" features at the same time, herdr's distinctive answer is that it made the layout of the execution environment itself something agents can operate. Each layer slimming down to a single function feels less like regression than like attention design: the work of paring what's on screen down to a state with not one layer surplus to its current role.

That said, "best" here means best for the current role, not forever. The honest way to test the judgment is against actual usage, so in a few weeks I plan to look back at "how many times did I open Zed?"

It's one brew command to try. If you're running two or more Claude Code sessions in parallel, start by watching your own agents show up in herdr agent list.

What Do My AI Agent's Logs Look Like in OpenTelemetry?

Tatsuya Shimomoto — Thu, 16 Jul 2026 13:00:04 +0000

What this article covers: what you can see when you plug homegrown logs into OpenTelemetry. I convert them to OpenTelemetry format, visualize them, and sort out which elements are worth borrowing and which you can pass on.

I wanted to bring OpenTelemetry to my AI agent (telemetry = the standard machinery for recording and emitting data about a system's behavior; OTel from here on). The typical adoption path is to embed the SDK in the application itself, emit telemetry as it runs, and keep shipping it to a monitoring backend via a Collector (a resident process that receives, transforms, and forwards). But maybe you've hit walls like these:

Running an SDK and a Collector around the clock is heavy for a personal-scale agent. There's no ops team watching a monitoring dashboard
I already have my own structured logs. I don't want to instrument the same information twice
The standard for which attribute names to record LLM calls under (the GenAI semantic conventions) is still in development. The spec may still move, so it's hard to judge how far to lean on it

Contemplative Agent, the agent I operate, was in exactly this situation. It's a local-LLM agent that autonomously writes posts and comments on Moltbook (a social network where AI agents post and comment on each other). The text it reads from the feed is written by other agents, and prompt injection (smuggling instructions into a post body to steer the agent reading it) is a daily occurrence — an environment where input cannot be trusted.

When the agent misbehaves in this environment, investigating the cause requires being able to trace, precisely and after the fact, "what it read from outside and what it emitted at that moment." So I had designed all external input/output to be kept in full in an audit log (an append-only record for reproducing past decisions exactly as they happened). The "own structured logs" in the second wall above is this audit log.

At the same time, nobody in my environment watches telemetry continuously. What I need is "the ability to examine past runs in a standard UI when something is worth investigating." If that's the requirement, the telemetry doesn't have to be flowing in real time.

So this article does two things.

Instead of putting OTel into the production runtime, I built an after-the-fact conversion as an experiment. It takes the audit logs I already have, converts them into traces (a record of one processing run as a collection of timestamped intervals) after the fact, and ships them over OTLP (OTel's standard transport protocol) for visualization
I compared my log schema against the OTel standard and added the 2 fields it was missing to the agent's own logs: run_id and session_id, which identify a run

The first is a small conversion script — zero changes to the agent body, two dependencies. My local logs became visible as a waterfall (the chart that stacks processing intervals along a timeline) in Jaeger (an OSS trace viewer). On a day that had an incident, the picture tells you the moment you open it.

This article is a sequel to my previous piece, "Why Did My Agent Decide That? 3 Observability Patterns". There I wrote that what traces and metrics tell you by default stops at "what the request did." This is the continuation: what, then, can homegrown logs and the OTel standard actually connect? — answered by doing the conversion for real. It also reads fine on its own.

Prerequisites

Conversion target: append-only JSONL (one JSON per line) logs the agent has already written. Three kinds: LLM call telemetry / API audit / CAPTCHA solver audit
Python 3.10+, uv for package management
Only two dependencies: opentelemetry-sdk + opentelemetry-exporter-otlp-proto-http (both on the conversion-script side; nothing is added to the agent itself)
Viewer: Jaeger v2.19.0 (single binary; no Docker required, storage is in-memory)
Signals: traces only (of OTel's three signals, metrics and logs are out of scope this time)
The conversion script is public: contemplative-agent-otel — a small implementation, about 800 lines in total, built on the assumption that you swap in records.py (log loading and normalization) / mapping.py (the attribute-name mapping table) for your own log format

The processing flow is 5 steps.

Read the already-written JSONL logs
Normalize the 3 log kinds into a common record (records.py)
Attach standard attribute names to each record and convert it into a span (one interval of a trace) (mapping.py)
Send to Jaeger over OTLP/HTTP
Check the waterfall and the attributes in the Jaeger UI

I decided whether to adopt OTel by asking "who reads the telemetry?"

Put OTel into the agent, or pass? I organized this decision around a single question: "who is going to read that telemetry?" The reason is that OTel traces and my audit log, even when they record the same event, have different readers (consumers). Take one LLM call: on the trace side, what you want to see is the model name, token counts, duration, and whether it errored. On the audit-log side, what you need is the prompt body itself, the caller, and the identifiers used for reproduction.

	OTel traces	My audit log
Primary reader	People watching dashboards; alerting	People investigating incidents; the replay script used for reproduction
Prompt and other bodies	Not recorded by default (opt-in)	Stored in full (base64 + sha256)
Nature of the data	May be sampled; may be volatile	Append-only; every record; never deleted
Question it answers	"What's slow or broken right now?"	"Why did that decision come out the way it did?"

The right column is not a property of audit logs in general — it's a design I chose because my goal is offline reproduction of incidents. Your logs may well not store everything. What generalizes is the yardstick: different consumers mean different retention policies.

The emblematic case is body text. The GenAI semantic conventions specify that prompt and response bodies are not recorded by default (they end up on a viewer's screen, so not carrying sensitive data is the default). My audit log does the exact opposite: full retention. In incident investigation, without "the exact bytes the decision saw," you can't reproduce it offline.

So this isn't a question of which one is right: same event, different consumers, opposite retention policies. Once you frame it this way, the choice stops being binary.

Put OTel into the runtime (if someone reads telemetry in real time)
Do nothing
Convert existing logs into traces after the fact — get the connection to the standard vocabulary, and the visualization, without touching the runtime

This article is an implementation of option 3.

That said, option 3 only held up because two conditions happened to be true for me: nobody reads telemetry in real time, and the incident-investigation reader was already well served by a thorough audit log. If either is missing, the answer changes. Even with no monitoring team, if you don't have structured logs yet, going straight for option 1 (the runtime SDK) is probably faster. "No monitoring team = no OTel" is not a general rule.

You might also wonder: "isn't using only part of the standard just cherry-picking the spec?" OTLP is a public protocol, and the semantic conventions are an agreement on attribute names. You don't have to instrument the app with the SDK — any program that speaks OTLP can feed data in. This is a usage the standard anticipates, and this article's conversion script simply uses the OTel SDK as "a library for offline export."

What I borrowed from OTel, and what I passed on

Before getting into the implementation, here's how I judged each element of OTel, up front. The biggest lesson this time was that it's not an all-or-nothing "adopt the standard or don't" — you can pick borrow-or-pass per element.

OTel element	Verdict	Why
GenAI semantic conventions vocabulary (`gen_ai.*`)	✅ Adopt	External tools and people can read it without explanation. Copying attribute names costs zero dependencies
Run-ID machinery (trace ID / session ID equivalents)	✅ Adopt (implemented `run_id` / `session_id` in the logs)	One field turns trace reconstruction from estimation into measurement
OTLP (transport protocol)	✅ Adopt (as the exit of the offline conversion)	A common language every viewer speaks. Costs only 2 packages
Runtime SDK instrumentation	❌ Pass	Would double-instrument alongside the existing audit log, and add dependencies to the agent
Running a resident Collector	❌ Pass	Heavy for a single process at personal scale. On-demand conversion when needed is enough
Bodies not recorded by default (redaction)	Partial	Followed on the trace side (don't carry untrusted text onto the screen). The audit log does the opposite — full retention, needed for offline reproduction
Importing attribute names from the semconv package	❌ Pass	Import paths are still unstable while in development, so attribute names are self-managed as string constants

Passing on an element doesn't mean that part of OTel has no value. There are mainly 4 things that are hard to get without runtime instrumentation.

Exact causal structure — which operation called which. Only IDs issued at execution time capture this precisely
Context propagation across services — the machinery that stitches one trace across multiple services
Real-time visibility — latency and errors happening right now, visible as they happen
Metrics — numbers aggregated continuously while running. You can technically produce them from an after-the-fact conversion too, but they're fundamentally a signal meant to be measured live

Of these, the last 3 have no reader in my operation — a single process, with nobody watching continuously. The only one genuinely missing is the first, and its minimal patch is row 2 of the table, the "run-ID machinery."

The sections below implement the ✅ elements in order.

My logs mapped almost 1:1 onto the standard vocabulary

My homegrown telemetry writes one line per LLM call. A line looks like this (simplified):

{"ts": "2026-07-15T09:12:03+09:00", "model": "gemma4:e4b", "prompt_eval_count": 1560, "eval_count": 210, "duration_ms": 36400, "done_reason": "stop"}

The next table matches these fields against the GenAI semantic conventions attribute names (the naming table for how to record model name, token counts, finish reason, and so on in OTel).

My log field	OTel attribute
`model`	`gen_ai.request.model`
`prompt_eval_count` (input token count)	`gen_ai.usage.input_tokens`
`eval_count` (output token count)	`gen_ai.usage.output_tokens`
`done_reason` (`stop` / `length`)	`gen_ai.response.finish_reasons` (array-typed, so wrap as `[done_reason]`)
`num_predict` (generation cap)	`gen_ai.request.max_tokens`
`temperature`	`gen_ai.request.temperature`
`error_kind` (`timeout` / `http_429` etc.)	`error.type`
`ts` + `duration_ms`	start / end time of the span (one interval of a trace)
`caller` (which processing stage made the call), `prompt_sha256`, etc.	No counterpart → custom namespace `ca.audit.*`

It mapped almost 1:1. For a simple LLM call log like this one (model, token counts, stop reason, parameters), I suspect the set of things you'd want to record converges to the same collection no matter who designs it (logs that also cover tool calls, streaming, or routing won't map this cleanly).

This is where riding the standard vocabulary pays off. If the attribute name is gen_ai.usage.input_tokens, external tools and people alike can read it with no explanation.

The interesting part is what didn't map. caller (the key I use to aggregate logs by processing stage during incident investigation) and prompt_sha256 (a hash for matching identical prompts without storing the body) have no standard counterpart. These two are investigation keys specific to my operation. For just looking around in a UI, the standard attributes are mostly enough; for offline reproduction of an incident, these are the ones that matter.

Practical handling of the Development status (official status name: Development)

The GenAI semantic conventions (semconv below) are in Development status as of this writing (2026-07) (canonical repository). The opentelemetry-semantic-conventions package does have attribute-name constants, but they live under an underscore-prefixed (= unstable) _incubating import path.

The conversion script doesn't import from the package. Instead, the attribute names are defined as string constants in my own code, with the referenced semconv version pinned in a comment. If the standard moves, catching up is a single-file change.

The conversion script has exactly 3 design points

Let's pin down spans properly here. At the top I described a trace as "a collection of timestamped intervals." Each of those intervals is a span: it has a name, start and end times, and attributes (key-value pairs), and spans connect through parent-child links into one trace. In a Jaeger waterfall, one row is one span.

So the core of the conversion is "read JSONL, create spans with past timestamps." There were 3 design decisions.

1. Spans can be created at past timestamps

The OTel SDK lets you explicitly set a span's start and end times in epoch nanoseconds. I restore them directly from the log's ts and duration_ms.

```python:emit.py (excerpt, simplified)

Create the span at the log's recorded time (the core of the conversion script)

span = tracer.start_span(
name, # e.g. "text_completion gemma4:e4b"
context=parent_ctx,
kind=kind, # SpanKind (CLIENT etc. for LLM calls)
attributes=attrs,
start_time=start_ns, # ts converted to epoch ns
)
if is_error:
span.set_status(Status(StatusCode.ERROR, error_type))
span.end(end_time=end_ns) # start_ns + duration_ms




For logs that don't record a duration (API audit, solver audit), I made the spans **zero-width**. Giving them a plausible width by estimation would mean fabricating latency that was never measured. Jaeger draws zero-width spans as thin markers, so they're also visually distinct from the LLM spans that carry real durations. The fact that "logs without a duration become zero-width" was itself feedback into the log schema (it pinpointed the next improvement candidate: add elapsed time to the API audit log too).

### 2. Trace grouping is reconstructed from gaps in time

A trace is normally bundled by IDs issued at execution time. Past logs don't have those. So I merge the 3 log kinds in timestamp order and **start a new trace whenever a gap of a fixed length (default 300 seconds) opens up**. A schedule-launched agent repeats "a few minutes of activity → long silence," so this cut lines up well with an actual single run. Conversely, for workloads where the logs never pause — parallel runs, always-on workers — this cut will merge things incorrectly.

But it is still an estimate. I put a `ca.convert.grouping = "time-gap"` attribute on each trace's root span, making **the data itself say it is a reconstruction, not a measurement**.

Flip that around and this was **a gap in my own log design that only became visible against OTel**: add one field to every log line — a per-run ID (the equivalent of OTel's trace ID) — and this reconstruction turns from estimation into measurement.

Even if you never adopt the SDK, the one mechanism worth borrowing up front is "stamp every run with an ID."

I implemented it in my agent while writing this article: a single write function shared by all audit logs now stamps `run_id` (per process) and `session_id` (per agent session). That is the only agent-side change in this whole effort, and converting past logs still works with the agent untouched. The converter bundles by ID when `run_id` exists and falls back to time gaps only for older logs. If you're designing logs now, I recommend putting it in from day one.

### 3. Untrusted text is dropped at the conversion entrance

The third design decision is the boundary of what gets onto the Jaeger screen.

The audit log contains raw text that came from outside (CAPTCHA challenge texts, server error response bodies), stored as base64. As described at the top, this agent's input comes from a social network where prompt injection is routine — "strings from outside may be attack input" is the default assumption. Put such text into span attributes, and **attacker-controlled strings flow straight onto the Jaeger screen (= into screenshots, = into this article)**.

The conversion drops bodies at the parser stage and passes through only a sha256 hash and a classification code (`http_400` etc.). There is no "flag to include bodies" either. On top of that, I added a regression test that scans every attribute value of every test fixture and asserts that no body fragment appears. Not "be careful later," but "make it structurally unable to appear, then visualize" — that's the safe side.

To be clear, the goal here is not to keep bodies secret but to avoid carrying external strings onto the screen. The sha256 is a correlation identifier for matching identical inputs, not an anonymization mechanism (short boilerplate strings can be brute-forced back). If your logs hold confidential data, consider a keyed hash (HMAC) or simply not surfacing the hash at all.

## Looking at it in Jaeger — the incident day was obvious the moment I opened it

Jaeger v2 is a single binary; once started, it accepts OTLP directly and keeps everything in memory. Because it takes OTLP directly, there's no need for the Collector I called "heavy" at the top, either.



```bash
# Start the viewer (no Docker; data disappears when it exits)
# For Apple Silicon. On Intel Macs read darwin-amd64; on Linux, linux-amd64
curl -sLO https://github.com/jaegertracing/jaeger/releases/download/v2.19.0/jaeger-2.19.0-darwin-arm64.tar.gz
tar xzf jaeger-2.19.0-darwin-arm64.tar.gz
./jaeger-2.19.0-darwin-arm64/jaeger    # OTLP :4318 / UI :16686

# Convert and send (example run; see the repository README for setup)
contemplative-agent-otel --date 2026-07-15
# => emitted 1031 spans across 5 runs -> http://localhost:4318

A normal day looks like this. Reading the screen: time runs along the horizontal axis, one row is one span, parent-child structure is indentation, errors are red marks. One trace is one agent run, with LLM calls carrying real durations (text_completion gemma4:e4b, 36 seconds or 1.3 minutes) and zero-width API calls lined up in time order.

Open a span and the mapping table from earlier appears directly as attributes. Standard attributes like gen_ai.usage.input_tokens: 1560 sit alongside custom ones like ca.audit.caller: core.skill_selection.

And here's what converting the day with the incident produces. On that day, a retry storm (failed calls whose retries cascade until calls and logs balloon) had in fact hit the LLM backend.

Where a normal run has tens to hundreds of spans, one trace with 30,442 spans and 33 errors towers over everything. The log file for that day was also nearly 30 times its normal size. As numbers, this information was already in the logs — but "you know it's an abnormal day the moment you open it" is a strength unique to trace visualization.

You can follow the errors too. In the run where the CAPTCHA answer was rejected by the server, captcha solve and POST /verify are marked red as a pair. The error bodies were dropped by the conversion, so all the screen shows is the classification: error.type: http_400.

Gotcha: send 30,442 spans at once and they're silently dropped

In the OTel SDK, span.end() doesn't send immediately — a BatchSpanProcessor (an SDK-internal component that queues spans and sends them in batches) does the exporting. In the incident-day conversion, spans beyond the default queue length of 2048 were discarded with only a one-line warning (Queue full, dropping Span.). For real-time instrumentation that's a reasonable self-defense; for an offline bulk conversion, "send everything" is the correct spec.

Since the number of records to convert is known before sending, I sized the queue to the record count and that solved it. If you use the SDK for bulk conversion, watch out: the defaults assume real time.

Summary

Instead of a binary "adopt OTel or don't," I tried the third option: after-the-fact conversion of existing logs. Zero changes to the agent, two dependencies — and you get the Jaeger waterfall, error display, and standard attributes, all of it
My homegrown LLM call log mapped almost 1:1 onto the GenAI semantic conventions. The fields that didn't map remain as a list of "information needed for offline reproduction of incidents"
Traces serve humans investigating on a screen (bodies unrecorded by default); the audit log serves machines re-executing for incident reproduction (bodies stored in full). Different consumers flip the retention policy, so in a conversion, dropping untrusted bodies at the entrance is the safe side
Master 3 things — span creation at past timestamps, zero-width spans, trace reconstruction from time gaps — and any structured log from which times and run groupings can be recovered connects the same way
Add one run-ID field to your logs and the reconstruction itself becomes unnecessary. Even without the SDK, this one mechanism is worth borrowing from OTel up front (I implemented it while writing this article)
This conversion was also a way to measure my own log design against OTel as a yardstick. The missing run ID and the real-time assumptions of the export queue both became visible only against the standard

Fault Injection TDD Found 3 Silent Failures in My LLM Agent

Tatsuya Shimomoto — Wed, 15 Jul 2026 13:00:03 +0000

Pipelines with an LLM in the loop keep hitting walls like these:

The LLM returns JSON in a subtly different shape. Parsing succeeds, but the result silently comes back empty
A local LLM's response was cut off mid-generation (done_reason=length) and you only notice days later
The log only says outcome="error", so you can't tell whether it was a rate limit or a timeout

What they have in common: every test passes, yet production breaks silently. Processing doesn't stop, no exception is raised — the results, or the records of them, quietly degrade. In this article I call this a silent failure.

The local-LLM CLI agent I operate hit this family of failures 5 times. All 5 slipped past the tests I had beforehand and were only diagnosed after the fact. So I adopted this workflow: build a fault catalog (a taxonomy of failures) from the history of production incidents, write tests that deliberately inject those faults first, and land the minimal guard that makes them pass in the same PR. Taxonomically, this is fault injection testing × TDD. What I borrowed — and deliberately didn't borrow — from chaos engineering (the practice of injecting failures on purpose to verify resilience), which inspired the approach, is laid out in the body.

The first application found 3 silent failures that my ~1800 existing tests had missed.

What you'll get from this article: a pattern for adding fault injection testing to a single-process Python app — no daemon, deterministic, pytest-native (fault catalog → RED → GREEN) — plus the details of the 3 silent failures the first application actually surfaced

Prerequisites

Python 3.13 / pytest 9.0 (at time of verification; no bleeding-edge features are used, so nearby versions should work)
hypothesis 6.156.6 / responses 0.26.0 (both added as dev dependencies)
Target: a single-process Python app that calls an LLM server (Ollama etc.) via requests
The LLM call sits behind a swappable layer such as a Protocol (if it doesn't, carving out that layer is step zero)

This article is a sequel to my previous one, "Why Did My Agent Decide That? 3 Observability Patterns". If that one was about recording the agent's decisions, this one is about asserting on that recording channel — and injecting faults up front. It stands alone, though.

Chaos engineering was the inspiration; fault injection testing is what I adopted

My agent operates autonomously on a social network (Moltbook), continuously. As long as it keeps running, I can't know in advance which failure it will meet next. Searching for a way to build "operations resilient to unforeseen failures," I arrived at chaos engineering.

But measured against the five advanced principles of the canonical Principles of Chaos Engineering, I did not follow it verbatim.

Canonical principle	This work
Build a Hypothesis around Steady State Behavior	○ Adopted (asserts against the execution-log (telemetry) channel)
Vary Real-world Events	△ Half (faults come from incident history, but injection is a deterministic replay of a known catalog)
Run Experiments in Production	✕ Rejected
Automate Experiments to Run Continuously	△ CI runs them automatically, but derandomized — a regression test that pins the known, not an experiment that explores the unknown
Minimize Blast Radius	○ Adopted in extreme form (injection happens inside tests, so production impact is zero)

I rejected production experiments for 2 reasons. First, the target is a single local process — there is no "redundant fleet of instances" to kill in the first place. Second, the logs the agent accumulates are primary data that can't be regenerated; random injection in production risks corrupting them. I deferred random exploration to keep CI from going flaky.

The core of chaos engineering is not "testing" (asserting known expected behavior) but "experimentation" (discovering unknown weaknesses) — so once production experiments and random exploration are out, it can no longer be called chaos engineering. As stated up front, the classification is fault injection testing × TDD. What I borrowed from chaos engineering is exactly 2 ideas: "build the failure catalog from real-world events" and "assert steady state through an observation channel."

Tooling-wise, I also skipped the distributed-systems staples and built on pytest. This was the conclusion of pre-adoption external research.

Candidate	Verdict	Reason
chaostoolkit / toxiproxy	Rejected	Declarative experiment runner / resident network proxy. Built for distributed topologies; the scale of the target doesn't match a single local process
pytest fault-injection plugins	Rejected	Nothing usable exists (e.g. pytest-disrupt is a TODO-only scaffold)
agent-chaos	Reference only	Conceptually the closest, but coupled to the Anthropic SDK / DeepEval / pydantic-ai, with no pytest integration and no local-backend support
hypothesis + responses + a hand-rolled injection backend	Adopted	Sits on the existing test infrastructure (pytest) with just 2 dependencies

That settled the approach: inject faults deterministically inside tests. No daemon, no proxy — pytest itself is the execution environment.

The 2 elements I deferred (exploration, experimentation) are not discarded; they remain as the entry points for extending this toward chaos engineering later — a two-tier setup where PR CI stays deterministic while a nightly run randomizes with seed recording (any failing sequence found gets reproduced from the seed and pinned with @example), plus sandbox experiments running the real pipeline in random-injection mode. This pilot leaned deterministic because the state of "even the known incident history isn't asserted" had to be fixed first. Once the known holes are plugged, those 2 entry points are where I'll expand.

The pattern — the fault catalog becomes the test spec

The workflow is 3 steps.

Catalog: classify the history of production incidents into fault classes (failure taxonomies) and diff them against existing tests. The untested classes become the fault catalog
RED: for each fault, assert in a test how the system should behave when it's injected. The current implementation fails it (this is TDD's RED)
GREEN: land the minimal guard that passes the test, in the same PR as the test

The point is that the fault catalog doubles as a spec. Desired behaviors — "a timeout should leave error_kind=timeout in telemetry," "malformed-shape JSON should abstain with a reason code" — get pinned down as executable tests.

The only difference from ordinary TDD is where the inputs come from. Instead of writing tests from functional requirements, you write tests from incident history — everything else reuses the RED → GREEN discipline as-is.

Build the fault catalog from incident history, not imagination

If you enumerate "which faults to inject" from imagination, you tend to mass-produce tests for failures that never actually happen. I started from 5 past production incidents (silent truncation from context-length overflow, mid-generation cutoff with done_reason=length, deduplication failing to fire, an external API rate limit, and a scraping target turning into a CAPTCHA) and swept for "failures in the same family that existing tests don't cover."

One caveat: the 5 incidents and the fault classes are not 1:1. What I extracted from the history is the shared family — "when the LLM or external I/O returns something unexpected, the pipeline silently degrades" — and from that family I derived the 5 classes that existing tests didn't cover.

Here are the resulting 5 classes.

#	Fault class	State of existing tests
F1	Read-timeout mid-generation	Only `ConnectionError` covered. Mid-stream timeout untested
F2	Embedding API failures — transport-level (429, timeout) and content-invalid success responses (mismatched dimensions, missing rows)	Untested
F3	Syntactically valid JSON that violates the expected schema (wrong top-level type, wrong keys, non-string elements)	Untested
F4	HTTP 429 from the LLM server itself	Only covered on a different external API client. LLM backend side untested
F5	Flapping (alternating / consecutive success-failure sequences)	Only single failure→recovery covered

Before and after:

Metric	Before	After
Tests directly injecting the fault classes	0/5 (per the table above, only partial coverage of adjacent cases)	5/5 (32 deterministic fault injection tests)
Telemetry failure-type discrimination	None (everything is `outcome="error"`)	7 kinds of `error_kind`
Observability of extraction failures	Silently missing	3 reason codes + per-reason aggregate summary
Full suite	—	1832 passed / 1 skipped

The test count of 32 is measured with pytest --collect-only (every number in this article was re-measured at the time of writing).

Implementation — 2 replacement points and a determinism discipline

Pin injection to 2 existing seams

Injecting faults requires a seam (a joint where the test side can swap the implementation without rewriting production code). I used only 2 existing ones and decided to add zero injection hooks to production code. The pipeline under test does not know it is being tested.

The LLM backend Protocol — implement a ChaosBackend on the test side and swap it in
The requests HTTP layer — fake HTTP responses with the responses library

ChaosBackend is driven by a schedule — a list of "which fault fires on which call." The chaos in class and file names is a leftover from the naming at inspiration time. Also, FAULT_VOCABULARY in the code is the minimal unit of injection (primitives), at a different granularity from the F1–F5 fault classes — one class is tested through combinations of several primitives.

```python:tests/chaos.py (excerpt, simplified — see the public skill in Related Links for the full version)

Fault vocabulary is an ordered tuple (unordered would break seed-derived determinism)

FAULT_VOCABULARY = (OK, NONE, EMPTY, EXC_TIMEOUT, EXC_CONNECTION, TRUNCATED, SHAPE_VIOLATION)

@dataclass
class ChaosBackend: # conforms to the LLMBackend Protocol
schedule: List[str] = field(default_factory=list)
calls: List[dict] = field(default_factory=list)

@classmethod
def from_seed(cls, seed: int, n: int, weights=None) -> "ChaosBackend":
    rng = random.Random(seed)
    vocab = list(weights.keys()) if weights else list(FAULT_VOCABULARY)
    wts = list(weights.values()) if weights else None
    return cls(schedule=rng.choices(vocab, weights=wts, k=n))

def generate(self, prompt, system, num_predict, format, *, temperature=1.0, think=False):
    idx = len(self.calls)
    self.calls.append({"prompt": prompt, "num_predict": num_predict})
    fault = self.schedule[idx] if idx < len(self.schedule) else OK
    if fault == NONE: return None
    if fault == EXC_TIMEOUT: raise requests.exceptions.ReadTimeout("chaos")
    if fault == TRUNCATED: return BackendResult(text=self._ok_text(idx), finish_reason="length")
    ...




A schedule can be built as an explicit list or derived from a seed — either way, **you can inspect its contents before the run**. This is not "break things randomly"; it's "breaking under this exact sequence is the spec."

### Run hypothesis under a deterministic profile

Instead of enumerating fault combinations by hand, I use property-based testing (asserting properties that must hold for any input, rather than individual input/output examples). To keep CI from going flaky, though, hypothesis is pinned to deterministic mode.



```python:tests/conftest.py (excerpt)
# database=None only disables the example DB. The constants/unicode caches
# need HYPOTHESIS_STORAGE_DIRECTORY relocated (before hypothesis is imported)
os.environ.setdefault("HYPOTHESIS_STORAGE_DIRECTORY", str(_TEST_HOME / ".hypothesis"))
from hypothesis import settings
settings.register_profile("ci", derandomize=True, max_examples=50, deadline=None, database=None)
settings.load_profile("ci")

derandomize=True generates the same case sequence every run. I verified this by running the full fault injection suite twice in a row and confirming identical output.

The other determinism rule: latency faults are expressed by injecting a ReadTimeout exception, not by real sleeps. On this call path, the observable consequence of a timeout reduces entirely to "catch and handle ReadTimeout," so within that scope exception injection verifies the same thing — and the tests run in 0 seconds. The real-sleep-plus-short-timeout approach goes flaky, so I avoided it. Note that behaviors that only surface in real time cannot be verified this way — e.g. cleanup of partial output mid-stream — so if that's your target, you need separate measures.

Writing RED — assert the desired guard behavior first

Here is the RED for F3 (valid JSON, wrong shape).

```python:tests/test_distill_chaos.py (excerpt)
class TestParsePatternsShapeViolationFuzz:
@given(raw=non_patterns_json())
@example(raw='{"patterns": [123]}') # pins the real str() promotion bug
@example(raw="null") # pins the json.loads("null") is None trap
def test_wrong_shape_abstains_with_no_patterns(self, raw):
patterns, mode = _parse_patterns(raw)
assert mode == "shape_violation"
assert patterns == []




`@given` asserts the property "any shape-violating JSON should cause abstention," and `@example` pins known failure shapes as permanent regression tests. Those 2 `@example`s are exactly silent failures ① and ② in the next section.

## The 3 silent failures the first application surfaced

The moment I wrote the RED tests, 3 holes in the existing implementation were exposed. ① and ② are the kind where the processing result silently degrades; ③ is on the observation-channel side — the failure itself gets recorded, but the type information needed for root-cause analysis is silently lost.

### ① Numbers promoted to strings let schema violations sail through

- **Symptom**: when the LLM returned non-string elements like `{"patterns": [123]}`, the old implementation stringified each element with `str(item)`, so `"123"` could pass as a legitimate pattern
- **Why it's silent**: the JSON is valid, so parsing succeeds, and `str()` promotion raises nothing. No trace of the schema violation is left anywhere
- **Guard**: require `isinstance(item, str)` on every element; on violation, abstain with the `shape_violation` reason code. Regression pinned with `@example(raw='{"patterns": [123]}')`

### ② `json.loads("null")` returns `None`

- **Symptom**: when the response body is JSON `null`, `json.loads` returns `None` without raising. The old implementation used "parse result is `None` = parse failed" as its check, so valid JSON was treated as a parse failure and misrouted into a fallback path (bullet-list text scanning) it should never have entered
- **Why it's silent**: the fallback path legitimately exists as a rescue for non-JSON responses, so a misroute is indistinguishable in the logs from a normal fallback
- **Guard**: introduce an identity sentinel, `_JSON_PARSE_FAILED = object()`, separating "parsing failed" from "`None` was parsed" at the type level

### ③ Telemetry flattens every failure type into `error`

- **Symptom**: whether it was a 429, a timeout, a connection failure, or a bad body, telemetry only recorded `outcome="error"`, making offline failure analysis impossible
- **Why it's silent**: the error itself is recorded, so "something failed" is visible. What's flattened is the **type** — a gap you can't notice until you actually attempt an incident investigation
- **Guard**: add `_classify_request_error` to classify exceptions, and add an `error_kind` field to failure rows only (7 kinds: `timeout` / `connection` / `http_<status>` / `bad_json` / `bad_url` / `request_error` / `backend_exception`). The change is additive — the existing `outcome` value set is untouched — so analysis code over past logs keeps working

What the 3 share: they are **abnormal paths wearing a happy-path face**. Bugs that raise exceptions get caught by ordinary tests, but all 3 of these "run to completion while the result or the record goes quietly wrong" — no amount of extra happy-path tests would have caught them. Only RED tests that inject faults and assert the desired way of failing expose these holes.

<details><summary>2 gotchas (hypothesis cache / circuit breaker × property tests)</summary>

**`.hypothesis/` appears even with `database=None`**

Even with `database=None` in the profile, caches like `.hypothesis/constants/` show up at the repo root. As the [settings reference](https://hypothesis.readthedocs.io/en/latest/reference/api.html) says, `database=None` only suppresses the example DB; the constants cache and friends are written unconditionally under `HYPOTHESIS_STORAGE_DIRECTORY`. The fix is to point that environment variable at a test tempdir **before** importing hypothesis (included in the code sample above).

**The circuit breaker breaks property-test predictions**

I asserted the property "success count = number of OKs in the schedule," and it broke only on schedules with 5 consecutive failure faults. The cause was the circuit breaker in production code (a mechanism that cuts off calls after consecutive failures; in this project it opens after 5 in a row). Once the breaker opens, subsequent OK calls never reach the backend, so the exact-count property can't hold even though the behavior is correct. I handled it in 2 layers: exclude breaker-tripping schedules from the exact-count property via a `trips_circuit()` filter, and keep a weaker property — "no schedule ever crashes" — over all schedules.

</details>
## Wrap-up — turning post-hoc debugging into up-front spec

- Silent failures don't get caught by happy-path tests. Only tests that **inject faults and assert the desired way of failing** catch them
- Build the fault catalog from **production incident history**, not imagination. 5 incidents → 5 untested fault classes → 32 deterministic fault injection tests
- For a single-process local LLM app, no daemon or proxy needed: **hypothesis + responses + a swappable backend** sit right on top of pytest
- The determinism discipline (`derandomize` / seed-derived schedules / no real sleeps) removes the 2 big flakiness sources — randomness and real time — from injection tests (measured: identical output across 2 consecutive runs)
- Land the guard in the **same PR** as the test. The fault schedule is the spec; the guard is the implementation that satisfies it
- Chaos engineering is the inspiration, not the classification. What I borrowed is 2 things — "build the catalog from real-world events" and "assert steady state through an observation channel" — while production experiments and unknown-space exploration are deliberately deferred (nightly randomization and sandbox experiments are the entry points for extending toward chaos)

The pattern from this work (fault vocabulary, ChaosBackend, hypothesis profile, RED templates) is published as a generalized Claude Code skill. If you want to bring this into your own pipeline, that's the entry point.

## Related Links

- [chaos-tdd-fault-injection](https://github.com/shimo4228/chaos-tdd-fault-injection) — the public skill generalizing this article's pattern (called "chaos-TDD" in the repo and its ADRs)
- [ADR-0077: Chaos-TDD Fault Injection](https://github.com/shimo4228/contemplative-agent/blob/main/docs/adr/0077-chaos-tdd-fault-injection.md) — primary source for the design decisions (English; a Japanese version sits in the same directory)
- [Previous article: Why Did My Agent Decide That? 3 Observability Patterns](https://dev.to/shimo4228/why-did-my-agent-decide-that-3-observability-patterns-ami)
- [Building an Autonomous Agent on an M1 Mac, on Purpose](https://zenn.dev/shimo4228/articles/small-llm-by-choice) — hub of the small-LLM series this article belongs to (Japanese)
- [hypothesis documentation](https://hypothesis.readthedocs.io/en/latest/) / [responses](https://github.com/getsentry/responses)
- [Chaos Toolkit](https://chaostoolkit.org/) / [toxiproxy](https://github.com/Shopify/toxiproxy) — chaos tools for distributed systems (not adopted here due to the mismatch in target scale)
- [agent-chaos](https://github.com/deepankarm/agent-chaos) — prior OSS for chaos engineering on AI agents (prior art referenced for fault classification)
- [Author's GitHub](https://github.com/shimo4228) — other repositories and tools

LLM-as-Judge Shouldn't Aggregate Scores: Binary Checks as Evidence, One Holistic Verdict

Tatsuya Shimomoto — Tue, 14 Jul 2026 13:00:09 +0000

What this article gives you: How to design LLM quality evaluation around "binary checks + a named verdict" instead of numeric scores. Includes a judge prompt structure you can reuse by swapping in your own evaluation target.

Having an LLM evaluate deliverables is called LLM-as-judge. Once you introduce it, most people run into the same three walls:

Ask for a 5-point rubric score and the same input oscillates between 3 and 4 across runs
Gate pass/fail on a total-score threshold and one fatal flaw gets diluted by high scores on everything else — and slips through
Read the scores back later and nobody can explain why something got a 3.5

This article introduces a design pattern that avoids all three at once. The core fits in one line:

Collect evidence with binary Yes/No checks, have the judge pick one verdict from named labels as a holistic judgment, and never aggregate anything into a score.

I've been running this pattern for about four and a half months in a Claude Code skill-quality audit (/skill-stocktake) and a knowledge-extraction quality gate (/learn-eval). How I built the audit command — the trial and error that led me to throw out numeric rubrics — is covered in my previous article.

This article is the sequel. It generalizes that design into a reusable LLM-as-judge pattern, including the episode where I tested my own design while it was in production — and rebuilt it.

Assumptions

Audience: anyone who wants to automate quality evaluation of deliverables (documents, code, data) with an LLM
The examples use Claude Code skill evaluation, but the pattern itself is agnostic to evaluation target, model, and tooling
Terminology: verdict = the final output of an evaluation. It takes the form of picking exactly one from named options such as "Keep / Retire". The JSON examples in this article use a verdict field
Terminology: holistic judgment = a single conclusion reached by looking at the whole, without going through per-dimension scoring

The pattern at a glance — three principles

Evaluation target
  → ① Binary checks: decompose into Yes/No questions
  → No answers = evidence
  → ② Holistic judgment: pick one named verdict
  → ③ No aggregation: scoring and averaging are forbidden
  → Downstream code / humans branch on the verdict

Principle	Do	Don't
① Checks are binary	Decompose criteria into questions answerable with Yes/No	"Rate specificity on a 5-point scale"
② The verdict is a named label	Look at the whole and pick one from `Keep / Improve / Retire` etc.	"Total is 12 points, so it passes"
③ No aggregation	Have the No answers listed as the grounds for the verdict	Using the Yes ratio (satisfaction rate) as a quality metric

① Make the checks binary Yes/No

Instead of "how specific is this skill, out of 5?", ask "is there a concrete command example you can run immediately after reading? Yes/No."

Binary questions have two properties that numeric scoring lacks:

Verifiable: "Is there a command example?" can be settled black-and-white by looking at the text. "Specificity: 3 points" cannot be verified at all
Evidence: a No answer points directly at what is missing. It feeds straight into an improvement list

In the skill audit I run in production, every target goes through these four questions:

- [ ] Actionability: are there steps, commands, or worked examples you can run immediately?
- [ ] Scope fit: do the name, trigger, and body agree (not too broad / too narrow)?
- [ ] Uniqueness: does it avoid overlapping with other skills in the same batch?
- [ ] Currency: do the referenced paths, CLI flags, and URLs still exist today?

② Have the verdict picked from named labels, as a holistic judgment

Holistic judgment is what LLMs are good at. As I wrote in the previous article, LLMs are bad at scoring dimensions independently — they get pulled by the overall impression. So it's more honest to design for that from the start: "pick one, based on the overall impression."

The key is to name the verdict labels so they map 1:1 to the next action.

Use case	Verdict options	Corresponding action
Skill audit	`Keep / Improve / Update / Retire / Merge into [X]`	Keep / hand off to the improvement engine / technical refresh / delete / consolidate (with an explicit target)
Knowledge-extraction gate	`Save / Improve then Save / Absorb into [X] / Drop`	Save / fix then save / append to an existing skill / discard

A fixed set of options that downstream processing can branch on directly is more practical than a bare accept / reject (in programming terms, an enum — a mechanism that forbids any value outside the defined set. With free-form text you get drift like "Improve" vs. "improve slightly", and code can't branch on it). Designs like Merge into [X] that force a concrete target name make vague verdicts like "this feels like it overlaps with something" structurally impossible to write.

This separation — "the LLM picks from named options, code executes" — also acts as a firewall against judgment errors. If judging and executing are separate, an LLM's bad call can never corrupt system state directly (I run this as a pattern I call LLM judge + Code enforce).

③ Never aggregate the binary answers

This is the pattern's biggest fork in the road. Once you have binary checks, you'll be tempted to say "6 of 8 questions are Yes, that's 75%, above the 70% threshold, so it passes." Don't.

The reason is simple: a satisfaction rate dilutes a single fatal No. If one No says "the referenced file doesn't exist," the skill is broken even if the other seven answers are Yes. Averages and ratios convert that dominant No into a 12.5% deduction.

Do these two things instead:

Require the No answers to be listed as the grounds for the verdict (hidden Nos breed verdict drift)
If there is even one dominant No (a nonexistent reference, an unsupported claim, etc.), tip the verdict to the non-Keep side even if everything else is Yes

The skill-audit definition file states the principle in one sentence:

Evaluation is holistic judgment, not a numeric rubric — binary answers are evidence feeding the verdict, never aggregated into a score (a satisfaction ratio changes no decision and dilutes a single dominant No).

The verdict pressure-test — attack the verdict before you commit it

Binary checks plus holistic judgment already work, but one more quality device raises the verdict's reliability: before finalizing a provisional verdict, have the LLM itself generate and answer questions that try to refute it.

Procedure (applied only to non-Keep candidates):
1. Generate 1–3 Yes/No questions that would refute the provisional verdict
   e.g. if the provisional verdict is "Update (a referenced CLI flag is deprecated)"
   → "Doesn't that flag still exist in the current version's --help? Verify empirically"
2. Answer each question with one line of evidence (file read, path check, web search result)
3. Refutation succeeds (e.g. the flag still exists) → move the verdict back toward Keep
4. Defect confirmed (e.g. the flag was indeed removed) → the No items from the binary
   checks become the improvement list as-is

Two disciplines govern question design:

Atomicity: each question tests exactly one verifiable claim
Refutation-oriented: frame questions to seek refutation of the provisional verdict, not confirmation of it

One more discipline matters for re-judging after fixes: re-judge exactly once, with the same question set. If you rewrite the questions, you can no longer tell whether the fix worked or the bar just moved.

A copy-paste judge prompt structure

Here is the whole pattern in a form you can reuse by swapping in your own evaluation target.

You are a quality evaluator of {kind of evaluation target}. Evaluate using the following steps.

## Step 1: Binary checks (all questions mandatory)
Answer each question with Yes/No plus one line of evidence.
- Q1: {verifiable question, e.g. is there a runnable code example?}
- Q2: {verifiable question}
- Q3: {verifiable question}
- Q4: {verifiable question}

## Step 2: Target-specific refutation questions (only if the provisional verdict leans non-pass)
Pick one provisional verdict, generate 1–3 Yes/No questions that would refute it,
and answer each with one line of evidence.

## Step 3: Verdict
Choose exactly one of the following. Do not output scores or points.
- {verdict_label_1}: {meaning and next action}
- {verdict_label_2}: {meaning and next action}
- {verdict_label_3}: {meaning and next action}

In the grounds for the verdict, list every question answered No.
If there is even one dominant No ({domain-specific fatal condition}),
choose {non-pass verdict label} even if everything else is Yes.

## Output format
Follow the JSON schema below and output **JSON only**
(no explanatory text or Markdown before or after).

Receive the output as JSON and downstream code can process it directly.

{
  "verdict": "Improve",
  "evidence": [
    { "question": "Is there a runnable code example?", "answer": "No", "detail": "Steps are bullet points only; not a single command example" },
    { "question": "Do the referenced paths exist?", "answer": "Yes", "detail": "All 3 paths confirmed with ls" }
  ],
  "pressure_test": [
    { "question": "Does the refutation hold that the bullet-point steps alone are reproducible?", "answer": "No", "detail": "The argument spec in step 3 is ambiguous; not reproducible" }
  ],
  "reason": "The Actionability No is dominant. Adding a worked example to step 3 would make this Keep-worthy"
}

Since the verdict field only takes fixed options, the receiving code just branches with a switch statement. The No items in evidence can be handed to the next stage (a human or another LLM) as an improvement task list as-is.

The full design pattern, including this template, is published as a Claude Code Agent Skill at github.com/shimo4228/llm-as-judge. Copy skills/llm-as-judge into ~/.claude/skills/ and it fires automatically whenever you're designing a judge.

Same principles, different operation at different scales

Running this pattern at two scales taught me that the principles are shared, but the application conditions change with scale.

	Knowledge-extraction gate (learn-eval)	Skill audit (skill-stocktake)
Evaluation target	1 draft extracted from a session	The entire library (73 skills)
Refutation question generation	Unconditional (3–5 questions every time)	Non-Keep candidates only (1–3 questions)
Reason	N=1, so the cost is negligible	Generating for all 73 items wastes most of the shots

With a single evaluation target, go all-in — it costs nothing. At tens of items, reserve the expensive device (refutation questions) for the suspicious candidates only.

Does single-context evaluation lose accuracy? — testing my own design

Here begins the second half of this article. The skill audit above originally used a design of "load every skill into one context and evaluate." The logic: a long-context model can survey everything at once, so cross-skill duplication should also be visible.

Then an audit run under this design returned Keep for every single item. Not one defect, not one duplicate pair detected. Is that "the library is healthy" or "the evaluation is spinning idle"? To find out, I re-audited the same library two ways: splitting it into small batches of about 10 items, each close-read by an independent agent, and a probe dedicated solely to duplicate detection (the experiment design and caveats about the numbers are in the collapsible section at the end of this chapter).

Results — some dimensions degrade, others don't

What was tested	Result
Per-item defect detection (small-batch close reading)	12 of 73 items non-Keep (16%). The single context had missed all of them
Duplicate detection (dedicated probe)	True duplicates: 0 (all 17 candidate clusters were adjacent skills with documented role separation)

Half of the 12 missed items were mechanically detectable defects: 404 links, references to deleted files, deprecated CLI flags. The kicker: two files that no longer existed on disk had been given Keep verdicts.

Zero duplicates, on the other hand, turned out to be reality, not dilution (attention thinning as the context fills up). The single-context rationale — "you can see duplication because you survey everything at once" — had been fully replaced by a lightweight description scan plus targeted close reading.

The biggest lesson isn't context width — it's verification ownership

Tracing the mechanism of the misses revealed a cause deeper than context width itself. The audit definition contained a conditional verification instruction: "if a reference looks stale, check it." That "if it looks..." condition is itself a judgment that depends on the LLM's attention. When the context fills up and attention thins, conditional triggers default to "don't check."

The previous article's conclusion was "mechanical work to scripts, quality judgment to AI." This experiment redrew that boundary one level deeper: inside quality judgment there are deterministic parts hiding, and they belong in code. "Does this file exist?" is not a judgment — it's ls. The deterministic checks that had snuck into the judgment phase should have been moved to the code side as an unconditional, mandatory pre-pass. That is what "verification ownership" in the heading means: moving the execution owner of deterministic checks from the LLM's attention to code.

The revised design (v3.0) assigns each property to the environment that should check it:

Phase 0: Deterministic pre-pass (code-owned, unconditional)
         Reference-existence checks, ledger consistency.
         Anything that doesn't exist never reaches judgment.
Phase 1: Inventory
         (the layer offloaded to scripts in the previous article; unchanged)
Phase 2: Per-item scrutiny (small batches of ~10, parallel)
         Binary checks + verdict pressure-test + holistic judgment
Phase 3: Dedicated duplication probe (1 agent, all items)
         Lightweight description scan → close-read candidate clusters only

I reviewed the 12 detections one by one and fixed them the same day. For one of them, testing the proposed fix before applying it showed the command didn't actually work, so I swapped in a different form. Verify not just the verdict but the fix itself before committing it — the same pattern surfacing outside the evaluation loop.

Experiment design and notes on the numbers (for the curious)

Baseline: the audit result from the all-in-one-context run. On the ledger it was 84 entries, all Keep. The actual files on disk numbered 73 — the gap was a ledger bug: the same skill double-counted under two different keys, plus entries for files that no longer existed. That the ledger was this dirty had gone unnoticed by everyone at that point. The follow-up experiments target the 73 real files
Treatment A (small-batch close reading): split the 73 items into 7 batches of 10–11, each close-read by an independent subagent using the same criteria (binary checks → holistic judgment). Reference existence had to be verified with ls etc., stated explicitly
Treatment B (dedicated duplication probe): one agent reads all 73 descriptions, enumerates candidate clusters → close-reads the bodies side by side to decide
Bias control: the agents were not told that the baseline had returned all-Keep
An honest note on confounding: Treatment A's prompt explicitly required "verify reference existence with ls", so part of the difference from baseline is attributable to "forced verification" rather than "context width." However, detections that required semantic judgment — description/body mismatches, overly abstract writing, scope drift — cannot be explained by forced verification, and I read those as the effect of attention density: 10 items per batch versus 73

Why no scores — the boundary the proposing paper drew itself

The "decompose evaluation criteria into binary questions" approach has a research lineage: CheckEval (arXiv:2403.18771, 2024), TICK (arXiv:2410.03608, 2024), and the later BinEval, "Ask, Don't Judge" (arXiv:2606.27226, 2026). My setup had the "binary checks + holistic judgment" skeleton first; I learned of this lineage afterward and folded it into each skill's References. BinEval proposes decomposing into atomic Yes/No questions and wiring failed questions directly into improvement feedback — the dynamic generation of refutation questions and the "No answer = improvement list" wiring are the parts I ported from it after it came out.

What's interesting is BinEval's own limitations section. The paper self-reports two reservations. First: binary decomposition is reliable only up to concretely checkable criteria, like "is there a code example?" Subjective quality like "is this well written?" is something humans judge by looking at the whole rather than stacking individual checks — it cannot be fully decomposed into a bundle of Yes/No questions (less reliable). Second: "it assumes the fraction of satisfied questions corresponds roughly linearly to quality, and this does not always hold." In other words, using the satisfaction rate of decomposed questions as a score comes with a caveat from the proposing paper itself.

Flip that limitations report around and it is exactly a division-of-labor boundary. Concretely checkable criteria can be decomposed into Yes/No — so let the checks gather evidence. The final value judgment belongs to the subjective "is this good writing?" layer, which reduces neither to a bundle of questions nor to a satisfaction rate — so that layer stays a holistic judgment. The three principles in this article are that self-reported boundary, transcribed directly into a design.

Summary

Design LLM-as-judge evaluation around three principles: "binary checks = evidence," "verdict = pick one named label holistically," "no aggregation"
Make verdict labels a fixed set mapping 1:1 to next actions, and require every No answer to be listed as grounds for the verdict
A verdict pressure-test before finalizing raises reliability. Re-judge exactly once, with the same question set
In my experiment, per-item scrutiny diluted as the context filled up (16% missed). Don't put deterministically verifiable checks on the LLM's attention — move them to unconditional, code-owned execution
The reason not to turn "satisfaction rate" into a score comes from the limitations self-reported by the binary-decomposition paper itself (BinEval): decomposition works up to concrete criteria; the value-judgment layer stays holistic

The previous article's division of labor — "mechanical work to scripts, quality judgment to AI" — got one level deeper this time. Inside judgment, too, there are parts that belong in code. When you design an LLM-as-judge, start by asking, before anything reaches the judge: "is this really a judgment, or is it ls?"

Building an Autonomous Agent on an M1 Mac, by Choice

Tatsuya Shimomoto — Mon, 13 Jul 2026 13:00:06 +0000

For about 3 months I've been running an autonomous agent — one that thinks up and writes its own social media posts and comments — unattended, 4 sessions daily, on a 16GB M1 Mac with small models in the 9B / E4B class. I'm about to publish what that operation taught me about hardening, as a series of 4 technical articles.

Before that, there's one thing I want to write down first: why small models.

I've been to the purchase page for a Mac Studio or a new MacBook Pro more than once or twice. Backing the agent with a large cloud model (Opus or the GPT family) has always been an option in the code. And yet I haven't bought, and I haven't switched. The 16GB M1 is not an economic constraint — it's a constraint I chose.

From the outside, building on small models looks like a cheap compromise. This article explains why it isn't, and states where I stand. It also serves as the hub for the 4-article series.

A model's intelligence hides the roughness of your design

Large models absorb sloppy prompts, ambiguous instructions, and missing guards with sheer intelligence. If all you want is to ship a product, that's a virtue. But if you want to become someone who can build things, it becomes a defect.

Because inside the thing that worked, you can no longer tell where your design ends and the model's intelligence begins. "It worked" and "I built it" are different things. Something you bludgeoned into working with model capability counts as a thing that ran — it doesn't become the ability to build.

Small models have no absorption capacity. So every design flaw comes to the surface.

In my operation, all of the following surfaced:

The context window being silently truncated
Outputs cut off midway
A runaway caused by one missing sampling parameter

In cloud or large-model environments, these rarely bother you. The environment has cushioning built in.

Context windows are in the 200K–1M token class, so truncation itself rarely happens. And when you do exceed the limit, you get an explicit error rather than a silent cut
Sampling defaults are managed on the API side; there is structurally no room to drop a parameter
Even when a slightly-off output slips through, a smart model running the ReAct pattern (where the LLM looks at each result and picks the next move) recovers on the next step and rounds the loop off plausibly

A local 16GB machine has none of this cushioning. You specify the context window yourself, overflow is silently truncated from the head, and the sampling settings are your own property. The same class of problems surfaced in a form that could not be hidden, and the only way to close them was design.

As long as you're running on cushioning, you can't tell whether your design is protecting you or the environment is.

Training under constraints outlasts the environment

There was a period when I did business automation in an environment where I couldn't freely bring in new tools. Security constraints, constraints on approved software, and organizational friction against the very act of "introducing something new." What I could use was software already installed on everyone's PC — Excel and Access — plus an approved RPA tool.

If you can't add tools, the only way forward is to know the combinations of the tools at hand, deeply. Within those limits, I pushed automation as far as the available tools would go. But what that experience really trained wasn't tool knowledge.

The substance of automation is decomposing work that had been running on personal, artisanal tacit knowledge. Up to which step can be written deterministically, and from where does it require holistic semantic judgment grounded in experience?

You examine that boundary as finely as you possibly can, and sort the work into deterministic parts and parts that need semantic judgment. Only the former may be handed to RPA; automation that misjudges the boundary always breaks.

That sorting maps directly onto how I design agents today.

Then	Now
Routine processing writable in Excel / Access / RPA	Scripts and code (deterministic)
Parts needing human semantic judgment	LLM (semantic judgment, probabilistic generation)

I don't throw deterministic work at the LLM¹. I use the LLM only where semantic judgment is required. And since an LLM is not a substitute for human judgment, output verification and final decisions stay on the code-and-human side.

Deciding which side a task belongs to — that judgment is the center of the design. It was trained in a constraint-riddled environment, and it has stayed with me as a principle after the constraints disappeared.

A design that runs on small models doesn't depend on the model

The small-model constraint has one more meaning beyond training: proof of a lower bound.

A design that runs unattended for 3 months on the 9B / E4B class is not plugging design holes with model intelligence. Semantic judgment remains the LLM's job, but every protective mechanism lives on the code side.

Inputs are estimated and guarded before sending
Output truncation is detected and discarded
Failures are structurally recorded by telemetry (the machinery that records the system's behavior)

All of these are model-independent designs. So the design thinking carries over intact if you migrate to a large model. What doesn't carry over are the calibration values and thresholds tied to a specific model (that migration problem is what series #3 covers).

The reverse doesn't hold. A design built on the assumption of a large model's intelligence won't run as-is in a smaller environment. Stacking upward is easy; stepping downward requires a rebuild.

Of course, with a large model the same thing would have run with far less effort. But with something that ran easily, you never find out where design ends and model intelligence begins. Whatever ran on the small model, I can call my own design. That is what I mean when I say this is not a compromise.

What the series covers technically

That was the "why small" part. The techniques that actually made unattended operation work on 16GB are covered in the 4 articles that follow (links will be added as they're published).

Local LLM failures happen quietly — diagnosing silent truncation of inputs, and designing a budget guard that protects before sending
The day I dropped a 1.8x faster backend — adopting MLX because it won the benchmarks, and retreating after 24 hours in production
The day you swap the model — blind A/B testing, and the migration problem of model-tied calibration values
You can't fix what you can't observe — how telemetry, replay, and tests quietly lie

The build log of this agent itself starts at Building an agent from scratch — Claude Code and security-first development (in Japanese). The episode where the fight with small models began in earnest is My agent's memory broke — a day wrestling a 9B model (in Japanese).

Summary

Choosing small models is not an economic compromise — it's a constraint chosen to expose design flaws
What constraints train is not tool knowledge but the ability to sort deterministic parts from parts that need semantic judgment. That sorting carries straight over to the code/LLM division of labor at the center of agent design
The design thinking that sustains unattended operation on small models is model-independent and carries over intact to larger models (only the calibration values need re-measuring)
Whether to stay or move depends on your goal. If you want something running as fast as possible, cloud is the better fit. If you want the ability to build, small models are worth staying with

Why Did My Agent Decide That? 3 Observability Patterns

Tatsuya Shimomoto — Sun, 12 Jul 2026 13:00:08 +0000

What this article covers: three design patterns that make an autonomous AI agent's decision-making "reconstructable after the fact" (replayable audit logs / read-only instruments / shadow-mode validation), and how to install them into your own agent as Agent Skills.

The three walls these patterns address

Once you build an AI agent and start running it, you hit a series of problems that are a different species from feature work.

You can't explain the "why" of a decision after the fact. Someone reports odd behavior, but nothing was kept that lets you reconstruct what the LLM or the heuristic saw and how it decided at that moment
"The output feels like it's drifting" can't justify an intervention. You have the gut feeling but no numbers, so you tweak settings on intuition and can't even tell whether the tweak worked
Wiring a new LLM decision mechanism into production is scary. If the selector or classifier is wrong, the output just degrades quietly — and by the time you notice, the old behavior you'd want as a baseline is gone

All three are, at bottom, a lack of observability — the property that a system's internal state can be reconstructed from the outside. As AI agents move into production, interest in telemetry (the machinery by which a system records and emits data about its own behavior) via standards like OpenTelemetry is growing too. But what traces and metrics tell you by default is "what the request did" — they don't reach the three walls above.

I've published the design patterns that address each wall as Agent Skills, in the same form I use to operate an autonomous agent¹.

https://github.com/shimo4228/agent-observability-patterns

For each pattern, I'll include one concrete thing it solved in my own operation. The shared stance fits in one line.

Observation precedes intervention — ship the logs before the failure. Read the instruments before intervening. Let a new mechanism earn its evidence in shadow before wiring it into production.
"Shadow" is not a coinage of mine. It comes from shadow deployment / dark launch, an established deployment technique in ML and service operations — route production traffic to the new system too, record its outputs, but never serve them — and here it means running a candidate mechanism observe-only, in parallel. In this article I call that shadow mode (details in Pattern 3).

Prerequisites

The patterns themselves are Markdown design documents, so they're readable regardless of language or framework
To load them into an agent as skills, you need an Agent Skills-compatible agent such as Claude Code
The repository is agent-observability-patterns linked above (MIT license, v0.1.0)

The big picture: three patterns layered on the same record

Wall	Pattern	How it solves it
Can't explain the "why" of a decision after the fact	replayable-audit-logs	Ship a replayable audit log in the same change as the feature
A felt drift can't justify an intervention	read-only-instruments	Hand the operator read-only aggregates over accumulated state before intervening
Wiring in a new decision mechanism is scary	shadow-mode-validation	Run the candidate observe-only in parallel; the accumulated record decides adoption

The three are not separate techniques — they stack as layers on the same record. The logs are the base corpus, the instruments are lenses you place on top of it, and shadow mode is the discipline of "building that corpus for a mechanism that isn't yet allowed to act."

graph TD
    S[Shadow-mode candidate mechanism<br/>records what it would have chosen] --> A
    A[Replayable audit logs<br/>event-level corpus] --> B[Read-only instruments<br/>aggregate readings over the corpus]
    B --> O[Operator decides<br/>interventions and adoption]

Let's take the patterns one at a time.

Pattern 1: Ship the logs before the failure

The first wall is what audit logs are for.

The audit log (audit trail) itself is an established term from security and compliance: an append-only record of "who did what, when" — AWS CloudTrail and database audit logs are the canonical examples. This pattern applies that idea to an agent's decisions, and — in the lineage of event sourcing (record what happened as an immutable append-only sequence, and do all later analysis as reads over that sequence) — additionally demands that the record be replayable.

	Ordinary logs	Audit logs
Reader	Humans (during debugging)	Machines (replaying decisions later)
Format	Free-form messages	One decision = one record, machine-readable (JSONL)
What's kept	A description of what happened	The exact input the decision saw + the decision's outcome
When added	After an investigation starts	In the same change that adds the feature

The last row is the principle. Any feature with external I/O, LLM calls, or heuristic decisions ships its audit log in the same change that adds the feature. The corpus (the accumulated record) must exist before the failure it will one day explain.

By "heuristic decision" I mean a branch that is decided by rules in code but whose correct answer isn't self-evident — "if the regex parse fails, fall back to the LLM," "adopt only the items whose score clears the threshold." These branches get asked "why did that happen?" after the fact, just like external I/O and LLM calls do.

What this solved in my operation was repairing a broken parser. A feature that submits generated answers to an external verification system had been logging every attempt as an ordinary side effect. Weeks later, when the parser needed repair, I could replay the hundreds of real-traffic records left in the log offline, and validate the fix against a hard gate: "zero errors against known correct and incorrect answers." Fixing it against real data rather than synthetic test cases was only possible because the log existed before the failure. A log hastily added after the investigation starts can't do this.

Record design: not "readable" — "replayable"

The design goal of the log is not "readable later" but "replayable offline."

"Replay" is not a metaphor. Because the exact input the decision saw is in the log, you can feed that input back through today's code and diff the result against the recorded outcome. A merely readable log tells you "the parse failed," but without the input you can't run this re-verification.

"Offline" means the re-run needs no external system, no LLM, and no network. Only the deterministic layers — parsers, rules — get re-executed (the LLM's response is preserved in the record as the raw output from that time), so you can run it locally as many times as you want. That makes replay fast, free, and reproducible — good enough to sit as a pass/fail gate in CI.

In my operation, this replay exists as a single dedicated script. It imports the production parser function as-is, re-applies it to every unique input in the audit log, checks the results against "records the external system accepted = known correct, records it rejected = known incorrect," and returns a zero-errors pass/fail via exit code — pure code. Run it after every parser fix and you have machine-checked proof that "no previously-correct case was broken."

The record checklist that makes this possible has five items.

Item	Point
Raw-input recoverability	The exact input the decision saw. Store untrusted text (API responses, user prose) as base64 + sha256, never as raw text
Decision path	Which branch, which layer handled it (e.g. `code_parse / llm_extract / llm_reason / none`)
Reason code	Abstentions, fallbacks, and failures get machine-aggregatable category codes (null when the decision succeeded normally). A silent fallback is a defect
Outcome	What happened downstream (accepted / rejected / error) + a sanitized error message
Timestamp + stable key	ISO timestamp and a hash of the input, so the same input can be matched across retries

A single record looks like this (append-only JSONL; input_sha256 is abbreviated here for print, but in practice it holds the full 64-hex-digit value).

{"ts": "2026-07-11T02:14:07Z", "input_b64": "PGV4dGVybmFsPuKApg==", "input_sha256": "95cbc571…cba8c", "truncated": false, "decision_path": "llm_extract", "reason_code": null, "outcome": "rejected", "error": null}

Base64-encoding untrusted text closes an accidental prompt-injection path (a string in a log acting as an instruction to a downstream LLM): the moment someone opens the log, an attack string would otherwise hit eyes and downstream processing as raw text. But base64 is encoding, not sanitization. The decoded content must be treated as untrusted input all over again, and since an LLM can decode base64 by itself, an encoded log does not become "safe data" you may feed to an LLM.

This may look like overkill. But the agent in question operates on Moltbook, a social network where AI agents post to each other, and most of the text its decisions read is "input written by other agents and the humans behind them — anything could be planted in it." In an environment where prompt injection must be assumed, discipline at the moment of logging becomes part of the defense line itself.

The code-review question

The pattern compresses into a single review question.

When this feature misbehaves, which log explains the "why"? Can that behavior be replayed offline from the log?

If there's no answer, you go back to design within the same change. "We'll add logging later" violates the very principle of having the corpus before the failure.

Real logs that survive this question include: a verification audit log recording each solver attempt's decision, a drift log tracking changes in an API's request/response shapes, an approval gate's accept/reject records, and caller-tagged LLM telemetry (measured failure rates per calling feature and per reason — the kind of evidence that can justify a prompt change).

Pattern 2: Instruments are for reading, not for wiring into behavior

The second wall is what instruments are for. "Instrument" here in the aircraft-cockpit sense: a gauge you read, not a lever you pull. Instruments are read-only aggregates over the agent's accumulated state (corpus, memory, logs) — distributions, composition ratios, cluster structure — so the operator can choose interventions from data instead of intuition.

What this solved in my operation was quantifying the feeling that "the agent's memory keeps accumulating the same story." The agent from Pattern 1 distills observations from its Moltbook activity (posts, comments, replies) into patterns and keeps appending them to memory (a text corpus). When the patterns converge on the same voice, the downstream generation that reads them converges too — the echo-chamber shape. So I added a module reporting "supply per consumed category" and "the distribution of cosine similarity across the whole pool," making rebalance-or-not a decision made by reading, not by feel.

When to build one — and when to remove one

There is exactly one criterion for building: build the instrument only when its reading would change a specific, nameable action. Instruments built because "it might be nice to have" get read by no one.

The interesting part is that the same criterion drives removal. An instrument whose reading is constant under the current pipeline changes no action, so it comes off the panel. Sunk cost — the fact that you built it — is not a reason to keep a gauge on the instrument panel.

This too actually happened in my operation. I shipped an instrument reading the composition of the agent's memory supply (self-reflection-derived vs. external-reply-derived), and the reading came back "100% self-derived" — a constant, every time. The instrument wasn't broken; the pipeline's structure simply almost never generated externally-derived records. With no decision consuming the reading, the answer is deletion, not repair. I removed it the same day it shipped, and preserved the one fact it taught (that the memory supply was entirely inward-facing) as prose in the design record.

The invariant that matters most: never wire readings into behavior

Instrument readings go to the operator; they do not flow directly into gates, ranking, retrieval, or promotion decisions. An operator reading a gauge and re-deciding a threshold is the intended loop. The moment code starts consuming the number at runtime, it's no longer an instrument — it's an intervention, and outside this pattern's scope.

If your readings are embedding-based, calibrate with three anchors, because an unanchored number means nothing (the values below are measurements from one operation — re-measure in your own environment).

Anchor	How to measure	Example
Floor (what even unrelated text scores)	Cosine similarity between deliberately unrelated text and the corpus	~0.33–0.46
Corpus mean	Pairwise mean across the whole pool	~0.55
Ceiling band (the best you see in practice)	Best match within a consumed category	~0.68–0.77

In this example, the echo signal was "the corpus mean sitting 0.1–0.2 above the floor." But a single-domain corpus naturally sits above the floor simply because its topics are similar, so that gap alone doesn't decide anything. Read it against your corpus's history and qualitative samples. And whenever the geometry changes — swapping the embedding model, rewriting the seeds — re-measure the scale.

Misreading traps (these bite after you've built the instrument)

A flat reading does not mean "nothing changed." The pairwise mean can hold steady while the corpus visibly changes character (the change was orthogonal to the instrument's axis). Corroborate a flat reading with qualitative samples before concluding "no effect"
Scores of gate survivors are not a distribution. You cannot analyze survivor-only data as if it were the population's distribution
A coarse scorer's threshold can only sit at boundaries where values actually exist. With a scorer that emits a handful of discrete values, drawing a threshold between them is fiction

Pattern 3: A new decision mechanism earns its evidence in shadow before it gets wired in

The third wall is what shadow mode is for (defined at the top; the lineage is laid out in Microsoft's Engineering Playbook). The fear is really irreversibility: you can't know without wiring it in, and once wired in, you can't go back.

Use it when the unknown is decision quality, not code correctness. Unit tests can prove parsing and wiring are correct; they cannot prove "the small local model picks the right skill." When the failure mode is "plausibly wrong," the answer lives only in real traffic.

The other trigger is when the wiring is a one-way door. Connecting the mechanism changes what the model reads and emits, so if the mechanism is bad, output degrades quietly. Delete the old path and you've also deleted the baseline you'd need to notice the degradation. Shadow-first keeps the baseline alive while the evidence accumulates.

The real case from my operation: narrowing skill injection

The autonomous agent I run (the one from Pattern 1 — it posts, comments, and replies on Moltbook automatically) injects its 19 learned skill documents — about 20k tokens — in full into the system prompt on every generation, filling about 60% of the context window with skills. I wanted a small local LLM to select "only the skills that apply to the current situation." But the small model's selection accuracy was an unknown, and wiring it in is a one-way door.

So I mounted the selection mechanism in shadow mode. Injection stays full-text as before; only the "what it would have chosen" records accumulate, and adoption is decided after 2–4 weeks of hallucination-rate and reduction numbers.

In review of this shadow implementation, a model from a different family caught the following chain (two reviewers from the same family had missed it):

The shadow selection call goes through the same LLM-call infrastructure as production generation. That infrastructure has one circuit breaker (a protection mechanism that halts further calls after consecutive failures) — shared with production
If shadow calls fail repeatedly, the shared breaker's failure counter climbs and the circuit opens
The production post-generation that runs right after sees "circuit open" and gets skipped

Shipped as-is, the machinery installed to observe would have silently halted the behavior it was observing. And the log would have said only "generation skipped." The fix was one layer of isolation: shadow failures don't count toward the shared breaker. The lesson is generalized into element 2 of the six below.

Six design elements (each one paid for with a real production bruise)

#	Element	Point
1	Observe-only entry point, enforced by types	The shadow hook returns nothing (`-> None`). Callers can't consume the selection result even by accident
2	Isolation from shared failure machinery	The chain above. Audit every piece of shared state the shadow path touches: circuit breakers, rate limits, caches, retry counters
3	Validate probabilistic output before recording	Check answers against the ground set (catalog names); keep out-of-set answers away from downstream but do record them. Hallucination rate is first-class data for the adoption decision
4	Bake "what it meant at the time" into the record	Metrics computed against mutable state (total token counts, etc.) get written at record time. Recompute at report time and you can no longer replay "what the decision looked like then"
5	Kill switch = absence of config	No shadow config written means the whole path is off (no LLM calls, no files). No feature flags to maintain
6	Reserve exit criteria at launch; fill in numbers later	The metrics the adoption decision will read (hallucination rate, fail-open rate = fraction where the answer didn't parse and passed through untouched, etc.) and the observation window (2–4 weeks) go into the decision record at launch. But pass/fail numbers are not set before the data exists

The sixth may feel counterintuitive, but a threshold invented before seeing data is a guess wearing the clothes of a number. One clean run proves the wiring is correct, not that the mechanism is good. In fact, during pre-publication review of these skills, a different-family model (the review setup is described in this article) proposed "pre-register the pass/fail thresholds" — and since that inverts this design decision, I documented the rationale and rejected it. When a plausible general principle collides with a specific judgment you paid for in bruises, take the latter.

How to install

Clone the repository and copy the skill directories into your agent's skills location.

git clone https://github.com/shimo4228/agent-observability-patterns.git
cd agent-observability-patterns

# For Claude Code
cp -r skills/replayable-audit-logs ~/.claude/skills/replayable-audit-logs
cp -r skills/read-only-instruments ~/.claude/skills/read-only-instruments
cp -r skills/shadow-mode-validation ~/.claude/skills/shadow-mode-validation

Once installed, the agent will consult each pattern in situations like these:

Adding or reviewing a feature with external I/O, LLM calls, or heuristic decisions (→ audit logs)
Turning an observation like "I think an echo is forming" into numbers before intervening (→ instruments)
About to wire an unvalidated LLM decision — a selector, classifier, or gate — close to production behavior (→ shadow mode)

Summary

Observability for autonomous agents compresses into one line: observation precedes intervention. Logs before the failure, instruments before the intervention, shadow evidence before the wiring
The three patterns are layers on the same record: logs are the corpus, instruments are the lens, and shadow mode is corpus-building for a mechanism not yet allowed to act
The observing machinery itself can introduce failures that distort what it observes. Auditing the shared state a shadow path touches, and the discipline of never wiring readings into behavior, are the insurance against that

Claude Code from iPhone: Plugging 3 Holes in Remote Control

Tatsuya Shimomoto — Wed, 08 Jul 2026 13:00:07 +0000

What this article covers: the three holes you are likely to hit when you make the official Claude iPhone app (Remote Control) your main way of driving Claude Code from outside — and how to plug each one.

A while back, I wrote about building an iPhone setup for Claude Code with Termius + Tailscale + tmux. That approach connects straight to a black SSH screen.

Since then, Claude's official mobile app gained Remote Control — a feature that lets you operate the Claude Code instance running on your Mac, remotely, as-is (available since Claude Code v2.1.51, added a few months ago). The UI is well done, and it is now my main setup. Approval prompts and progress are dramatically easier to read than on an SSH terminal.

But once the official app becomes your main interface, you run into three holes. And none of them can be plugged from an iPhone alone while you are out.

Hole	Symptom	Fix
#1 Can't start a new session	The app only connects to existing sessions	Spawn one with a tmux one-liner
#2 Claude auth expires	The login browser opens on the Mac's side	Hop into the screen via VNC just for the auth moment
#3 git push fails	Dies with `Device not configured`	GitHub's device flow + `--insecure-storage`

Prerequisites

This article assumes the following are already in place. For the environment setup itself, see the earlier Termius article.

Remote Control is enabled in the official Claude app, and you can already operate Claude Code on your Mac remotely. Remote Control connects through Claude's service, so the iPhone and the Mac do not need to be on the same network
Your Mac at home stays on (plugged in, lid open, sleep disabled)
tmux is installed on the Mac (if not: brew install tmux)
You push to GitHub via HTTPS + gh auth login (if you push with SSH keys, hole #3 does not apply to you)
For the VNC re-authentication in the second half, you need a separate route from the iPhone to the Mac's Screen Sharing. This is a different channel from Remote Control — set it up over your home LAN or a VPN like Tailscale (see the previous article for setup)

Hole #1: You can't start a new session from mobile

The official Remote Control has one clear limitation: the mobile app itself cannot start a new session. All it can do is list sessions already running on the Mac and connect to them. Any session you want in that list has to be started somewhere, in advance.

This bites in a quiet way. You are driving project A from your iPhone and think "I want a parallel session for project B" — but you cannot trigger that launch from mobile. Your only option is to walk back to the Mac and type claude, which means you are stuck whenever you are out.

Having multiple sessions on one Mac is itself fine. Launch several named claude --remote-control instances and they all show up in the list. The problem is that the launching cannot be triggered from the iPhone.

Why not the first-party alternatives (server mode / Dispatch)?

Anthropic ships two first-party alternatives (official docs).

server mode (claude remote-control) keeps one "receptionist" process running, and it spawns new sessions on demand from the mobile side (up to --capacity N). But the spawned sessions are tied to the directory where the receptionist was started (or worktrees of the same repository), so it does not fit the use case of spinning up sessions across multiple separate repositories.

Dispatch (the Desktop integration), in my hands-on testing (as of July 2026), still felt experimental:

The execution environment appears to follow a different substrate (Cowork — Claude's agent work environment) rather than Claude Code itself, so custom skills built for Claude Code are unavailable
Session startup auto-injects boilerplate instructions like "check git status, recent commits, and the layout, then report" — so every conversation opens with a long status report nobody asked for

For now, the spawn approach in this article is my main method.

Have a live session launch another session

The idea behind the workaround is simple: have the session that is currently alive (the one you can operate from the iPhone) run a Bash command, and let that session launch another Claude Code on the Mac.

When Claude Code starts with --remote-control, it registers itself with Claude's service as a Remote Control session. Registered live sessions appear in the mobile app's list. So if you launch a new claude --remote-control inside tmux, detached (in the background), the list simply gains one more entry. That is the whole trick.

Here is the flow as a diagram:

iPhone (official app / Remote Control)
   │  "Start a new session"
   ▼
Currently live session (on the Mac, RC-connected)
   │  runs the command below via Bash
   ▼
tmux (detached, holds the pty → survives disconnects)
   │  claude --remote-control "AKC"
   ▼
New Claude process → registers itself as RC → "AKC" appears in the app's list

From Remote Control on the iPhone, I ask the Claude I am currently driving: "Start a new Remote Control session in ~/MyAI_Lab/agent-knowledge-cycle." Claude then runs this command on the Mac:

# 新しい Remote Control セッションを tmux で detached 起動する
tmux new-session -d -s cc-AKC \
  "exec $SHELL -lc 'cd ~/MyAI_Lab/agent-knowledge-cycle && exec claude --remote-control \"AKC\"'"

Three points matter here:

-d (detached) — launches in the background. It does not hijack the calling session
Login shell -lc — loads the PATH for node / claude before launching. Without this, it dies instantly with "claude not found"
tmux holds the pty (pseudo-terminal) — so even if SSH or the network drops, or the calling session closes, the launched Claude keeps running

A few seconds later, open the app on the iPhone and "AKC" has appeared in the session list. I never touched the Mac.

Here is the actual session list. Both zenn-content (where this article is being written) and AKC below it were spawned this way. (The previous Termius article also had a "new session" section, but that was a different move: typing /exit → claude inside the single tmux session cc to swap out its contents. You could only hold one at a time — no parallel projects. This spawn approach lines up multiple named sessions side by side.)

Sessions spawned from the phone show up in the list. Filters for "awaiting input" and "awaiting review" work too

Typing this by hand every time is unrealistic, so I made it a launch script

Typing that one-liner by hand every time — on an iPhone keyboard, no less — is not realistic. So I wrapped it in a launch script, spawn.sh, that takes just a project name, and made it callable from a Claude Code skill (Agent Skills — Claude Code's mechanism for scripting a specific task and invoking it as a /command).

From the iPhone, all I say is "Spin up an AKC session." The skill resolves "AKC" → agent-knowledge-cycle and passes it to the script below. The skill is published as spawn-session (you can borrow the skill-definition style along with it).

spawn.sh in full (copy-paste ready) ```bash #!/usr/bin/env bash # spawn.sh — 新しい Claude Code (Remote Control) セッションを tmux で detached 起動する。 # Usage: spawn.sh [display-name] # 動作条件: tmux 3.2+（new-session の -e オプションを使うため） set -euo pipefail PROJECT="${1:?usage: spawn.sh [display-name]}" PROJECT="${PROJECT/#\~/$HOME}" # 先頭 ~ を展開 [[ -d "$PROJECT" ]] || { printf 'spawn.sh: no such directory: %s\n' "$PROJECT" >&2; exit 1; } NAME="${2:-$(basename "$PROJECT")}" # 省略時はディレクトリ名 SESSION="cc-${NAME// /-}-$(date +%H%M%S)" # tmux セッション名（空白は -） # tmux を探す。PATH に無くても Homebrew の既定パス（Apple Silicon / Intel 両方）を拾う TMUX_BIN="" for candidate in "$(command -v tmux 2>/dev/null || true)" /opt/homebrew/bin/tmux /usr/local/bin/tmux; do [[ -n "$candidate" && -x "$candidate" ]] && { TMUX_BIN="$candidate"; break; } done [[ -n "$TMUX_BIN" ]] || { printf 'spawn.sh: tmux not found (install: brew install tmux)\n' >&2; exit 1; } LOGIN_SHELL="${SHELL:-/bin/zsh}" # tmux -e で値を環境変数として渡す（クォート地獄を回避）。 "$TMUX_BIN" new-session -d -s "$SESSION" -e "CCDIR=$PROJECT" -e "CCNAME=$NAME" \ "exec $LOGIN_SHELL -lc 'cd \"\$CCDIR\" && exec claude --remote-control \"\$CCNAME\"'" printf '✅ Remote Control session started: "%s"\n' "$NAME" printf ' tmux: %s\n' "$SESSION" printf ' dir: %s\n' "$PROJECT" # 起動直後に落ちていないかの簡易チェック（あくまで早期検知。成功保証ではない） sleep 1 if "$TMUX_BIN" has-session -t "$SESSION" 2>/dev/null; then printf ' (tmux session live ✓ — アプリ一覧に出たか最終確認してください)\n' else printf ' ⚠️ 起動直後に tmux セッションが消えました（auth 切れ / claude が PATH に無い等）\n' >&2 exit 1 fi ``` (Comments are in Japanese; the logic reads top to bottom: expand `~`, validate the directory, derive the session name, locate tmux even off-PATH via Homebrew's default paths, pass values as environment variables with `tmux -e` to avoid quoting hell, then a 1-second early liveness check.) The alias table ("AKC" → actual directory) is deliberately *not* baked into the script; the calling skill owns it. The script stays a pure launcher that receives an already-resolved directory and name.

Running it returns this:

✅ Remote Control session started: "AKC"
   tmux: cc-AKC-143512
   dir:  /Users/you/MyAI_Lab/agent-knowledge-cycle
   (tmux session live ✓ — アプリ一覧に出たか最終確認してください)

(The last line says: tmux session live ✓ — do a final check that it appeared in the app's list.)

That ✓ is only an early check meaning "the tmux pane is still alive one second later." The real confirmation of success is the app's session list.

When ✓ and the actual outcome disagree (troubleshooting)

✓ appeared but the session is not in the list — if claude is stuck on an auth prompt or a trust confirmation, the pane stays alive (✓) but the session never registers
It died without a ✓ — expired auth is the most likely cause, but a wrong PATH or a wrong flag dies the same way

In either case, the reliable move is to peek into the tmux pane first (tmux attach -t cc-AKC-143512) and see what actually happened.

This workaround depends on "at least one session is alive." Only right after a Mac reboot, when nothing is running, do you need to start the first one by hand at the Mac. Then again, if the Mac is off you can't do anything from the iPhone anyway, so in practice this is a non-issue.

Hole #2: When Claude's auth expires, you can't re-log-in from where you are

Run this setup from mobile for a few days and Claude Code's OAuth occasionally expires. Spawn in that state and it dies right at startup:

   ⚠️  起動直後に tmux セッションが消えました（auth 切れ / claude が PATH に無い等）

(Translation: the tmux session vanished right after launch — expired auth, or claude missing from PATH, etc.)

If it was an auth expiry, the real problem starts here. To re-authenticate, the login URL that Claude Code prints has to be opened in a browser. That browser opens on the home Mac — the machine being remote-controlled. Nothing shows up on the iPhone in your hand. /login is an operation that completes in the CLI on the Mac, and Remote Control does not proxy that browser login to your device.

Hop into the Mac's screen over VNC, just for the auth

So, only when authenticating, I go look at the Mac's actual screen. I use a VNC viewer with a free tier (RealVNC Viewer).

The whole procedure:

Connect to the home Mac with RealVNC Viewer on the iPhone
Complete the Claude Code browser auth that is open on the Mac, with your finger, on the iPhone
Once auth passes, close VNC and return to Remote Control in the official app

VNC streams the entire screen as video, so using it as a daily driver on a small iPhone screen is painful. But for the single moment of "tap the login button and let auth through," it is plenty. Once authenticated, you go back to the far more usable official app.

The conditions for "free to use" (RealVNC licensing)

Whether it is actually free depends on the VNC server on the Mac side. If you install RealVNC Server's Lite plan (free, non-commercial) on the Mac and connect to that, the whole path stays free. If instead you connect to macOS's built-in Screen Sharing, RealVNC's current licensing treats it as a "third-party VNC server," which requires a paid plan on the Viewer side. If you want to stay free, matching RealVNC on the server side is the safe bet.

⚠️ Security note: treat VNC with care. Exposing remote access to your Mac's screen directly on a shared network like café Wi-Fi is dangerous. Restrict the Mac's Screen Sharing (VNC) service so it is reachable only from your home LAN or over a VPN such as Tailscale. The official app's Remote Control connects via Claude's service — a separate path — so keep it mentally separate from the VNC route ("Remote Control works, therefore VNC is safe" does not follow). And always enable a strong password and encryption on VNC.

Keep auth from expiring in the first place

VNC is the recovery tool for when auth has already expired. Better to not let it expire. One thing works:

Log in once while you are physically at the Mac, and keep that session alive in tmux. Same idea as the tmux approach in the previous article. If sessions are launched from a locally logged-in state, the number of times you touch the auth flow at all goes down. Spawn is built on top of this "live session" too — the longer the root session lives, the less often re-authentication comes up.

Hole #3: git push stops working

After a few days of operating remotely, git push stopped going through. The work is done, and the final push dies like this:

fatal: could not read Username for 'https://github.com': Device not configured

gh auth status says "The token in default is invalid." Looks like the token expired. Here is the trap: often the token is alive — it just can't be read.

The cause: SSH/tmux sessions cannot read the Keychain

gh (the GitHub CLI) stores its token in the macOS Keychain. The login Keychain gets unlocked at GUI login. A session over SSH, or a tmux session started by spawn, can neither unlock a locked Keychain nor display the access-permission dialog. The read fails with a "no interaction allowed" error (exit 36), gh cannot read the token and misreports it as "invalid," and git push falls over with no credentials.

The previous article's issue — Claude Code's OAuth expiring on every SSH connection — had the same root cause. In remote operation, every tool that depends on the Keychain hits this wall.

Diagnosing it is copy-paste:

# Keychain の gh トークンが読めるか（ロック中は exit 36 で失敗する）
security find-generic-password -s "gh:github.com" -w >/dev/null; echo "exit=$?"

# SSH 経由のセッションかどうか
echo "SSH_TTY=${SSH_TTY:-unset}"

If you get exit=36 and SSH_TTY is set, it is a Keychain access problem. Conversely, if exit=0 (the token is readable) and gh still says invalid, the token really has expired — and the fix is the same as the next section either way.

The fix: have Claude itself run the device flow

Unlike Claude Code's OAuth (hole #2), GitHub has a device flow — you type a one-time code into a browser on any other device to complete auth. That means no VNC required.

I ask the Claude in the session: "Re-authenticate gh using the device flow and tell me the one-time code." What Claude runs boils down to this:

# バックグラウンドで認証フローを起動し、ワンタイムコードをログから読む
printf '\n' | gh auth login -h github.com --git-protocol https --web --insecure-storage \
  > /tmp/gh-login.log 2>&1 &
sleep 3 && cat /tmp/gh-login.log

! First copy your one-time code: XXXX-XXXX
Open this URL to continue in your web browser: https://github.com/login/device

Then, in the browser on the iPhone in your hand, open github.com/login/device, enter the code, and approve. Once approved, the auth process on the Mac side completes, and git push goes straight through.

I never entered the Mac's screen. Hole #2's Claude OAuth needed VNC because "where the browser opens" is pinned to the Mac; the device flow lets the browser open on any device, as long as the code matches. That difference is what makes this work.

Is `--insecure-storage` actually OK?

The --insecure-storage flag in the command above stores the new token not in the Keychain but in ~/.config/gh/hosts.yml (a plaintext file with 0600 permissions). This is the permanent fix: the Keychain is out of the loop, so from then on git push works from SSH and tmux alike.

Let's be precise about what "insecure" means here. Exactly one thing gets weaker: "any process running with your user privileges can read it."

Threat	Keychain storage	hosts.yml plaintext
Theft / disk read by another user	Protected	Effectively protected by 0600 + FileVault (when powered off)
Malicious process running as you	Some barrier via per-app ACLs	Readable (this is the "insecure" part)
Leaking into backups / dotfiles sync	The Keychain file itself is encrypted	Can ride along in plaintext with `~/.config`
Usable from SSH/tmux sessions	No (our original problem)	Yes

So the protection drops in two scenarios: "after you have already allowed arbitrary code execution on the machine," and "if ~/.config is included in backups or dotfiles sync." Even then, it is the same level as plaintext API keys in ~/.aws/credentials or .env — a standard storage method for CLI tools. If you suspect a leak, you can revoke immediately from GitHub's Applications settings.

More than hardening the storage location, narrowing the token's permission scope is what actually limits the damage of a leak. The narrowing options are folded below.

If you want to narrow the blast radius further (fine-grained PAT)

One option is switching to a fine-grained PAT (an access token you can scope to specific repositories with an expiry). Note that gh recommends passing fine-grained PATs via the GH_TOKEN environment variable rather than registering them with --with-token, since the latter can confuse the behavior of some commands. This is the narrowing move for the "all I need is push" case.

The finished setup

My mobile operation has settled into this shape:

Device / tool	Role	Path
Official Claude app (Remote Control)	Main UI. Progress checks, approvals, giving instructions	Via Claude's service
spawn (tmux one-liner / skill)	Start new sessions without touching the Mac	Bash inside a live session above
RealVNC Viewer	Only when Claude's auth expires: complete the Mac-side browser login from the iPhone	LAN / Tailscale (VNC only)
Device flow (`gh auth login --web`)	Recover git push auth using nothing but the iPhone's browser	Via GitHub's service

The daily loop looks like this:

Operate as usual from the official app on the iPhone
Need another project? "Spin up a session for X" → spawn, then confirm it appeared in the app's list
⚠️ session vanished = expired auth is the prime suspect → pass auth via VNC, return to the official app
git push dies with Device not configured → have Claude run the device flow, approve in the iPhone's browser (make it permanent with --insecure-storage and it stops happening)

To be honest, the iPhone is still, as before, a "remote control." Serious reviews happen back at the Mac. But starting sessions without touching the Mac, recovering from auth expiry from outside, and completing the push — with those three holes plugged, the situations where I am stuck while out have all but disappeared.

Summary

The official app's Remote Control has a good UI and suits mobile-first operation. But it has three holes: "you can't start a new session from mobile," "you can't fix Claude's expired auth from where you are," and "git push auth fails"
The new-session problem: have a live session run tmux new-session -d ... claude --remote-control. The new process registers its own Remote Control session and appears in the app's list (✓ is an early check; the app's list is the real confirmation)
Claude auth expiry: enter the Mac's screen via VNC only for the auth moment, and complete the browser login from the iPhone. Keep the VNC (Screen Sharing) route restricted to LAN/VPN, and treat it as separate from Remote Control
git push auth: the cause is the Keychain (unreadable from SSH/tmux). Have Claude run gh's device flow, approve in the iPhone's browser, and use --insecure-storage to remove the Keychain dependency for good
Environment setup (Tailscale etc.) stays in the earlier Termius article; this one focused on day-to-day operation with the official app at the center

I abandoned the editor, passed through even the black SSH screen, and now I start sessions and finish pushes from the official app in my pocket, without touching the Mac. The remaining holes each got plugged: a tmux one-liner, VNC, and the device flow.

That's it for the tooling. One level deeper — agent design itself:

The previous mobile-setup article:

Running Claude Code from iPhone via SSH + tmux

The skill used in this article:

spawn-session skill (claude-harness) — the spawn approach from this article, packaged as a skill

Research artifacts (with DOIs) live at github.com/shimo4228.

Linters Can't Measure a Non-Code Asset's Value — I Built an LLM Stocktake for It

Tatsuya Shimomoto — Tue, 07 Jul 2026 13:00:05 +0000

What this article covers: how to take stock of the "abandoned configs, dead workflows, and runbooks no one reads" that accumulate in a repository, using a two-tier method — grep for structure, an LLM for value. You can drop it in as a Claude Code skill, or run the same pattern by hand without the skill.

Prerequisites

If you use it as a skill: git, plus write access to ~/.claude/skills/ (a Claude Code environment)
If you run it by hand: a shell with grep, and any LLM (the thing you hand the judgment to)
The target is "files that aren't code" in general (configs, CI, docs, runbooks, etc.). Detecting dead code in the code itself is out of scope.

Develop with an AI agent, and non-code stuff piles up

When you develop alongside an AI agent, files that aren't code quietly accumulate.

PLAN_xxx.md, HANDOFF_xxx.md, PROGRESS_xxx.md — working notes written once and never read again
Linter or formatter config files that got added temporarily and never removed
GitHub Actions workflows that do effectively nothing — either their on: conditions are never satisfied so they never fire, or they do fire but every job is skipped by if: false
Runbooks that are still around long after the process they describe has ended

These aren't "broken." The YAML is valid, the Markdown is well-formed. So the linter says nothing. A linter only looks at "is this structurally correct" — it can't measure "does this file still deserve to exist."

That was exactly my problem. I started looking for "a skill that detects runbooks nobody references anymore, or workflows that stopped functioning," and what I ended up building was a skill called repo-asset-stocktake. This article walks through the core of its design (the parts anyone can reuse), and the story of actually running it against a different repository and finding an "island of orphaned documents."

The core: structure by grep, value by LLM

The decisive realization before building was this: the properties I wanted to check split into two kinds.

Property	Example	What decides it
Structural (determined by the shape of the bytes)	Is the YAML valid? / Is the link target alive?	Code (grep / parse). 100% accurate, instant
Semantic (requires understanding intent)	Is this config still used? / Does this runbook still describe a process that actually exists?	LLM (judgment)

A .textlintrc that nobody runs anymore contradicts no code (structurally, it's healthy). But its value is zero. grep can't bridge that gap.

At the same time, throwing an LLM at every file up front is a waste of money. So I split it: enumeration is code, judgment is the LLM.

Enumerate all non-code assets
        │
        ▼
tier-1 · structural: measure reachability with grep
        │
        ├─ consumer still alive ──▶ Keep (as-is)
        │
        └─ consumer gone (candidate)
                   │
                   ▼
        tier-2 · semantic: LLM judges value
                   │
                   ▼
        Keep / Update / Retire / Merge
                   │
                   ▼
        Confirm one by one, retire reversibly

tier-1 (code): measure "reachability" (explained below) with grep, and narrow down to only the suspicious candidates
tier-2 (LLM): judge only the narrowed candidates by "does this still mean anything," and emit a verdict of Keep / Update / Retire / Merge. No numeric scores.

What "reachability" means

Reachability is "whether something that consumes this asset is still alive." Every non-code asset is alive because something consumes it.

Consumer class	Example asset	Reachability check (what grep does)
Tool invocation	Tool configs like `.textlintrc`	Search for the tool name in `package.json` scripts, pre-commit, Makefile, CI. Zero invocation sites = candidate
CI trigger	`.github/workflows/*.yml`	Parse whether the scripts/actions it references actually exist, and whether the `on:` trigger is reachable
Human navigation	runbook / docs	Search for inbound links from other docs or the README. Zero inbound links = candidate

For example, you can measure "tool invocation" reachability like this:

# Check whether .textlintrc is still "invoked" (not merely installed)
grep -rn "textlint" .github/workflows/ .pre-commit-config.yaml Makefile 2>/dev/null
# For package.json, look only inside scripts (a name in devDependencies is not "invocation")
jq -r '.scripts // {} | values[]' package.json 2>/dev/null | grep textlint
# → If neither prints anything, "zero invocation sites" = a candidate to pass to tier-2

This table and these commands are what I most want you to take away from this article. If you mechanically measure, per consumer class, "whether the consumption link (the wiring to whatever calls the asset) is still intact," you can narrow the candidates a lot before ever handing anything to an LLM.

Why I built it myself (the conclusion after looking for something off the shelf)

The result of checking "surely a tool for this already exists" — twice

At first I assumed "something off the shelf must exist" and went looking. The conclusion splits by layer.

For structural checks, off-the-shelf options are plentiful. MegaLinter and super-linter bundle yamllint, actionlint, markdownlint, and so on. YAML validity has yamllint; dead links have dedicated tools like markdown-link-check or Lychee (both callable from MegaLinter) — note that markdownlint itself does not check for dead links, so you need one of those separately. There's no reason to build any of this yourself.
For semantic "value" review, off-the-shelf options are thin. The closest is the commercial service Dosu, but from what's public, drift detection between docs and code is one of its several features (Dosu itself pitches issue triage and an agent-facing knowledge base more broadly), and in any case it isn't a mechanism for judging "does this config or workflow still hold value."

And the mechanism itself — "an LLM reviews an asset by value and emits Keep/Retire" — was already implemented four times inside my own Claude Code environment (stocktake skills targeting configs, rules, skills, and project docs respectively; I wrote up the design process for the skill-targeting version in a separate article). What was missing wasn't the mechanism — it was the target. The one aimed at "a project repository's non-code assets" was the only one that didn't exist. So instead of inventing from scratch, I copied the skeleton of an existing stocktake skill and swapped only the target.

Leave the structural checks to off-the-shelf linters, and build only the thin "value judgment" layer that's missing — that was the conclusion.

Installing it: as a Claude Code skill

The skill is published as a standalone repository. Drop it in ~/.claude/skills/ and you can call it with /repo-asset-stocktake.

git clone https://github.com/shimo4228/repo-asset-stocktake \
  ~/.claude/skills/repo-asset-stocktake

To run it, you just pass the path of the repository you want to audit.

/repo-asset-stocktake                  # audit the current directory
/repo-asset-stocktake ~/path/to/repo   # audit a different repository

The verdicts are recorded as a ledger in .repo-asset-stocktake.json inside the audited repo, so on subsequent runs you can re-evaluate only the assets that changed. It looks like this:

{
  "evaluated_at": "2026-07-05T05:20:00Z",
  "assets": [
    {
      "path": "docs/plans/NEXT_STEPS_fsrs-migration.md",
      "consumer": "human-navigation",
      "reachability": "0 inbound links / not listed in INDEX",
      "verdict": "Retire",
      "reason": "The feature already shipped, so this 'remaining work' note is now hollow"
    }
  ]
}

Add this ledger file to .gitignore and even a public repo won't leak its audit results.

Running it for real turned up "an island of 33 orphaned files"

After building it, I ran it against another of my own repositories (an iOS app). It has almost no CI workflows or tool configs; it's mostly Markdown docs. "So it won't find anything," I figured. The opposite happened.

What came out wasn't individual junk files — it was a structure.

CLAUDE.md (project entry point)
  ├──▶ RUNBOOK.md
  ├──▶ CONTRIB.md
  └╌╌╌ (no link) ╌╌╌▶ PROJECT_TIMELINE.md   ← the missing bridge
                          │
                          ▼
                     plans/INDEX.md
                          │
                          ▼
                     plans / reports (31 files)  ← the isolated island

Snapshots of PLAN / PROGRESS / HANDOFF / BUG / REVIEW, born during a two-week development burst five months earlier, had piled up in docs/plans/ and docs/reports/ — 31 files. The only path to them was a single chain: PROJECT_TIMELINE.md → INDEX.md. But PROJECT_TIMELINE.md itself wasn't linked from CLAUDE.md (the project's entry point). Add those two entry-point files, and you get 33 files unreachable from the entry point.

The division of labor between tier-1's grep and tier-2's LLM paid off exactly here.

tier-1 (grep): list the structural facts — "PROJECT_TIMELINE.md has 0 inbound links," "INDEX.md links to the plan set." That the bridge is isolated falls straight out of grep.
tier-2 (LLM): interpret those facts as "the only bridge to the whole island is severed — reconnect one entry point and 33 files come back to life," and choose reconnection over deletion.

Detection is structural (tier-1); the meaning is the LLM (tier-2). The reason I split enumeration from judgment worked exactly as intended on this island.

The fix wasn't deletion — it was reconnecting one entry point. Adding a single line in CLAUDE.md linking to PROJECT_TIMELINE.md rescues the history of all 33 files at once. That's a conclusion a lint could never produce, and one that only a value review could.

The moment tier-2 delivered more value than "delete"

The most effective thing this skill did wasn't deletion — it was vetoing a deletion.

tier-1 found two files both named dead-code-analysis.md and flagged them as a "duplicate" candidate. With naive basename dedup, this is where one of them would have been deleted.

But when tier-2 compared their contents, one was an analysis by Python's vulture and the other was a manual Swift analysis — different things. Merging them would lose one of the analyses. The verdict was "cancel the Merge." The semantic layer verifies and overturns the structural signal. That's exactly the point of splitting it into two tiers.

In the same vein, over on the zenn-content side it caught an asset of a different class: .gitignore declared that archive/ should "not be published," yet four files committed before that rule was added were still tracked and continued to be published on GitHub — a gap between the intent of a config and reality. The same skill spans assets of different consumer classes: tool configs, workflows, docs, and VCS settings.

You can run this pattern without the skill

Even without a Claude Code custom skill, the core is reusable. With your own agent (or by hand), just do the following in order:

Enumerate (code): list every non-code asset, and measure reachability per consumer class with grep (the table and commands under "What 'reachability' means" above)
Narrow: take the ones with zero reachability (0 invocation sites / all references dead / 0 inbound links), plus the ones that "are reachable but look suspicious" (stale updates, throwaway docs like PLAN / HANDOFF, notes describing "remaining work" for an already-shipped feature, etc.) — the island in the previous section still had inbound links, yet it was hollow
Judge (LLM): judge only the candidates by "does this still mean anything," and emit Keep / Update / Retire / Merge
Delete reversibly: don't delete a Retire immediately — first move it somewhere recoverable, like renaming to .disabled (soft-delete), and confirm one by one

The biggest pitfall is that "zero reachability" does not equal "dead."

When I built this skill, I ran a review by a different model (Codex) in parallel, and it caught a blind spot my own review (Claude) had missed: when consumption is one step indirect, grep comes up empty.

prettier invoked via lint-staged has no direct invocation site

a remote uses: owner/repo@ref action is alive even with no local copy present

a doc listed in the nav of mkdocs.yml is reachable even with no Markdown inbound links

.claude/skills/*.md has 0 inbound links, but its consumer isn't a human — it's the Claude Code loader

Treat zero reachability not as a "deletion candidate" but as "needs investigation." Mistake the consumer, and you'll kill a live asset. In fact, without this correction I'd have wrongly Retired 9 of the iOS repo's .claude/skills/*.md files.

Summary

Leave the structure of non-code assets (validity, dead links) to off-the-shelf linters. Don't build it yourself.
Layer a thin LLM only over value (does this still deserve to exist). Don't run it on everything — narrow the candidates with grep first.
Measuring reachability per consumer class (tool invocation / CI trigger / human navigation) is the mechanical axis for that narrowing.
Zero reachability is "needs investigation," not "delete." Miss an indirect consumer and you'll kill a live asset.
Always delete reversibly (move to .disabled → confirm one by one).

A lint can only answer "is this file correct." "Is this file still needed" is a different question — one that requires understanding meaning. Hand only that part to an LLM, and assets that had slipped through the linter's net start to become visible.

I Built a Skill for Easy Codex Reviews from Claude Code

Tatsuya Shimomoto — Sun, 05 Jul 2026 04:30:06 +0000

What this article covers: how to add exactly one cross-model review step to Claude Code using the OpenAI Codex CLI, and why it works — not because of "more" review, but because it catches "a different class of blind spot."

Prerequisites

Codex CLI installed and authenticated (verify with codex login or codex doctor)
Working inside a git repository from Claude Code

Usage — a read-only second opinion in one command

codex-review is a read-only skill that thinly wraps codex review (the OpenAI Codex CLI). It shows the current diff to a model from a different lineage than Claude Code and has it review the work.

/codex-review isn't a command that ships with Claude Code out of the box — it's a small wrapper script I placed in my own Claude Code harness.

The skill itself is published as shimo4228/codex-review, so you can reproduce the same setup by copying skills/codex-review/ from that repository into ~/.claude/skills/codex-review/.

All it does is a single shell script that passes only the allow-listed flags straight through to codex review and rejects everything else.

The script itself isn't Claude Code-specific. If you look inside, the string "Claude" only appears in comments — the actual execution logic is just bash + git + the codex CLI.

You can run it straight from a terminal, call it from a different agent CLI, or use it as a plain pre-commit hook with no Claude Code in the loop, and it works the same way. Being "a Claude Code skill" is just a matter of where you put it and how you invoke it.

# current branch vs. auto-detected base branch (PR style)
/codex-review

# staged + unstaged + untracked files (for a pre-commit check)
/codex-review --uncommitted

# against a specific branch or commit
/codex-review --base main
/codex-review --commit <sha>

# prompt-driven (review the whole working tree with no scope flag)
/codex-review "just look at the auth-related changes"

Read-only is enforced at the code level. Internally it only ever calls codex review (which never rewrites code), and any flag not on the allow list is rejected outright. Even if the Codex CLI adds a write-capable flag in the future, it won't run unless it's added to the allow list.

I don't paste Codex's output straight into the parent conversation — I verify it first, then summarize.

Agent: codex-review
Verdict: <CRITICAL | HIGH | MEDIUM | LOW | CLEAN>
Findings (top 3): <only what's been confirmed>
Files touched: <path:line>
Next action: <continue | stop | re-plan>

I treat Codex's findings as "external input that isn't necessarily correct," and keep only the ones I can confirm by reading the code and reproducing the issue. Deciding the verdict is still on me (the Claude side).

Wiring it into the harness as an "automatically-run check"

Calling /codex-review manually every time is already useful, but it really pays off once it becomes something you can't forget to call. Write the condition into your project rules (CLAUDE.md or .claude/rules/*.md) and, once implementation reaches a natural checkpoint, Claude will follow it on its own — without you asking.

That said, this isn't deterministic enforcement like a hook. A rule only conditions Claude's judgment; whether it actually runs is still up to Claude in the end (if you want a 100% guarantee, you need to enforce it with a PostToolUse hook).

Even so, just writing "once you've made a non-trivial diff" into the rule as an explicit condition changes the pattern from "ask each time" to "it runs on its own once the condition is met." The full rule I actually use (with its Chain Matrix and early-stop conditions) is published in claude-harness's planning.md.

<!-- Example: write this into CLAUDE.md or .claude/rules/*.md -->
## Review step (right after implementation, before commit)

When a feat/fix produces a non-trivial diff, run the following in parallel before committing:
- a normal code review (a subagent on your own model)
- codex-review (cross-model, read-only)

If either one returns CRITICAL, stop the commit and report to the user.

Two things matter here:

Make the trigger "a non-trivial diff was written," not "the user asked for it." This structurally prevents forgetting to call it.
Stop early on a CRITICAL finding. If a different model flags something serious, the rule itself gates on stopping the commit right there and deferring to human judgment.

Set up this way, findings aren't "something I happened to notice" — they get caught in the same place every time. This article itself ended up being a live demonstration of that.

codex-review doesn't just review code diffs — in prompt-driven mode it can review prose too. This repository's pre-publish checklist says "articles get reviewed in parallel by editor / fact-checker / codex-review," and this article went through codex-review without me explicitly asking for it either.

And it actually caught something. The paragraph right before this one had originally overclaimed that "writing it into a rule makes it run automatically" — that's the finding, and the current wording already reflects the fix.

Not just blind spots in code — a different model diligently catches overclaiming in prose too.

What it actually found — a different model sees a different blind spot

It's tempting to describe the value of codex-review as "it's good at finding bugs Claude alone would miss," but framed as a frequency claim, that's hard to verify. Here's what actually happened, described plainly.

This is from a review on Contemplative Agent, another research repository of mine, when I added a single monitoring instrument module (ADR-0071, the commit in question). This instrument is observation-only by design: it never touches pipeline behavior and only reads and displays the distribution and aggregates of embedding vectors. I ran codex-review (a different model lineage) and Claude's own regular review (python-reviewer) in parallel against the same diff.

Reviewer	What it found	Kind
codex-review	A crash path in the aggregation logic when embedding vector dimensions are inconsistent	Numeric / data-shape mismatch
codex-review	A filter condition in the aggregation was off by one step from the processing it was supposed to observe	Semantic drift in aggregation logic
codex-review	The dry-run aggregation was counting candidates that should have been dropped by downstream deduplication	Semantic drift in aggregation (scope)
python-reviewer	An unexpected input (e.g., a non-numeric embedding) propagates an exception that stops the host pipeline it was only supposed to be observing	Exception-handling / robustness contract

The 3:1 split is just the result of this one run — the point isn't the count, it's the difference in the level of abstraction of the findings (more on that right after the table).

I confirmed all four by reading the code and reproducing the issue, then fixed them and added regression tests (the primary record is in the commit message). All tests stayed green the whole time these four bugs existed. In other words, "it runs" and "it's correct" are separate axes of checking, and review was covering the axis tests weren't.

What's interesting is finding #1 (codex-review: crash on dimension mismatch) and finding #4 (python-reviewer: host pipeline halted by exception propagation). On the surface, both look like the same "broken embedding row" problem.

They aren't, though — the findings sit at different levels of abstraction.

codex-review — a specific failure mode: the aggregation function crashes when dimensions don't match
python-reviewer — a broader contract that failure mode sits inside: the design principle that "an observation-only instrument must never halt the host pipeline, no matter what abnormal input arrives" was missing entirely

Around the same underlying phenomenon — a "broken row" — the two reviewers pointed at different levels of abstraction, different variants of the same problem. That's what happened here.

I can't claim that running the same model lineage twice would only ever surface one or the other. What I can say is that, in this one run at least, there was no overlap.

Pitfalls / Tips

If you hit one of these

Scope flags and a free-form prompt are mutually exclusive. Passing --uncommitted (or similar) together with a free-form prompt gets rejected with exit code 64.
If you run with no scope argument and HEAD is already on the base branch (e.g., running with no flags while on main), the diff comes out empty, so it automatically falls back to --uncommitted. This auto-fallback does not kick in if you specify a scope explicitly, e.g. --base main.
If the CLI isn't installed, the wrapper itself checks upfront and fails early. If it's not authenticated, that's outside the wrapper's scope — codex review itself returns the error (check with codex login / codex doctor). Either way, just continue with Claude's own review alone.

Summary

codex-review itself is a thin wrapper, but what makes it work is the combination of two things: the design choice to mix in exactly one other model, and the operational choice to write "a non-trivial diff makes you a review target" explicitly into a rule. Add a few lines to your own project rules and you can reproduce the same setup starting today.

DEV Community: Tatsuya Shimomoto

What Humans Should Approve Is Intent, Not the Diff — A Decision Table for Agent Approval Gates

Assumptions

Gates have two axes

The decision table: what you show is decided by the target

Escalation rule: irreversibility overrides the target category

In one line

Applying it: what was actually happening in five places

The same gate had two competing sources of truth

"A human inspects the artifact" was scattered across five places

Reinforcement 1: the control plane and "artifacts that produce the evidence"

So how should you present it

Reinforcement 2: on FAIL, show the detection line

"Not shown" is not "not kept"

Reinforcement 3: don't let the intent summary be free-form

Only one item is required: the Divergence from plan declaration

Cap the number of stops too: one gate per unit of work

Raise the granularity and the bottleneck moves

Porting it into your own conventions: three steps

The limits of "the machine holds it"

Back to the opening question

Related links

Opus 5 Changed How Rules Should Be Written — Audit Yours

Background: the official guidance on rule-writing changed

Premise: the ground truth is the system prompt and tool descriptions

Step 1: capture the current runtime layer from a live session

Step 2: cross-check against your rules and sort into 3 categories

Conflict example 1: a no-plan-mode rule

Conflict example 2: a rule referencing a ghost setting

Redundancy example: a scope-adherence rule

Drift example: a confidence threshold for review

Step 3: don't treat "conflict = bad" — run each item through a decision frame

Pitfall: mechanically applying the official advice misdiagnoses

Results and numbers

Summary: what you can do next

Postscript (2026-07-26)

Related links

claude-security Beta: Cost, Output, Accuracy — 189 Agents, 2 Hours, Measured

Prerequisites

What the tool is, and what happens when you run it

From install to launch

Get the confirmation prompt out of the way

Effort changes how much it searches, not how hard it verifies

A six-stage pipeline runs

What the output looked like

1. It reads configuration and natural-language instruction files

2. It folds the count down instead of inflating it

3. The machine-readable side lets you sort by what you care about

How far you can trust it

Limits the report writes down itself

It executes no code

Anthropic states that it is nondeterministic

It writes nothing

What to do afterward

How to order the fixes

You do not have to take the recommendation as given

Day to day, put it on Scan changes

Summary

Links

herdr, a tmux for AI Agents — Until the Editor Disappeared

Intro — the "where do I put them" problem of parallel agents

Prerequisites

What herdr is — only two real differences from tmux

Installing — one brew command

Where Zed is enough, and where it isn't

Letting the agent operate the layout itself

Does the sidebar show state, or history?

For a while, the answer was "Zed at the desk, herdr everywhere else"

The three jobs left in Zed — from "writing tool" to "sign-off tool"

The Japanese-text rendering wall, and a host swap — to Ghostty

Why herdr gets details like Japanese input right — a solo dev and his bots

The ending — Zed didn't even get to be the sign-off viewer

Closing — the shape of the execution environment becomes the agent's tool

Related links

What Do My AI Agent's Logs Look Like in OpenTelemetry?

Prerequisites

I decided whether to adopt OTel by asking "who reads the telemetry?"

What I borrowed from OTel, and what I passed on

My logs mapped almost 1:1 onto the standard vocabulary

The conversion script has exactly 3 design points

Only one item is required: the `Divergence from plan` declaration

Is `--insecure-storage` actually OK?