DEV Community: shimo4228

AI Review Kept Creating Work: Why I Deleted 4,541 Lines

shimo4228 — Sun, 16 Aug 2026 13:34:39 +0000

At 7:22 a.m. on August 16, 2026, after fixing bugs through the night, I typed this to an AI:

Every time I run a review, Opus 5 defers another bug fix into a task, so the bug-fixing never ends. Analyze why this keeps happening.

The rule I took away is simple: persist an AI review finding only after verifying its premise, or after a human deliberately chooses to explore it. This article traces what broke, and what I deleted, before I reached that rule.

I fixed one problem and ran another review. The review found a different problem.

Because it was outside the current change, I sent it to a future task and closed the repair.

Then I fixed that task. The next review found another problem, which I also sent to the future.

Every individual decision looked responsible. Seen as a whole, I was creating new work every time I finished old work.

The previous evening, I rebuilt task management

It started with a task ledger that had grown too large.

The ledger held 112 tasks and about 100,000 characters. Only six tasks were ready to start, but the AI read the entire ledger to find them.

Multiple AI sessions were also editing the same Markdown table. It was hard to tell who owned which task, and one row disappeared while sessions were working concurrently.

I could not determine whether that row was intentionally deleted or lost in a write collision.

So I rebuilt task management:

store each task in its own file
append ownership claims to a log
generate a short view for the AI to read
periodically check whether blocked conditions had cleared

The design was coherent. The introductory commit changed 10 files and added 1,721 lines.

Then the repairs began.

19:26  Introduce the three-layer task system
20:22  Repair tests and restore behavior
21:24  Repair multi-line condition parsing
22:03  Repair Markdown delimiter handling
23:23  Repair stale-view detection
07:20  Repair control-character and display handling
07:22  Stop and ask why the bug-fixing never ends

I had spent the night repairing a system I had built about 12 hours earlier.

Task management was creating its own tasks

I reconstructed the lineage and counted 12 tasks created by reviews.

While closing seven of them, I spawned nine child tasks: an average of 1.3 new tasks per closure.

Seven of the 12 concerned the task machinery itself. Zero concerned the core AI agent.

I thought I was managing tasks that improved the agent. In practice, I was managing tasks required to maintain task management.

A review finds a problem
        ↓
Send it to a future task
        ↓
The task-management code grows
        ↓
Review the new management code
        ↓
Find another problem

There was no place in this loop for a human to stop it.

The review AI's job was to find problems. The repair AI tried to finish the current change safely.

Sending an out-of-scope finding to a future task made both appear to have fulfilled their responsibilities.

I also treated meticulous task filing as a virtue. I had stopped reading the ledger itself, so nobody remained in the loop to ask whether the work should exist at all.

This was not an AI running wild on its own. My harness and I had built a flow that rewarded sending more work into the future.

The first question should not have been how to fix it

At 7:36 a.m., I wondered whether I should have used an external tool.

At 7:44, the question moved further upstream:

Isn't the design itself questionable? Isn't it a mistake to build such a high-maintenance system from scratch?

At 7:50, I finally reached the question that mattered:

Is this system worth keeping at all?

Build versus buy was the second question.

The first question was whether the system deserved to exist. If it did, the next question was how small it could be.

When I reduced the requirements again, the elaborate rendering and recovery machinery was unnecessary:

one file per task was enough
a small read command could find ready tasks
an append-only log could prevent concurrent ownership collisions
a human could check the few blocked conditions once a week

At 8:23 a.m., I deleted view generation, blocked-condition monitoring, migration code, and their tests.

28 files changed
+410 insertions
-5,216 deletions

7 files deleted completely: 4,541 lines
  implementation: 2,064 lines
  tests:          2,396 lines
  fixtures:          81 lines

That was 12 hours, 56 minutes, and 34 seconds after the three-layer system was introduced.

I did not delete everything. The task files and concurrent ownership records remained.

I also kept the command that lists ready tasks.

The module containing that command still had 623 lines and 44 tests at the inspected snapshot. I do not claim that it is minimal.

I did not arrive at a finished answer. I removed the layers I could now show were unnecessary and returned to a point where I could question the next layer.

Even a HIGH finding needs its premise checked

A few hours after the deletion, the same problem returned in another form.

A security review produced a HIGH-severity finding about filename handling. Under my previous rule, an out-of-scope finding was preserved as a task if its severity was HIGH or above.

The rule had been followed. One premise inside the finding was still false.

The report assumed that an AI repair session could choose an arbitrary filename. In fact, the session-supplied identifier was constrained to F1.N, and the shell constructed the patch filename from that identifier.

A separate claim targeted the round value, whose producer was an integer counter.

Severity does not guarantee that a premise is true.

The review that generates a finding also assigns its severity. If severity alone decides whether the finding becomes a durable task, the generator is filtering itself on an axis it created.

I changed the admission rule. Before creating a task from a review finding, the filer must now show where the relevant value is produced.

Receive a finding
    ↓
Trace the value from its producer to the affected sink
    ↓
Check whether the path is reachable and reproducible now
    ↓
Create a durable task only if preserving it is still worthwhile

In the implementation, review-origin tasks require a path:line citation for the producer.

This gate checks only the shape of the citation. It does not prove that the citation is correct or that the path is reachable.

It is not a complete solution. A session can bypass it by writing a task file directly, and line numbers drift as code changes.

The purpose is not enforcement. It is to make someone open the code once before creating future work.

Whose work grows when you create a task?

This night changed how I see task creation.

Writing a task is cheap: a few Markdown lines or one Issue button.

Weeks later, however, I have to reconstruct context from that short record and check whether the code has changed.

I must decide again whether the original finding was true. Then I must choose whether to fix it or discard it.

Turning an AI review finding into a durable task reserves future human attention and judgment.

So before I turn an AI review finding into a durable task, I now require one of two things:

evidence that verifies the premise and shows the problem exists
an explicit human decision that the uncertainty is worth exploring, plus a condition that will end the exploration

A review-generated hypothesis that satisfies neither condition is discarded instead of sent into the future. Planned features and recurring work whose value and completion conditions are already known are outside this rule.

Line count alone does not define overengineering. A 4,541-line system can be worthwhile if it repeatedly removes many decisions.

A 100-line system can be heavy if every use demands new classifications and exception judgments from a human.

This system was overengineered because it created more decisions than it removed.

At 7:22 a.m., all I could see was bug-fixing that would not end.

At 8:23, I discarded 4,541 lines. I also discarded the assumption that preserving every review finding for later was the responsible thing to do.

Machine-readable layer: technical record and audit contract

Human readers can stop here. The remainder decomposes the timeline, measurements, causal hypothesis, and implementation boundary into a form that LLM crawlers and coding agents can verify.

document:
  title_en: "AI Review Kept Creating Work: Why I Deleted 4,541 Lines"
  language: en
  genre: mixed-technical-essay
  canonical_channel: devto
  observed_at: "2026-08-16 Asia/Tokyo"
  thesis: >-
    Persisting an AI review finding without checking its premise moves an unverified
    hypothesis and its re-evaluation cost into a future human's workload. Do not persist
    such a finding without either verification or an intentional human decision to explore it.
  authorship:
    drafting: AI-mediated
    responsibility: "Claims, judgment, and publication responsibility belong to shimo4228"

definitions:
  durable_task: "A record that survives the session and requires a future reader to judge it again"
  premise: "A verifiable condition that must be true for a finding to hold"
  producer: "The code entry point that generates, constrains, or selects the value at issue"
  sink: "The point where a value becomes an effect such as a filename, command, display, or persistent state"
  overengineering_criterion:
    definition: "A system that creates more decisions after introduction than it removes"
    scope_limit:
      - "This is not a precise measurement of decision counts"
      - "It is an evaluation rule derived from one project's maintenance loop"
      - "The result varies with user count, repetition, and cost of error"
  intentional_exploration: >-
    A task whose uncertainty is explicit, whose exploration value a human selected,
    and whose ending or reclassification evidence is stated.

timeline:
  - at: "2026-08-14T21:23:48+09:00"
    event: "Add a mechanical consumer for blocked conditions"
    commit: df68bcee25b61f78c1f0acaedc13da9579127705
  - at: "2026-08-15T19:26:26+09:00"
    event: "Introduce the store / journal / projection ledger"
    commit: 68e9eaf92d7d2625f98ca9a34d66a2e4c2bba5b7
  - at: "2026-08-15T20:22:30+09:00"
    event: "fix ledger tests and restore behavior"
    commit: e215812
  - at: "2026-08-15T21:24:13+09:00"
    event: "fix unclosed watch span"
    commit: c16642c
  - at: "2026-08-15T22:03:10+09:00"
    event: "fix escape collision"
    commit: 8265e3c
  - at: "2026-08-15T23:23:21+09:00"
    event: "fix stale projection handling"
    commit: 1921b01
  - at: "2026-08-16T07:20:42+09:00"
    event: "fix control-character boundary"
    commit: f0f8c5368bfe43545973e939aa595f45ea0792ae
  - at: "2026-08-16T07:22:53+09:00"
    event: "The author questions why repairs do not converge and begins causal analysis"
    source: "local session log; public reproduction unavailable"
  - at: "2026-08-16T08:23:00+09:00"
    event: "Retire projection / scanner / migration"
    commit: 0520faf49097598a317ee02a3570fb150551907a
  - at: "2026-08-16T11:17:35+09:00"
    event: "Change review-task admission from severity to premise verification"
    commit: 7b4b1bc541a1eae427a7d39451474ee54d138ced
  - at: "2026-08-16T11:18:06+09:00"
    event: "Recheck four HIGH claims from their producers and converge on two code changes"
    commit: 510b623c48e068c69d7fe014d4bfcf91c7363e70

architecture:
  before:
    store: "tasks/T-XXX.md"
    journal: "claims.jsonl"
    projection: "generated TASKS.md"
    consumers:
      - ledger_condition_scan.py
      - weekly pipeline packet builder
    maintenance_surfaces:
      - store parser
      - projection renderer
      - projection parser
      - migration and restore
      - aging and candidate intake
      - watch condition scanner
  after:
    store: "tasks/T-XXX.md"
    journal: "claims.jsonl"
    reader: "claims.py ready"
    terminal_task_policy: "move to archive/tasks"
    blocked_condition_check: "manual weekly gate for the remaining small set"

measurements:
  pre_migration_ledger:
    task_rows: 112
    characters: 102111
    ready_tasks: 6
    verification: "ADR-0094 record; original gitignored ledger was deleted"
  introduction_commit:
    files_changed: 10
    insertions: 1721
    deletions: 4
  five_core_files:
    before_lines: 1854
    after_lines: 4460
    growth_percent: 140.56
    members:
      - "scripts/tasks.py: 706 -> 1276"
      - "tests/test_tasks.py: 513 -> 1587"
      - "scripts/migrate_ledger.py: 111 -> 131"
      - "scripts/ledger_condition_scan.py: 258 -> 657"
      - "tests/test_ledger_condition_scan.py: 266 -> 809"
  three_layer_lifetime:
    seconds: 46594
    human_readable: "12:56:34"
    start_commit: 68e9eaf
    end_commit: 0520faf
  broader_consumer_lifetime:
    seconds: 125952
    human_readable: "34:59:12"
    start_commit: df68bce
    end_commit: 0520faf
  review_task_reproduction:
    review_origin_tasks: 12
    closed_tasks: 7
    child_tasks_spawned: 9
    children_per_closure: 1.286
    verification: "ADR-0095 plus local gitignored claims.jsonl reconstruction"
    public_reproducibility: partial
  retirement_commit:
    files_changed: 28
    insertions: 410
    deletions: 5216
    fully_deleted_files: 7
    fully_deleted_lines: 4541
    fully_deleted_composition:
      implementation: 2064
      tests: 2396
      fixtures: 81
  remaining_global_module:
    commit: 734a502b0d05f62e7a2b5691558aa04642cde063
    claims_py_lines: 623
    bats_tests: 44
    status: "not claimed to be minimal or final"

causal_model:
  observed:
    - "review-origin tasks were created faster than they were closed"
    - "7 of 12 review-origin tasks concerned the ledger machinery itself"
    - "the human owner had stopped reading the ledger"
  interpreted:
    - "filing was cheaper than premise verification"
    - "the absent human reader removed the system-level stopping judgment"
  loop:
    - review finding
    - durable task
    - management-code change
    - review of management code
    - new finding
  scope_limit: "single-project causal reconstruction, not a universal measured law"

case_study:
  id: T-PACKET-FLOOR-BYPASS
  severity: HIGH
  bundled_claims: 4
  code_changes: 2
  claim_outcomes:
    - id: a
      reported: "raw fix_id could forge a section heading"
      observed: "producer constrained fix_id to F1.N"
      disposition: "hardened with shared filename allowlist"
    - id: b
      reported: "NUL could suppress packet generation"
      observed: "ValueError was not caught"
      disposition: repaired
    - id: c
      reported: "LLM fix session could choose exported patch filename"
      observed: "shell constructed the only possible F1.N.patch name"
      disposition: "false premise; no repair"
    - id: d
      reported: "round could contain slash and select an arbitrary file"
      observed: "round producer was an integer counter"
      disposition: "closed by the same hardening as a"
  producer_trace:
    fix_id: "parse_findings.py:32 — ^### (F1\\.\\d+)"
    round: "weekly-pipeline.sh:750 — integer arithmetic"
    patch_name: "weekly-pipeline.sh:850-851 — $out_dir/$safe_fid.patch"
  scope_limit: >-
    Two code changes do not mean that exactly two of the four claims were true.
    Producer constraints, a real defect, and unreachable-path hardening overlapped,
    allowing multiple claims to close through the same changes.

decision_rules:
  admit_durable_task_when:
    - "producer and sink are cited"
    - "the path is reachable in the current revision"
    - "the behavior is reproduced or bounded by evidence"
    - "the task names a closure condition"
    - "future value exceeds future re-evaluation cost"
  allow_intentional_exploration_when:
    - "a human explicitly chooses exploration value"
    - "uncertainty is stated"
    - "evidence or an event that ends exploration is stated"
  discard_when:
    - "the premise is contradicted by the producer"
    - "the affected path no longer exists"
    - "neither verified repair nor intentional exploration applies"
  mechanical_gate:
    rule: "review-origin spawn requires --producer PATH:LINE"
    validates: "citation shape only"
    does_not_validate:
      - truth
      - reachability
      - direct task-file creation outside the journal
      - future line-number stability

claims:
  C1:
    statement: "The three-layer ledger existed for 12:56:34."
    evidence: [R1, R2]
    confidence: high
    scope_limit: "duration starts at the three-layer introduction commit"
  C2:
    statement: "Seven fully deleted files totaled 4,541 lines."
    evidence: [R2]
    confidence: high
    scope_limit: "distinct from the retirement commit's 5,216 total deletions"
  C3:
    statement: "Five core files grew from 1,854 to 4,460 lines."
    evidence: [R0, R1, R2]
    confidence: high
    scope_limit: "scanner files predated the three-layer commit"
  C4:
    statement: "Closing seven review-origin tasks spawned nine child tasks."
    evidence: [R2, "local claims.jsonl"]
    confidence: high
    scope_limit: "raw journal is gitignored; public reproduction is unavailable"
  C5:
    statement: "Seven of twelve review-origin tasks concerned ledger machinery; zero concerned the agent core."
    evidence: [R2, "local claims.jsonl"]
    confidence: high
    scope_limit: "classification is preserved publicly in ADR-0095"
  C6:
    statement: "The HIGH case bundled four claims into two code changes and included a false filename premise."
    evidence: [R3, R5]
    confidence: high
    scope_limit: "not all four claims were false positives"
  C7:
    statement: "The remaining coordination module was 623 lines with 44 Bats tests at the inspected snapshot."
    evidence: [R6]
    confidence: high
    scope_limit: "later revisions may differ"
  C8:
    statement: "Persisting an AI review finding reserves future human attention and judgment."
    evidence: ["causal_model"]
    confidence: medium
    scope_limit: "applies to AI review-origin findings; planned work with known value and closure conditions is outside this rule"

non_claims:
  - "AI code review is generally harmful."
  - "All review findings should be discarded unless already reproduced."
  - "A fixed line count defines overengineering."
  - "An external task tool would necessarily have solved the problem."
  - "The remaining 623-line module is minimal or final."
  - "All four HIGH subclaims were false positives."
  - "A PATH:LINE citation proves a finding true."

references:
  - id: R0
    title: "Introduce the ledger condition scanner"
    url: "https://github.com/shimo4228/contemplative-agent/commit/df68bcee25b61f78c1f0acaedc13da9579127705"
  - id: R1
    title: "Introduce the three-layer ledger"
    url: "https://github.com/shimo4228/contemplative-agent/commit/68e9eaf92d7d2625f98ca9a34d66a2e4c2bba5b7"
  - id: R2
    title: "Retire task-ledger machinery"
    url: "https://github.com/shimo4228/contemplative-agent/commit/0520faf49097598a317ee02a3570fb150551907a"
  - id: R3
    title: "Trace packet filename producers"
    url: "https://github.com/shimo4228/contemplative-agent/commit/510b623c48e068c69d7fe014d4bfcf91c7363e70"
  - id: R4
    title: "Move review admission from severity to premise verification"
    url: "https://github.com/shimo4228/claude-config/commit/7b4b1bc541a1eae427a7d39451474ee54d138ced"
  - id: R5
    title: "Record the HIGH finding"
    url: "https://github.com/shimo4228/contemplative-agent/commit/f0f8c5368bfe43545973e939aa595f45ea0792ae"
  - id: R6
    title: "Snapshot of the remaining claims module"
    url: "https://github.com/shimo4228/claude-config/commit/734a502b0d05f62e7a2b5691558aa04642cde063"

Read-only audit prompt for your coding agent

Audit this repository's durable tasks and the path that promotes AI review findings into tasks. Remain read-only.

Investigation:
1. Enumerate the canonical task store, archive, journal, projection, readers, writers, and hooks.
2. For each component, cite its consumers and recent-use evidence with file:line or commit references.
3. For review-origin tasks, trace each relevant value from producer to sink.
4. Separate findings persisted by severity alone, findings without reproduction conditions, and explorations without closure conditions.
5. Count tasks about the task-management machinery itself and reconstruct their parent-child relationships.
6. Compare the decisions the machinery removes with the decisions it creates.

Output:
- Verified facts: each with file:line, commit, or command evidence
- Unverified assumptions: the producer or observation needed to refute each one
- Keep / Reduce / Retire candidates: reason, impact, and recovery method
- Intentional exploration: items requiring a human value judgment and proposed closure conditions
- Minimal target architecture: the smallest reader / writer / state arrangement that meets the requirements

Constraints:
- Do not edit files, create tasks, mutate state, run git operations, or change settings.
- Do not persist newly discovered problems as tasks.
- Do not infer truth from severity.
- Label a finding "unverified" when its producer cannot be confirmed.
- You may produce an implementation plan, but do not implement before explicit human approval.

Sources and references

Can Six-Month-Old AI Code Survive Today's Review? A 25-Bug Triage

shimo4228 — Fri, 14 Aug 2026 13:00:07 +0000

Six months ago, I had Claude write a CLI tool. All 694 tests were green; mypy and ruff were clean.

Back then I even wrote an article called "Never trust LLM output", listing six defenses.

The other day, I had two lineages of current-generation AI review that proud piece of work end to end.

Every finding I put through verification (31 in total, counting duplicates) turned out to be a real bug. About 25 after deduplication, zero false positives.

But the subject of this article is not "what bugs were in there."

What I wanted to know was: by today's standards, what level was the code written by the AI generation of six months ago — and the harness that propped it up?

Here is the report card, up front:

Skeleton (design): still holds. Only 2 findings around Vision required design changes, and the fixes fit in a patch release
Details that touch the outside world: below shipping grade. Reversible-card submission to Anki and extraction of large images had never worked correctly from the day they were implemented
Aging: no confirmed case. Not one confirmed instance of "the world moved after the code was written and broke it" — every breakage I could confirm was "broken from day one"

The rest of the article backs these three lines with numbers.

The setup: two generations, six months apart, head to head

The test subject is pdf2anki, a Python CLI that auto-generates Anki flashcards from PDFs, implemented over 10 days in February 2026 (the record from that time is in another article).

Between the side that wrote it and the side that audited it lies six months of generational difference — in the models, and in the harness (the full set of skills, rules, and hooks loaded into Claude Code).

	Writer (2026-02)	Auditor (2026-08)
Model	Claude Opus 4.6 (partly Sonnet 4.5) — per commit Co-Authored-By	Claude Fable 5 + GPT-5.6-Sol (via Codex CLI)
Harness	Early ECC adoption. The period of stacking thick scaffolding from community skill collections	After scaffold dissolution. A thin setup with skills and rules heavily cut
Quality assurance	TDD + that generation's code review	The 2 commands in this article
Code state	694 tests passed, mypy/ruff clean	Same (no feature work since the Feb implementation)

ECC (Everything Claude Code) is a community-built collection of Claude Code extensions. Its philosophy was to lift the weaknesses of then-current models with bundles of skills and rules (which this article calls "scaffolding"); my harness started there and, after swelling and a stocktake, headed toward dissolution.

All measurements below are as of 2026-08-13.

Models, prices, and tool behavior keep changing — if you reproduce this, substitute whatever is current at that time.

What I did: run two review lineages in parallel

I used only 2 commands. Both are read-only; neither modifies the code.

The first is Claude Code's built-in /code-review. Point it at the whole directory and set effort to high so it sees everything.

/code-review src/pdf2anki high

The second is codex-review, a skill that calls the OpenAI Codex CLI as a reviewer. It runs in prompt-driven mode, launched with an explicit focus:

bash ~/.claude/skills/codex-review/codex-review.sh "Review the entire codebase under src/pdf2anki/ (not just the diff). Focus on: correctness bugs, error-handling gaps, API misuse (Anthropic SDK, PyMuPDF, Gradio, Textual), security issues, cost-tracking accuracy in cost.py, cache integrity in cache.py, and concurrency/state bugs. Report concrete findings with file:line references. Ignore style nits."

The reason for stacking a model from a different lineage is that models from the same lineage share the same blind spots. I wrote up this design intent in the previous article.

Here is the scale of the audit, in numbers:

Item	Measured
Codex (GPT-5.6-Sol) findings	15 (P1 = top priority ×6, P2 = runner-up ×9). All confirmed real by reading the code
Claude (Fable 5) detection flow	8 perspectives → 44 candidates → 32 after dedup → 16 correctness candidates individually verified → 16 CONFIRMED / 0 REFUTED (several reproduced by actual execution in a venv)
Combined (after dedup)	~25. 23 of them fixed and released as v0.3.1 (39 files, +739/−361)

Verdict 1: fatal or trivial?

Twenty-five bugs sounds catastrophic, but they are a mixed bag. Sorted by severity:

Severity	Count	Contents
Broken features (fail when used)	4	Reversible-card submission to Anki (fails every time) / Vision extraction of large images (total loss) / import inconsistency in TSVs mixing card types / bulk Vision submission of very long PDFs
Silent leaks (money and data)	~13	Budget check bypassed in batch runs / costs overstated 3× / prompt-cache billing not counted / one bad response losing every card in the file / configured model selection ignored, etc.
Robustness / UX	~8	TUI display skips / unhelpful crashes on error / retries applied twice, etc.

Two things follow from this distribution.

First, almost none of these bugs crash loudly. Most of them either fail silently or silently record the wrong amount of money.

That is why demos ran to completion, tests stayed green, and nobody noticed for half a year.

Second, the top two broken features (reversible-card submission and large-image extraction) had never worked correctly from the day they were implemented. They sent a note type name that does not exist and called an API in a way that does not exist — this can be deduced without any guesswork.

So the level of the six-month-old code is not "works, but rough." It is: "shipped in a state where part of the main functionality was dead, and neither the tests nor the author could notice."

On the other side, here is the evidence for the skeleton. Of the 25 findings, only 2 — around bulk Vision submission — required design changes (deferred this time); the remaining 23 closed with local fixes.

The entire fix fitting into a 39-file, +739/−361 patch release is what it looks like when the skeleton holds up even under current-generation eyes.

The representative bugs (with code — only if you're curious)

① A nonexistent Anki note type name (broken feature)

# src/pdf2anki/anki_connect.py:95-102 before the fix
# Anki's standard name is "Basic (and reversed card)". This name does not exist
model_name = "Basic (and target: reversed card)"

The tests were green because the tests themselves asserted the same wrong string.

# tests/test_anki_connect.py:200 before the fix
assert payload["params"]["note"]["modelName"] == "Basic (and target: reversed card)"

② A nonexistent PyMuPDF API call (broken feature)

# Reproduced during verification (pymupdf 1.26.7). Equivalent to src/pdf2anki/image.py:156 before the fix
import pymupdf
src = pymupdf.Pixmap(pymupdf.csRGB, pymupdf.IRect(0, 0, 100, 80))
scaled = pymupdf.Pixmap(src, 50, 40, None)          # the correct form works
bad = pymupdf.Pixmap(src, pymupdf.Matrix(0.5, 0.5))  # TypeError

This TypeError was swallowed by a broad except Exception: continue and lurked as "only images above a certain size silently disappear." The verification agent reproduced extraction returning 0 images on a PDF containing a 2000×2000 image.

③ A price table with the previous generation's prices written in (silent leak)

# src/pdf2anki/cost.py:17-21 — per-model prices ($/MTok, input/output)
- "claude-haiku-4-5":  (0.80, 4.00)   # actual is $1 / $5 (20% under)
- "claude-opus-4-6":   (15.00, 75.00) # actual is $5 / $25 (3x over)
+ "claude-haiku-4-5":  (1.00, 5.00)
+ "claude-opus-4-6":   (5.00, 25.00)

Verdict 2: why did the tests stay green for half a year?

The 25 broken places shared 2 common mechanisms. The mechanisms are worth more than the individual bugs.

Mechanism 1: the tests go green together with the assumptions.

Tests replace the real external things (Anki, PyMuPDF, the API) with fakes (mocks). Nearly all of these bugs lived outside that replacement boundary.

When the same AI writes the implementation and the tests in the same session, its assumptions get copied to both sides. The tests can only verify "does the implementation match my assumptions" — never "do my assumptions match the world."

Testing as a practice is not powerless. The fixes added a regression test that pushes a large image through the real PyMuPDF, so this class of bug will be caught by tests from now on.

What is powerless is a mock written with the same assumptions as the implementation.

Mechanism 2: the writer's (the AI's) knowledge is already stale at the time of writing.

The price table in the code said $15/$75 for Opus 4.6 and $0.80/$4.00 for Haiku 4.5 (actual: $5/$25 and $1/$5, respectively). These wrong numbers exactly match the prices of the previous generation — Opus 4/4.1 and Claude 3.5 Haiku.

The Opus 4.6 launch announcement (2026-02-05) says "Pricing remains the same at $5/$25 per million tokens", and the implementation was on 02-08.

In other words, the model wrote the previous generation's going rate, memorized during training, straight into the code as the new model's price.

What's more, a research doc inside the repo, written the day after implementation, had the correct prices on record. The right answer was sitting right there — and the code got the memory instead.

Full disclosure: the first draft of this article also explained this as "a price cut after the code was written broke it." Until fact-checking refuted it, I believed that plausible memory myself.

Staleness sneaks in whether the writer is an AI or a human.

Both of these mechanisms produce "broken from day one." And every breakage I could confirm was one of the two; confirmed cases of "the world moved after the code was written and broke it" came to zero this time.

The six-month-old code had not decayed with age. It was broken from the start, and simply ran unnoticed.

The countermeasures map onto the mechanisms. Against replicated assumptions: cross-checking against the real thing, independent of the implementation (a different-lineage review, or verification against the real dependency). Against staleness: re-verification after writing, not a single check at write time.

The 2 commands in this article were a cheap way to run both at once.

What the audit side revealed: the two models looked at different places

The overlap between the two models' findings — counting same-file, same-substance findings as one theme — was 4 themes (5–6 by finding count). Only about 20% overlap by count.

Category	Count	Tendency
Independently agreed	4 themes	Budget bypass, cache-key flaw, note type name, ignored model selection
Codex only	~10	Strong on external contracts: API pricing, the prompt-cache billing scheme, Anki's TSV import spec
Claude only	~10	Strong on runtime failure paths: nonexistent API calls, exception propagation, data loss on partial failure

Flip that around: either one alone would have found only around 60% of the ~25.

The 4 independently-agreed themes worked as a priority signal. When two models from different lineages point at the same spot, that spot is almost certainly real.

In practice, I fixed in this order: both-agreed + billing → total-data-loss → UX.

How much of the scaffolding could be dissolved?

The other thing I wanted to measure in this audit is the harness row of the setup table.

Six months ago, I stacked thick scaffolding (bundles of instructions — skills, rules, procedure docs) to supplement the model's judgment. These days I keep dissolving it.

Period	Harness state
2026-02 (implementation)	Early ECC adoption. Thick scaffolding to lift the then-current model
2026-03	ECC itself had swollen to 116 skills. Disabled the plugin; cut my imported 33 skills down to 16
2026-07	Cut roughly 8–9k tokens of always-on rules. The judgment: "instructions that compensated for an old generation's weakness are shackles on the new one"

Can a thin-harness, current-generation set properly inspect the products of the thick-harness era?

The result: all 31 findings put through verification (15+16) were real, zero false positives. That precision came without any author-written procedure docs.

Saying the procedures are gone would be inaccurate, though. /code-review itself has a multi-stage procedure built in — "explore from 8 perspectives → a verification agent tries to refute." The procedure moved from the author's bolt-on into the tool's built-ins.

Meanwhile, 2 mechanisms in a layer separate from instruction scaffolding were still earning their keep.

The first is machine gates. The fix commit was bounced 3 times by pre-commit hooks (security scan → formatter → secret detection), and passed all 3 through real fixes.

The layer that machine-checks LLM output is still doing its job in the current generation.

The second is human intervention points. The review agent sat stalled, waiting on verification, for about 40 minutes — and only ran to completion after I sent a resume instruction.

The "press the button when it stalls" human is still required.

The scaffolding that taught judgment could be peeled away. The machinery that inspects, and the intervention point a human presses when things stall, remained. That is where the dissolution stands today.

Operational tips (fine print for reproducing this)

Review agent stalls: a background review being quiet does not mean "task complete." It may simply be stalled. This time, after ~40 minutes of no activity, one message — "collect the verification results and write the final report" — got it to finish in ~2 minutes
Always check price/spec fixes against current primary sources: fixing a price table because "the AI said so" just plants one more stale fact
Fix "both-agreed + billing" first: findings two models reached independently carry the lowest false-positive risk, and billing bugs cost real money while they sit
Cost: the Claude-side review subagents consumed about 200k tokens in total (95,229 + 106,860). A full-codebase effort-high review is not cheap — running it at milestones is the realistic cadence

Wrap-up: how to read the report card

The level of the code written by the six-months-ago generation set (Opus 4.6 + a thick harness) was this:

Skeleton: pass. Only 2 findings around Vision needed design changes; the fixes fit in a patch release
Details that touch the outside world: fail. 4 broken features, ~13 silent leaks. Some features had never once worked
Aging: no confirmed case. Every confirmed breakage was "from day one." It simply went unseen for half a year

And the green tests taught us nothing about this gap. 694 green tests coexisted with "part of the main functionality is dead" for half a year.

What worked as the measuring instrument was two reviews from different lineages. Either one alone saw only ~60%, so being two lineages is the point.

The harness answer also fits in one line: the scaffolding that taught judgment could be peeled away; the machinery that inspects and the intervention point a human presses remained.

If you have code an AI wrote months ago, still running with green tests, the same measurement is 2 commands away.

And six months from now, the code today's Fable 5 writes will rotate into the audited seat before the next generation. What grade will it get? Keeping it in a shape where that can be measured is, I think, the preparation available to us now.

Sources

Anthropic, "Introducing Claude Opus 4.6" (primary source for the $5/$25 launch pricing)
Claude Platform Docs, "Pricing" (current prices as of 2026-08-13)
Anki Manual, "Card Generation" (standard note type names)
PyMuPDF, "Pixmap" (Pixmap constructor spec)

Claude Code's Memory Has No Vectors — Try ADRs Before Memory RAG

shimo4228 — Wed, 05 Aug 2026 13:00:07 +0000

There is not a single vector in Claude Code's memory implementation.

Counting my project's auto-memory directory: 89 plain Markdown files plus one MEMORY.md that indexes them. No embeddings, no vector DB, no chunking — nothing (measured on 2026-08-05).

$ ls ~/.claude/projects/<project>/memory/*.md | wc -l
89
$ find ~/.claude/projects/<project>/memory -type f ! -name "*.md"
# → One manual backup only. Zero vector assets

Code search is the same. Current Claude Code (as of 2026-08) searches code with Grep / Glob / Read, and there is no built-in mechanism that builds an embedding index. This isn't someone's claim — it's a fact you can verify today by opening your own Claude Code.

What's interesting is that this is not "not implemented yet" — it's the result of trying it and removing it. Early versions used RAG + a local vector DB, and the author, Boris Cherny, has explained why it was dropped. He first said this on Hacker News in February 2025 and restated it in an X post in February 2026. Here is the X version:

Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn't have the same issues around security, privacy, staleness, and reliability.

If you're giving an agent memory, the default answer is memory RAG or graph RAG. My position is the opposite. If you're building memory RAG or graph RAG, first try plain ADRs (Architecture Decision Records) and structured memory. I believe it will solve most of your problem.

Let me say upfront: this is not an anti-RAG argument. My own agent (a self-built agent I run separately from Claude Code — details below) runs on the full benefit of embeddings. This article is about scope: under which conditions RAG wins, and under which conditions letting the model read wins.

This article does two things:

Decompose RAG into a single question — "who makes the relevance judgment, and when" — and derive a decision rule for when RAG wins and when it loses
Back that rule with observations of Claude Code's implementation, plus failures and measurements from my own environment (embedding clustering over-merging, a prefix-cache A/B test)

Decomposing RAG — who makes the relevance judgment

The core of RAG is not "search." It is who decides, and when, what is relevant to this context.

Embedding-based RAG places that judgment like this:

At write time: documents are converted to embedding vectors and stored. From then on, the only information available for retrieval is this frozen representation
At read time: the query is vectorized with the same embedding model, and "relevance" is judged by vector similarity (cosine similarity being the standard choice)

The similarity computation itself runs at read time. But the judge is the embedding model — a model that doesn't read the conversation's context, doesn't grasp the question's intent, and was frozen at training time. In other words, RAG is a design that delegates the relevance judgment occurring at every read to a weak, frozen judge that doesn't read context. Under conditions where LLMs are weak or expensive, this is still a rational division of labor.

Note that what I decomposed here is bare embedding search (single-stage dense retrieval). Stacks with query rewriting, hybrid search, and metadata filters move part of the judgment outside the embedding — but that is an evolution in the direction of piling corrective devices around a weak judge. This weakness is well known in practice, which is exactly why rerankers — feeding the search results to an LLM for re-judgment — became a standard later stage. A reranker is a design that concedes: "for the final judgment, letting the model read is more reliable." For frontier-class LLMs, context processing is the core job, and there is no dimension on which vector similarity beats their ability to read the full text and judge "which of these matters for the current question." Compared to bare embedding search, letting the model read as much as it can read wins on judgment quality.

If that's true, the design question changes. Not "how do we search" but "can the model read it".

The decision rule — when RAG is needed, when it isn't

Here is the decision map up front. Two axes.

	Frontier-class LLM	Weak LLM (small local model, etc.)
Readable scale (memory, hundreds of docs)	Structured memory + ADRs. No vector search — let the model traverse from an index and read	Embeddings are effective (you can't afford the cost of reading)
Unreadable scale (large corpus)	Agentic search (grep + read) or RAG	RAG

The right column and the bottom row are RAG's legitimate territory: searching large corpora, fuzzy recall that keywords can't catch, and cases where the LLM itself is weak. Building these with RAG is correct.
The problem is the top-left. If you're building an embedding pipeline for an agent's self-memory — a store that you (or the agent) write to and that the LLM can read in full — you are delegating a high-value judgment to a weak judge. This article's claim applies to that quadrant only.

One clarification: the top-left claim is not "eliminate search." It is a reassignment — move the relevance judgment back from the embedding to the LLM, and demote retrieval to plain file reading. A new design problem remains: how to maintain the index (more below).

The rest of the article backs this top-left quadrant with three pieces of evidence.

Evidence 1: Claude Code's implementation — search is demoted to a tool

Claude Code's memory runs on plain Markdown plus an index alone. Pulling the structure from the official docs (as of 2026-08), it comes down to three points:

Only the index is resident: only the first 200 lines (or 25KB) of MEMORY.md are loaded each session
Write-time compression is enforced: as the index approaches its limit the harness warns, and exceeding it returns an error demanding a rewrite. The compression protocol — "keep one entry per line, push details out to topic files, consolidate and delete stale entries" (the gist of the official docs) — is enforced by the tool, not by the model's good intentions
Bodies are read on demand: topic files are not loaded at startup; when needed, Claude reads them with the ordinary file tool (Read)

The third point is the crux. Retrieval is not an independent pipeline — it has been demoted to an ordinary tool call. The judgment of "what to read" is made at runtime by the LLM looking at the index, not by an embedding.

Anthropic itself does not frame this structure as an ad-hoc omission. The September 2025 post Effective Context Engineering for AI Agents explicitly states the "just in time" approach as a design principle: instead of pre-processing all data, keep lightweight identifiers (file paths, etc.) and load with tools at runtime. "Structured note-taking (agentic memory)" — the agent writing Markdown notes — is also recommended in the same post, and Claude Code's implementation today remains consistent with this published direction.

The same post (2025-09) also states the trade-off: "runtime exploration is slower than retrieving pre-computed data." The speed counterargument is addressed in Evidence 3 below.

Evidence 2: embeddings can't see "the concrete difference" — my failure

I have been running an autonomous agent unattended for months on a small local model on an M1 Mac, and within that I have concretely failed with embedding clustering. The record is public, in ADR-0046.

I built duplicate detection for the agent's auto-extracted skills using embedding cosine similarity + clustering:

The skills share boilerplate vocabulary, so skills that are behaviorally distinct scored 0.90+ cosine similarity
Linkage clustering chained those pairs, over-merging 18 skills into a single blob
Switching back to passing all skill bodies to a single LLM call for classification recovered a sensible result: 18 → 5 groups + 8 independent

This was not a "bad threshold tuning" failure. The ADR records the alternatives considered — raising the cosine threshold from 0.80 to 0.90 didn't separate them, and neither did changing the linkage method, because boilerplate vocabulary dominates this store. The concrete difference — same vocabulary, different instructed behavior — simply does not show up as distance in embedding space. An LLM reading the full text sees it. At readable scale, letting the model read wins — a real instance of the top-left quadrant.

Evidence 3: "select and inject" breaks the cache — a prefix-cache A/B

"Reading everything is slow and expensive, so retrieve only the relevant part and inject it" — that's RAG's speed argument. But when you measure it, a hidden cost appears: injection whose content changes every call breaks the prefix cache every call. The prefix cache is the mechanism that, when the head of the prompt matches the previous call, reuses that evaluation and skips input processing (it exists both in local llama.cpp and in frontier APIs).

I ran a controlled A/B locally (Ollama / gemma4:e4b, num_ctx=32768, M1 16GB) on 2026-08-05 (raw data public). We compare prefill time (prefill: the evaluation of the input prompt portion).

Condition	Prefill time
~32K-char system prompt, first call (cold)	38.85s
Repeat call with byte-identical system (5/5 reproduced)	0.061–0.064s
Repeat call with part of system (skill-injection section) swapped	Full cold every time (6.3–7.1 ms/tok)

Note: only the third row is token-normalized (ms/tok) because rotating the skill-injection section made the system size vary between 29K and 57K chars. The "full cold every time" verdict rests on all cold calls consistently landing at 6.3–7.1 ms/tok.

Byte-identical is roughly 600x faster. Swap any part, and even with a shared head, the cache barely helped.

Production telemetry (30 days, n=1,213) shows the same structure. The configuration that selects and injects skills per call (n=604) has a p50 of 56.1 seconds; the configuration that injects everything without selection (n=609) has a p50 of 28.3 seconds. The side that "injects only what's needed" is 2x slower in wall-clock time. The production aggregate is not a controlled experiment, so I won't overclaim — but it is consistently explained by the same mechanism as the A/B above: selection changes the system prompt every call, so every call pays a cold prefill.

Measurement caveat — token-count-based monitoring cannot see the cache effect

On the Ollama path, prompt_eval_count reports the full prompt token count (6,202 in this experiment) even on cache hits. Actually evaluating 6,202 tokens in 0.061 seconds is physically impossible, so hit detection is only possible via prompt_eval_duration. If you only watch token counts, "inject everything" will keep looking expensive.

This is a local llama.cpp measurement, and partly specific to a prompt layout whose shared head is only ~2K chars. But frontier API prompt caching is also conditioned on prefix matching from the start of the prompt, so the principle is the same — everything after a variable injection point gets recomputed every call. If you put the stable part first and inject retrieval results at the tail, you protect the cache for the head; but a design that splices retrieval results into the middle of the system prompt, or has a long shared section after the injection point, pays the same cost measured here. The lesson: "what to inject, and where" is not just a retrieval-precision question — it is a cache-design question.

Objection: "you just moved the cost to write-time discipline"

I think the strongest objection to all of the above is this:

RAG's advantage is zero write cost. Throw everything in, search later. ADRs and structured memory demand write-time discipline — compression, indexing, curation. Haven't you just traded search complexity for a documentation discipline that most teams cannot sustain?

My response has two parts.

First, that discipline can be carried by the agent, not by humans. As Evidence 1 showed, Claude Code's auto-memory has the agent itself doing the work of choosing what to keep, compressing it, and updating the index. My 89 files weren't written by me being meticulous — the agent accumulated them from session learnings. The folk wisdom that "documentation discipline never lasts" is about human teams; with an LLM on the write side, the premise has changed.

Second, reads and writes are asymmetric. A write happens once per piece of information; the read-time relevance judgment happens every session, every time. RAG delegates that every-time judgment to a weak judge, placing the cost and the quality degradation on the high-frequency side. Having a strong judge (an LLM) compress and organize once at write time is not a cost shuffle — it is a correct move to the low-frequency side.

My own agent runs on the full benefit of RAG

In the decision table I wrote "with a weak LLM, embeddings are effective." The very agent whose failure I showed in Evidence 2 is exactly that case — far from not needing RAG, it runs supported by embeddings.

This agent runs on a small local model (8B class) as a deliberately chosen constraint. Measured cold prefill is 6.3–7.1 ms/tok (same agent and same model as the A/B in Evidence 3). It cannot afford to "read and judge every time" over a knowledge store of hundreds of patterns.

So pattern classification, duplicate detection, and noise filtering are built on embeddings (ADR-0019). Strictly speaking this is not retrieval-augmented generation itself but "embeddings as a judge" — yet it is the same core design this article decomposed: delegating relevance judgment to a frozen judge. The effect shows up in numbers — replacing LLM-call-based classification with embeddings cut LLM time by about 20 minutes per day (ADR-0019, recorded as of 2026-04; that is an embeddings-vs-LLM-calls comparison, not a comparison against a "let the model read" configuration). In this environment, this division of labor clearly wins.

There is no contradiction with the failure in Evidence 2. That case was a one-off judgment over 18 skill bodies — readable scale — so moving back to the LLM was correct. This case is also readable in scale, but the classification fires daily at high frequency, and an 8B model cannot pay the "read every time" cost, so embeddings are correct — the same agent, with the decision rule applied per mechanism.

In other words, my environment sits in the right column of the table (weak LLM), where the RAG-side design is the right answer. Conversely, an agent with Opus or a GPT-class frontier model as its backend has its self-memory in the top-left quadrant (frontier-class LLM x readable scale) — and building an embedding pipeline there means outsourcing judgment to a weak judge while a strong judge is standing right there. The same design is correct in one quadrant and wrong in another — that is this article's decision rule.

Why "ADR" specifically

Among structured memory formats, there is a reason I recommend ADRs (Architecture Decision Records) in particular. The thing that rots fastest in an agent's memory is "why we decided this," and the ADR is the format that pins it down as a unit of decision + context + rejected alternatives.

Write-time compression is built into the format: the act of writing an ADR is itself the compression of "what was decided, and on what grounds"
Expiry is handled: ADRs carry status (accepted / superseded), and new decisions explicitly override old ones. The problem of contradictory memories cohabiting in a vector store doesn't arise — the format prevents it
The graph can be drawn by hand: reference links between ADRs ("ADR-0046 refines ADR-0016") are drawn by the author at write time. The relationships that graph RAG tries to infer at runtime get fixed at the moment they are best known

My public repo has 89 English ADRs (a different store from the 89 auto-memory files at the top — the equal count is coincidence), and Claude Code traverses them with grep and Read at the start of a session. Vector search never enters the picture.

Summary

The core of RAG is delegating relevance judgment to a weak, frozen judge that doesn't read context. Where LLMs are weak or expensive it remains a rational division of labor — but in the quadrant where a frontier-class LLM handles a readable scale, it is degraded context processing
The decision rule is two axes: readable scale x LLM strength. For large corpora, fuzzy recall, and weak LLMs, RAG remains legitimate
If you are in the agent self-memory x frontier-class LLM quadrant, before building an embedding pipeline, try an index + plain Markdown + ADRs. Claude Code runs that design in production today, and the write discipline can be carried by the agent
"Select and inject" has a measurable hidden cost: it breaks the prefix cache after the injection point on every call

All numbers, failures, and ADRs from my environment are in a public repo. Verification and refutation are welcome.

Give Claude Code a Second Harness

shimo4228 — Sun, 02 Aug 2026 12:24:40 +0000

The CLAUDE.md files, rules, and skills you have built up are useful in everyday development. But they can be too heavy for experiments or autonomous loops. Sometimes you want to swap the whole set for a different one.

Claude Code, however, has no single profile switch that replaces only your personal harness while preserving the repository configuration. The closest mechanism is CLAUDE_CONFIG_DIR, which points Claude Code at a different configuration directory. But it switches only half of the harness. Skills and agents move; CLAUDE.md and rules from the original environment still come along. On my machine, that meant 16 files and 3,404 words on every launch.

That is the missing piece. The solution is to combine a configuration-directory switch with path-based instruction exclusion.

The first half of this article explains why the combination is necessary and what it preserves. The second half gives you a planning prompt for your coding agent. The agent inspects your environment read-only and stops after producing an implementation plan.

Prerequisites

Tested with Claude Code v2.1.220 on macOS
In this article, “harness” means the set of CLAUDE.md, rules/, skills/, and agents/
I call the new directory selected by CLAUDE_CONFIG_DIR the “alternate side”

The configuration directory switches only half the harness

CLAUDE_CONFIG_DIR is an environment variable that points Claude Code at another configuration directory. Here is what moves—and what stays behind.

Item	Part of the harness?	Switches to the alternate side?
`skills/` and `agents/`	Yes	Switches
`CLAUDE.md` and `rules/`	Yes	Does not switch; the originals still load
`settings.json` (including hook definitions)	No	Switches
Session history and plugins	No	Switches

Only the top half of the four-part harness changes.

The bottom two rows follow the official documentation:

Override the configuration directory (default: ~/.claude). All settings, session history, and plugins are stored under this path
(Environment variables)

Skills and agents do not appear in that sentence, but they switched in my test. When I launched Claude Code with the alternate directory and asked for the available skills, my usual custom skills were gone. Only the skills placed on the alternate side appeared.

Credential storage depends on the operating system. On Linux and Windows, credentials live under the configuration directory. On macOS, they live in the system Keychain. In the macOS environment tested here, changing the configuration directory does not move a credential file with it.

CLAUDE.md and rules remain because they are found through another mechanism:

Claude Code reads CLAUDE.md files by walking up the directory tree from your current working directory
(How Claude remembers your project)

When the working directory is below your home directory, that walk necessarily passes through $HOME. This makes $HOME/.claude/ part of the search path regardless of where CLAUDE_CONFIG_DIR points.

How I verified the ancestor-directory behavior

I launched Claude Code with an empty configuration directory. Nothing could come from that directory, but Claude Code still loaded 16 files from $HOME/.claude. It recorded them as project files rather than user files.

Launch condition	Files loaded from `$HOME/.claude`	Recorded type
Default configuration directory (`$HOME/.claude`)	16	User (`User`)
Empty alternate configuration directory	16	Project (`Project`)

The label changed; the files still loaded. Placing the working directory outside the home directory removed this path. On v2.1.220, pointing a symlink outside the home directory back to a directory under it did not help because Claude Code resolved the real path.

Claude Code does provide official ways to disable customizations broadly. None draws the boundary needed here.

Official mechanism	What it disables	Why it does not fit this case
`--safe-mode`	CLAUDE.md, skills, hooks, plugins, and other customizations	Repository customizations stop too
`--bare`	Automatic loading of CLAUDE.md, hooks, plugins, MCP, auto memory, and more	It is a minimal mode for scripted calls, and repository customizations stop too. Skills remain explicitly invocable
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`	User, project, and auto-memory instruction files	It also removes the repository CLAUDE.md

The target behavior is narrower: replace the personal harness while keeping the repository’s instructions and mechanical gates. Claude Code has no single official switch for that selective profile change. Issue #30380, which requests per-session disabling of the global CLAUDE.md, is closed as not planned.

The human decision is where to draw the boundary

This is the boundary used here:

Item	Treatment
Everyday `$HOME/.claude`	Leave unchanged
Personal CLAUDE.md, rules, skills, and agents	Replace with a second set
Repository CLAUDE.md, rules, settings, and hooks	Preserve
Project memory	Separate by default; share only if needed

This gives you a harness for experiments, autonomous loops, or a separate account without changing the environment you use for everyday development.

The mechanism has two parts:

Use CLAUDE_CONFIG_DIR to change the location of settings, skills, agents, and history.
In the alternate settings.json, use claudeMdExcludes to exclude the original $HOME/.claude/CLAUDE.md and rules/ from instruction loading.

claudeMdExcludes accepts absolute-path glob patterns. In a terminal, ~ (tilde) is normally shorthand for your home directory. It is not expanded here. Writing ~/.claude/** fails to exclude anything without producing an error in v2.1.220.

On a company- or school-managed machine, IT may deploy a machine-wide CLAUDE.md. Personal settings cannot exclude those managed instructions. If your machine is not centrally managed this exception does not apply.

Everything after this point depends on the reader’s filesystem, existing settings, symlinks, and authentication state. Do not copy a generic sequence of commands from an article. Have your coding agent inspect your machine and turn the design into an environment-specific implementation plan.

Give this article to your agent and ask for a plan

Give your coding agent the URL of this article and the following prompt. At this stage, the agent must not change any files.

Read this article and create an implementation plan for adding a second, purpose-specific Claude Code harness to this machine.

Important:
- Perform read-only inspection for this request.
- Do not create, edit, move, or delete files. Do not change settings, create symlinks, or perform Git operations.
- Put commands in the plan, but do not run commands that mutate state.

Preferences:
- Candidate TARGET_CONFIG_DIR: ~/.claude-alt
- Contents of the second harness: undecided. Look for existing candidates on this machine.
- Share project memory: no

Goals:
- Do not modify the everyday ~/.claude directory.
- Make it possible to place a separate CLAUDE.md, rules, skills, and agents in TARGET_CONFIG_DIR.
- When Claude Code starts with TARGET_CONFIG_DIR, do not load the original ~/.claude/CLAUDE.md or ~/.claude/rules/**.
- Preserve CLAUDE.md, .claude/rules, settings, and hooks from the working repository.

If you cannot access the article, treat the inspection and planning requirements below as canonical.

Read-only inspection:
1. Check claude --version, the operating system, and the resolved HOME path.
2. Inspect the structure of ~/.claude, the candidate configuration directory, and the repository's .claude directory. Do not copy their contents into a public report.
3. Look for candidates for the second harness. List possible sources separately for CLAUDE.md, rules, skills, and agents. If none exist, propose an empty structure.
4. Check for existing settings.json files, symlinks, projects/, and managed policy.

Requirements for the plan:
- Do not modify the everyday ~/.claude directory or files in the repository.
- Resolve the candidate configuration directory to an absolute path. Reject /, HOME, ~/.claude, any path below ~/.claude, and an existing symlink as the target.
- Do not overwrite existing files. For same-name assets, show the diff; skip identical content and make differing content a human approval point.
- Do not dereference symlinks when copying. Show each link target and its risk.
- Preserve existing keys in settings.json and merge the resolved absolute `<HOME>/.claude/**` glob into claudeMdExcludes. Do not use `~/.claude/**`.
- Show the destination of each CLAUDE.md, rule, skill, and agent file.
- Take a no-exclusion baseline before adding the exclusion, then compare it with the result after the change.
- Use a temporary directory and `--settings` for verification. Do not rewrite the real settings.json or existing hooks merely to run the test.
- Limit the InstructionsLoaded hook to `matcher: "session_start"`. After each process exits, wait for the asynchronous log to stabilize with a bounded timeout. Compare unique `file_path` values and their sources.
- Success means zero files loaded from the original ~/.claude, loading the second CLAUDE.md and rules when present, and preserving repository instruction files.
- Use /context or InstructionsLoaded records as evidence. Do not infer success from what the LLM says it knows.
- Remove temporary verification files afterward. Recheck the final settings.json diff and JSON syntax.
- Keep project-memory sharing out of the main plan. Describe it only as an alternative, including mixed conversation history and the deletion risk from `claude project purge`.

Output:
1. Summary of the current state
2. Proposed structure and the reason for choosing it
3. Implementation steps, each with target paths, commands, intended changes, and a rollback method
4. Verification steps and success criteria
5. Human approval points
6. Unverified assumptions and risks

Stop after presenting the plan. Do not implement it until I explicitly approve it.

Review the proposed target paths, no-overwrite behavior, rollback method, and success criteria. If the plan is sound, explicitly tell the same agent: “Implement this plan and verify every success criterion it defines.”

The planned layout should look like this:

~/.claude-alt/
├── CLAUDE.md        # when the second-harness plan includes one
├── settings.json
├── rules/
├── skills/
└── agents/

The essential addition to settings.json is one key. The plan must merge it without discarding existing keys.

{
  "claudeMdExcludes": [
    "/Users/you/.claude/**"
  ]
}

/Users/you/ is illustrative. The real file must contain the absolute HOME path detected on the reader’s machine.

Judge success by the load source

For a quick manual check, launch Claude Code with the alternate configuration, run /context, and inspect Memory files.

For an automated check, use the InstructionsLoaded hook. Each time Claude Code loads a CLAUDE.md or rule, the hook receives the absolute file_path, memory_type, and load_reason. The planning prompt requires a controlled comparison using those records.

On my Claude Code v2.1.220 environment, changing only the exclusion produced this result:

Load source	No exclusion	With exclusion
Original `$HOME/.claude`	16	0
Repository	1	1
Total	17	1

With ~/.claude/**, the total remained 17. Replacing it with the absolute path reduced the total to 1. These are v2.1.220 measurements. Judge success from the file_path breakdown, not from whether the command prints an error.

Do not ask the LLM, “Do you know this instruction?” to verify loading. The model may know the same information from repository memory or training data. Use Claude Code’s own /context output or InstructionsLoaded records.

Optional: share project memory while keeping the harnesses separate

Project memory also lives under the configuration directory, so it disappears from the alternate side by default. If you want to reuse it, do not create a link immediately. Change “Share project memory” in the planning prompt to “consider sharing,” then ask the agent to produce a separate plan.

Linking all of projects/ shares future projects without creating new links, but it also shares conversation transcripts. Sessions from both environments appear together in /resume.

Do not run claude project purge while sharing this directory. In a local v2.1.220 --dry-run, one project produced 72 deletion targets. If projects/ is a symlink, deleting projects/<slug>/ follows the link and removes the everyday environment’s transcripts and auto memory. Other purge targets—file history, tasks, configuration entries, and input history—belong to the alternate side.

Summary

The design has two parts:

Use CLAUDE_CONFIG_DIR to switch the configuration directory. Skills, agents, hooks, and history move here.
Add claudeMdExcludes to the alternate settings.json. This removes the original CLAUDE.md and rules from the remaining instruction-loading path.

The result is a Claude Code launch that swaps the personal harness as a unit. There is no single switch for this boundary, but you can get the same behavior by explicitly excluding the part that would otherwise remain.

Always verify the load sources after adding the exclusion. The dangerous failure mode is silent: an ineffective glob produces no error.

References

How Claude remembers your project — CLAUDE.md load order and claudeMdExcludes
Hooks reference — the InstructionsLoaded hook
Environment variables — environment variables including CLAUDE_CONFIG_DIR
CLI reference — --safe-mode and --bare
Issue #30380 — request to disable the global CLAUDE.md (closed as not planned)

Banned from Wikidata Overnight — I Believed Every Edit Was Compliant, but All 109 Items Were Deleted as Promotion

shimo4228 — Thu, 30 Jul 2026 13:00:04 +0000

On the morning of July 16, 2026, I noticed a notification from Wikidata. When I opened it, my account was indefinitely blocked. The reason field said "Promotion-only account." My first reaction was: "Wait — I didn't do anything wrong."

I pulled the deletion log. Between 01:58:29 and 02:00:51 — about two and a half minutes — all 109 items I had created were mass-deleted. Six weeks of building, gone overnight.

"Subjects listed on Wikipedia or Wikidata get cited by AI. So register your own work as entities." — If you follow discussions of LLM-era discoverability (so-called GEO: Generative Engine Optimization), you have probably seen this idea somewhere. I acted on it, and this is how it ended.

The puzzling part is that I believed every individual edit was policy-compliant. I knew you must not write your own Wikipedia article. But Wikidata's policy is different: editing items about yourself (an item is Wikidata's unit of registration — a person, a work, a concept each gets one) is generally tolerated as long as you meet the sourcing requirements. The guideline Wikidata:Autobiography says so explicitly.

Under that understanding, I attached references to every statement and passed Wikidata's constraint checks. My talk page — the public per-user contact page where policy warnings land first — never received a single warning. The block notice names no individual edit as a violation either.

And yet I lost everything. Why?

This article does three things.

Why the total loss happened — I explain the structure in which the judgment is rendered per account, not per edit
The incident response I closed out the same day — the withdrawal decision, purging references, and designing the apology, exactly as I did them
How to convert failure into norms — the procedure I used to turn this incident into a design decision record (ADR) and into rules my agent loads on every session

What I was building, and what happened

Some context first: why was I registering DOIs and the like as an individual? I have no ambition to become an academic or a researcher.

Writing implementations with AI coding tools, I came to feel that the cost of implementation is steadily evaporating, and that any code I write will be obsolete soon.

Then at least I wanted to keep a dated record of judgment — "at this point in time, this is what I was thinking." So I started consolidating my thinking into public repositories (the "research repos" below) and papers, attaching DOIs (persistent identifiers for papers and software; services like Zenodo let you mint them yourself), making it all citable with dates.

The Wikidata investment was an extension of this.

But a record that never gets discovered might as well not exist. And what decides whether something gets read today is generative AI, more than search engines.

Readers ask an AI; works the AI cites get found, and works it does not cite might as well not exist. So I also invested in the groundwork for being discovered by AI (GEO).

What encouraged me was GEO measurement data. Multiple studies of generative-AI citation sources report that Wikipedia consistently ranks near the top of ChatGPT's cited domains (second tier, just behind Reddit) — see Semrush and Ahrefs, among others.

I have not found any measurement isolating the effect of Wikidata itself, as far as I searched.

Still, I reasoned: "Wikidata is what supplies the machine-readable entity data behind Wikipedia. If I register my author, repositories, and papers as items, cross-linked with identifiers like DOIs, an AI should be able to establish that this author and these works exist and connect this way." — That was my read, and I invested in this layer.

Looking back, the boundary between measurement and hope was already blurred at this point.

Here is what I built: an author item, repository items, paper items, and bibliographic items for the external works my papers cite — all connected with citation-relation properties.

The edits were executed by an AI agent (Claude Code) in a semi-automated setup, with me approving before any batch of writes. Every edit carried references. Over six weeks it grew to 109 items.

When	What
06-07	Created author item + 3 papers + 7 repositories
06-12	Added 27 bibliographic items for cited works + citation edges
06-13 to 26	Added external identifiers, more paper items
Late night 07-15 to 16	Created 19 new bibliographic items + bulk-synced citation edges from 14 to 33
07-16 01:58–02:01 UTC	All 109 items mass-deleted + account blocked indefinitely

The block reason: "Promotion-only account: Promoting Tatsuya Shimomoto." The deletion log reason: "Spam / advertising: Mass deletion." Anyone can still verify this judgment through Wikidata's public API. (Throughout this article, every count and timestamp is a value you can confirm via the public API; wherever no log survives, I flag it explicitly as the author's recollection.)

# Count deletions related to the account in the 2026-07-16 mass-deletion log
curl -s "https://www.wikidata.org/w/api.php?action=query&list=logevents&letype=delete&leuser=The%20Squirrel%20Conspiracy&lelimit=500&format=json" \
  -A "audit/1.0" | python3 -c "
import json,sys
evs = json.load(sys.stdin)['query']['logevents']
mine = [e for e in evs if 'Shimo4228' in e.get('comment','')]
print(len(mine))  # -> 109
"

Note that the Wikidata API rejects requests without a User-Agent with a 403, so the -A flag is required.

The block reason text is also retrievable verbatim via the API

# Fetch target, timestamp, and reason of the currently active block
curl -s "https://www.wikidata.org/w/api.php?action=query&amp;list=blocks&amp;bkusers=Shimo4228&amp;bkprop=user%7Cby%7Ctimestamp%7Creason&amp;format=json" \
  -A "audit/1.0" | python3 -m json.tool

The cause: judgment is per account, not per edit

Mechanical validation (presence of references, constraint checks, schema validity) measures the technical validity of each individual edit. What Wikidata's human administrators look at is the pattern drawn by the account's entire contribution history.

My 109 items — author, the author's works, the author's papers, the works the author's papers cite — varied in kind, but they all traced a web pointing at one single person. Any administrator opening this history for the first time sees "promotion-only," regardless of how careful each individual edit was.

Indeed, the block notice contains not one citation of an individual violating edit. The granularity of the judgment is simply different.

An immediate indefinite block with no warning is not unusual, either. Wikidata's blocking policy states explicitly that a warning is not required before a block, and blocking accounts judged promotional or spam indefinitely without warning is routine operation across Wikimedia projects (Wikipedia's policy likewise provides that promotion-only accounts may be blocked "without warning, usually indefinitely").

Warnings and dialogue are afforded to those inside the "good-faith user" category. My account was classified outside that category — that is what this judgment means.

One more factor, I believe, determined the timing of detection: speed. About ten hours before the block, late at night, I was running the creation of 19 bibliographic items plus a bulk sync of citation edges. During the run, the agent kept reporting that "some items fail to post." (No log of this survives, so this part is the author's recollection.)

My read was "probably a transient server error on Wikidata's side," so I waited a while and resumed posting. The posts went through, and I even thought, "good, the server has recovered." The next morning: the block.

I cannot see what the moderation side actually detected, so I cannot prove the causal link between speed and detection. But as a lesson, I should have leaned toward assuming it. A non-bot account pinned against the throttle ceiling is the most conspicuous thing on the new-item feed and edit-rate patrols.

Repeated rate-limit errors are not "an outage that resolves if you wait." They are an alarm: "this speed and volume is anomalous for this account type." I should have stopped on the spot and reassessed. The errors clearing and the posts going through was not "recovery." I had merely resumed the single most conspicuous behavior on the patrol surface.

In hindsight, there was also a hole in how I approved. I had authorized the creation of 19 items through one single decision — "OK to run the bulk sync?" A judgment I might have paused on item by item had been diluted to one nineteenth by batching.

Where the misunderstanding came from: "Wikidata allows self-registration"

So where did the understanding I opened with — "Wikipedia forbids it, but Wikidata tolerates it" — come from, and where did it turn into an error? I traced four sources.

Wikidata's policy really is looser than Wikipedia's. As we saw at the start, the guideline generally tolerates editing your own item, and that understanding itself is not wrong. Moreover, initiatives like WikiCite, which aggregates scholarly bibliographic data in Wikidata, have a culture of researchers curating publication data themselves. The error was the leap from "individual edits are tolerated" to "therefore any amount accumulated across the account is safe"
Survivorship bias. Deleted self-registrations are unobservable. All you ever see are the cases that survived
Measuring the wrong axis. My validation loop measured only technical compliance — references, constraints, schema — and kept returning PASS. It never measured the axis of "what does this look like to an administrator"
Misreading notability. A DOI is an identifier I can mint through my own actions, but I had misconstrued it as usable evidence of notability (the standard for whether something merits an item)

And here is the fact that stings the most: I half knew. Self-citation on Wikipedia was something I had already considered in advance and rejected as a conflict-of-interest violation. Even so, I redrew the boundary — "the prose encyclopedia is off-limits, but structured data is different" — and went ahead. Draw the boundary in the wrong place, and half-knowing still ends in an incident.

Same-day response: withdraw, purge references, apologize

Once I finished confirming the facts (pulling the deletion log above), the first fork was: appeal the block, or withdraw?

Option	Decision	Reason
Appeal the block	Rejected	Even if it succeeded, this layer would remain revocable at the operators' discretion at any time. What was rejected was not the quality of my execution but the premise itself: manufacturing your own authority data
Withdraw + document the lessons	Adopted	Remove all references to the revoked layer, extract design principles from the failure, and build a structure that never chooses this path again

The first task in withdrawal was purging dead references. My research repos embedded sameAs links to Wikidata item IDs (QIDs) in their machine-readable metadata (JSON-LD knowledge graphs and the like) — and every one of those targets was now a 404.

Leaving identity links pointing at 404s is misinformation aimed at crawlers, and — ironically — an imitation of a classic spam pattern. I removed every QID reference from 9 repositories plus the distribution mirrors within the same day.

Careless removal breaks other things. I used a script with a two-gate check: "still valid as JSON" and "the only change is the removal."

# The gist of purge_wikidata.py (a one-off script for that day, never committed):
# after dropping the target lines, pass two gates before writing
text = "".join(out_lines)
after = json.loads(text)          # gate 1: still valid JSON?
assert after == strip_wd(before)  # gate 2: is the only semantic change the removal?

Two things that actually bit me during the bulk removal

Hardcoded values inside generator scripts: I deleted QIDs from the data files, but a page-generation script had QIDs written directly into it, so regeneration resurrected them. When removing something from generated output, suspect the generator
Missed restatements of counts: statements like "there are 20 ADRs" inside the repos — human review missed 6 files. What caught them was an existing verification script that scans every numeric claim in every file against the file contents as ground truth. It was a case where mechanical scanning beats human thoroughness

Finally, the apology. I made one design decision here: post to the talk page a pure apology containing no request for an unblock. The moment you attach an unblock request, the apology reads as a negotiating tool.

By the time I posted it, I had already deleted the automation skill for Wikidata writes and the API credentials (the companion repository that published those procedures was deleted the same day as well; the research repos themselves remain — what I removed was the means of writing). I could demonstrate "never again" through the state of things, not words.

I consider the administrator's judgment fair. What this incident actually left the community was the administrative labor of reviewing and mass-deleting 109 items.

Even while blocked, you can write to your own talk page

Even under a site-wide editing block, if the block's parameters do not forbid editing your own talk page, you can post to it via the login API (logged-out attempts were rejected by the automatic IP block). I made the post with a one-off script using an existing BotPassword, and deleted the credentials after running it.

Prevention: the record goes into an ADR, the behavior into rules

In parallel with the incident response, I fixed this failure in place at two layers — so it would not end as "let's be careful next time." A record of the judgment (ADR) and rules that change day-to-day behavior.

First, the record layer. An ADR (Architecture Decision Record) is a document format that records a design decision together with its context and the alternatives that were rejected. What it preserves is the reasoning — "why we withdrew, why we will never do this again" — so future sessions do not relitigate the same argument. I wrote one ADR, "entity grounding is done self-sovereign," with four core points:

Design any layer governed by a third party under the assumption it can be revoked — does the strategy survive if that layer vanishes entirely? Never place load-bearing references on a revocable layer
Use only mentions that others created of their own volition; never choose the path of creating them yourself — commissioning someone else to create them is equally out
When revoked, remove dead references promptly — but never rewrite dated historical records — present-tense claims and historical records have different roles
No circumvention — alternate accounts, logged-out editing, or proxying through third parties only converts a disagreement with governance into a permanent adversarial relationship

I also set a principle about the order of response. A "do not climb" sign does not stop someone who is climbing. What stops them is the absence of any means to climb.

So that day's response ran in this order: deleting the means (skill, credentials, repository) → mechanism (writing the lessons into rules) → words (the apology). The strength of a prohibition ranks: absence > mechanism > signage.

However, what actually works as prevention is not the ADR. Records are not guaranteed to be reread. What changes behavior is the instruction files of the executing agent, Claude Code. The agent automatically loads its rule files at the start of every session before doing anything, so a lesson written there never needs to be "remembered." This time I wrote it into three places:

Where	What
Rules shared across all projects (`~/.claude/rules` — auto-loaded every session)	"Repeated rate-limit errors are an alarm. Stop the bulk write and report to the human." "Never pass N writes through one approval. Approvals must state count and scope"
Learned notes (`~/.claude/skills/learned` — consulted during bulk external-write work)	Before executing, ask: "if an administrator seeing this account for the first time opened its history, what would it look like?" (a platform-agnostic self-check)
The project's own instruction file (loaded when working in that repository)	Total ban on self-registration paths (variants — alternate accounts, anonymity, proxies — are equally guilty)

With this, "stop and report when errors repeat" no longer depends on whether I or the agent happens to remember it on some future late night. The lesson lives in the agent's boot sequence, not in human memory.

I decided against enforcement via hooks, by the way. "Bulk writes to external platforms" cannot be detected deterministically, and the cost of false positives halting work outweighs the benefit. Enforce mechanically only where detection is mechanical; keep everything requiring judgment as rules and questions. That dividing line is itself part of the prevention design.

Conclusion: separate the effect from the pathway

The lesson of this incident is not "Wikidata is useless." The effect and the pathway are different things.

The effect: multiple measurements support that being mentioned in authoritative sources like Wikipedia helps AI citation. The hope for Wikidata sits on that same line (though the honest summary of what I found is that direct measurements of Wikidata alone are thin)
The pathway: the pathway of creating those mentions yourself can suffer total loss through account-level governance judgment even when every individual edit is permitted. In my case, it did. So I switched to relying only on earned mentions — ones that others created of their own volition

GEO literature talks about the effect; almost none of it teaches the distinction between pathways. My 109 items were the tuition for that distinction.

Looking back, my error was not one of effort but of classification. Among the places that hold identifiers and proof of existence, there are places where self-registration is legitimate (self-service identifiers like DOI minting, machine-readable metadata in your own repositories) and places where you must wait to be written about by others.

Wikidata — even though each individual edit is tolerated — turned out to be the latter, just like Wikipedia, as a place for registering an entire web centered on yourself. I had classified it as the former, and that was the six-week, 109-item error.

There is one more lesson specific to the era of AI agents. An agent, with no malice at all, effortlessly reaches the speed and volume that platforms treat as hostile.

Individual edits are careful precisely because an agent does them — referenced, constraint-check PASS. With that per-edit compliance intact, the account grew into a 109-item "web pointing at one person" over six weeks, and on the final night it was writing at a speed pinned against the rate limit.

The only human decisions made that night were approving "OK to run the bulk sync?" and choosing to resume after the errors. Nowhere in this incident is there a human who willfully broke the rules.

So what I want you to take away is not "a procedure for safely bulk-writing to external platforms." The conclusion is: don't.

Beyond that, if you operate agents that write to external services, put these five questions in your agent's rule files — so the work of good faith can stop before it grows into a web of promotion:

[ ] If this account's history plus today's writes are viewed together, what do they look like to an administrator seeing them for the first time?
[ ] Are the write targets skewed toward yourself, your works, and references to you?
[ ] Did the approval state the count and scope explicitly (or did one decision dilute N items)?
[ ] Is there a predefined stop condition for when rate-limit errors start appearing?
[ ] Have you enumerated what breaks if this entire layer gets revoked?

And the final question is this: the place you are about to write to — which side is it on? When in doubt, imagine what that platform's administrator would see upon opening your account history.

My footing now rests on layers under my own control. Whether anything gets built on top of that is for others to decide.

What Humans Should Approve Is Intent, Not the Diff — A Decision Table for Agent Approval Gates

shimo4228 — Wed, 29 Jul 2026 03:41:20 +0000

What this article covers: How to catch drift from your intent while it's still cheap to undo (just before commit or publish) without slowing your agent's autonomous execution down. You get a decision table that mechanically determines, from the kind of thing being changed, whether the gate should show a human the full diff text or just an intent summary — plus the one required item that keeps the summary honest: a three-valued Divergence from plan declaration. It ports directly into your own workflow or your team's code review conventions.

As you extend how far an agent runs on its own, you eventually hit this fork.

Lean toward not stopping, and drift from intent only becomes visible after the fact. The thing works, but it's pointed the wrong way. By the time you notice, it has piled up and you can't afford to unwind it
Lean toward stopping, and the human becomes the bottleneck. A review queue forms, and the point of autonomous execution evaporates
So you compromise: "let's at least show the diff." This is the worst of the three. You're paying the cost of stopping, but the volume means nobody reads it, and intent still isn't protected

I suspect the third one is the most widespread. The gate survives as a formality, and in substance you're back to the first case.

Here's what to re-examine. Who is that automation for? If going fast is itself the goal, humans are in the way — but what you actually want is for the thing you intended to get built fast. If so, what to cut is not human involvement itself, but what the human is involved in.

So this article keeps the number of times a human is stopped the same, and moves only the layer they judge at. Whether the artifact is correct becomes the machine's primary responsibility, and the human holds the layer above it: what this is aiming at, and what changes as a result. That changes what the gate shows. That routing is the decision table below.

Assumptions

The examples come from Claude Code 2.1.220 plus my own harness (the set of rules and skills under ~/.claude)
But the deliverable is a way of writing conventions, so it isn't tool-dependent. It works the same in Cursor, in Devin, or in your company's code review policy
Background assumed: you have an AI agent implement things, and you have some approval step before commit

Gates have two axes

When you design an approval gate, there are actually two independent questions.

Axis	Question	What decides it
First axis	When do you stop	Reversibility. An edit you can undo with `git checkout` doesn't stop; publishing externally, minting a DOI, or committing does
Second axis	When you've stopped, what does the human judge	The kind of target (the subject of this article)

Usually only the first axis is settled. "Stop before commit." "Confirm before publishing." — the stop condition is written down, but what to show when you stop isn't. My own harness was like that.

That blank doesn't stay blank. Every time you write a gate, it gets filled in by whatever interpretation is handy at that moment, and the default it fills in with is almost always "show the diff." From the writer's side that looks safest. Show everything and nothing gets missed. The result is the third case from the opening, mass-produced.

The decision table: what you show is decided by the target

This is the deliverable. When a gate stops, what you put in front of the human is determined by the kind of thing being changed. (Below, "deterministic gate" means checks a machine can answer Yes/No on: lint, type checking, tests, secret scan.)

Target	What to show	Why
Behavior-shaping artifacts (`CLAUDE.md` / `AGENTS.md` / rules / skill and agent definitions / public documentation)	Full text	The text itself is the intent. Reading it is already work at the intent layer
Control plane (hooks / permission settings / permission definitions like `--allowedTools` / scheduled task definitions) and artifacts that produce the evidence the checks run on (tests / fixtures / lint config / coverage thresholds / CI definitions / review agent prompts / dependencies)	Full text	These move the gate itself, and the evidence the gate rests on. Fold them into a summary and "a change that loosens the checks" disappears
Implementation code and generated output	Intent summary (what it aims at, and what changes as a result)	Correctness that mechanized checks can decide belongs to deterministic gates and review agents. The diff text and the PASS list don't go on the approval screen
When a deterministic gate FAILs	The detection line itself (mask only the secret's actual value)	This is a state absent from the approved plan, so there's nothing to fold a summary into (the cross-check mechanism is described later). Bypassing it is a decision to disable one check, which puts it in the same class as the control plane

Escalation rule: irreversibility overrides the target category

The table is a principle, and it doesn't decide everything on its own. Even within "implementation code," padding tweaks in a UI and changes to authorization logic, DB migrations, billing, data deletion, or key rotation call for different things to hand a human. So add a one-line override.

For irreversible, high-impact changes, show the full text (or the relevant diff) regardless of target category.

Which is to say the first axis (reversibility) affects not only "when you stop" but also "what you show." The principle is decided by the kind of target, and irreversibility escalates it toward showing full text — that two-stage form is the accurate statement. No third axis is needed.

In one line

Correctness that mechanized checks can decide belongs to the machine; the human holds intent. The one exception is targets where the text is the intent — there, reading the text is itself the intent judgment, so you show the text. Showing full text isn't a backslide into the old way; it's the same principle showing up differently.

There's one more principle you need alongside the decision table.

A review agent is an inspector, not an approver. If the side proposing and the side inspecting come from the same lineage, the inspection inherits the proposer's blind spots wholesale (this article calls that the generator–verifier gap). So approval is composed of two things, "the deterministic gate PASSed" plus "the human's intent judgment," and you never build a path where approval closes on an LLM alone. Making review heavier does not mean approval can be delegated to an LLM.

Written into a conventions file, it looks like this (excerpted from the real thing; internal links and some references are omitted).

## Artifacts are the machine's, intent is the human's

Correctness that mechanized checks can decide belongs to deterministic gates and review agents;
**do not put humans back on artifact inspection** (an assignment of primary responsibility, not a
guarantee — residual risk is caught by the escalation rule). A review agent is **an inspector, not
an approver** (generator–verifier gap). Approval is composed of "the deterministic gate PASSed"
plus "the human's intent judgment"; **never create an approval path that closes on an LLM alone.**

Applying it: what was actually happening in five places

Before I made the decision table, my harness had the second axis blank. What follows is my own environment, but any setup with the same blank will produce the same shape.

The same gate had two competing sources of truth

For the intervention point just before commit, two files said different things. Nowhere was it written which one was canonical (i.e. which one you ultimately follow).

File	What it said
`rules/common/planning.md`	"Verify results check — just before commit" (what to show was unspecified)
`skills/implementation-chain/SKILL.md`	"diff approval just before publish / deposit / commit"

The origin was a single sentence I'd written earlier in a record of a different design decision.

I just do a final check before commit

It doesn't say what the "final check" is of. Downstream, that sentence got read as "diff approval," and settled in wearing the face of a canonical rule.

"A human inspects the artifact" was scattered across five places

Skill	Wording before	Result of applying the decision table
`implementation-chain`	"diff approval just before commit"	Implementation code → changed to intent summary
`readme-writer`	"a human approves the diff and applies it"	Public documentation → rewritten as show full text (same target, different reason)
`release-doi`	"visually confirm the latest commit is newer than the previous release"	A predicate a machine can check → hand it to the machine
`paper-deposit`	Open the generated PDF's first page and eyeball it (paraphrased)	Same → replaced with machine verification of font embedding
`harness-sync`	"`git diff` is the review gate"	What's being synced is rules / skills → leave it as is

Those five were written on different days, for different purposes. All of them landed on the same default with no coordination, so this isn't individual carelessness. Wherever there's a blank, "show the human everything" fills it. Which is why patching the individual wording wouldn't stop it — I needed one convention covering the second axis itself.

And that rightmost column shows where the decision table earns its keep. Grep and all five look identical, but wiping them out uniformly would take out gates you need. harness-sync did end up staying: what it syncs is rules and skills, so reading the text is itself the intent judgment.

Reinforcement 1: the control plane and "artifacts that produce the evidence"

In the first version of the decision table, I put hooks and permissions on the "implementation code and config" side. A review from an external model (Codex CLI) hit exactly there as its top-priority finding.

Codex's finding

> The central target boundary permits executable harness controls to be hidden behind proposer-authored summaries, and the new rule conflicts with the existing security false-positive workflow.
> - [P1] Classify executable harness controls as behavior-shaping — …

Suppose the agent disabled the pre-commit secret scan hook. It can follow the convention to the letter and still summarize it like this.

Intent summary: Tidied up the pre-commit check configuration and strengthened the checks

The control plane "moves the gate itself," so folding it into a summary turns a change that loosens the checks into a well-meaning sentence.

And hooks and permissions aren't the only way to weaken things. Rewriting the tests so they pass against your implementation is the quietest path of all. Here the agent doesn't even need to lie.

Intent summary: Fixed the implementation; all tests PASS

That's exactly true. The one thing that never surfaces in the summary is that what got rewritten was the tests. Not just "the tests passed" but "what counts as passing" is under review. Fixtures, coverage thresholds, lint config, CI definitions, review agent prompts, and dependencies all have the same property.

So how should you present it

The answer is: don't choose summary-or-text per commit. If one commit contains both implementation and evidence, show the summary, and alongside it show the full text only for the parts that are evidence.

Intent summary: Changed the login-failure retry limit from 5 to 3 (plan step 2)

⚠ This change touches the evidence the checks rest on — review the text below

  tests/test_auth.py:41
  -    assert client.retry_limit == 5
  +    assert client.retry_limit == 3

The human reads only those last two lines. The implementation-side diff isn't shown. All you have to look at is whether the tests were loosened to match the implementation, so volume can't beat you. The example above corresponds to a change decided in the plan, so it goes through. If the plan said nothing about retries and only the test threshold dropped, that's where you stop.

Which files are on the evidence side can usually be picked up from the path.

# Extract candidates for "produces the evidence" from changed files, to show as full text
git diff --cached --name-only | grep -E \
  '^(tests?/|\.github/workflows/|\.claude/|hooks/)|(_test\.|\.test\.|pyproject\.toml|package\.json|ruff\.toml|codecov\.yml)'

This is not a complete classifier; it's a conservative candidate extractor. It misses tests embedded in ordinary code, custom directories, lockfiles, and inline snapshots. Add the evidence-side files you know about in your own repo and grow it.

In my environment, this extraction has moved out of the convention's prose and into a PreToolUse hook (evidence-file-notice.sh). When it detects a git commit, it scans the staged files against a pattern like the one above (the real list is a bit broader — it also covers spec/, conftest.py, pre-commit config, and so on), and if anything matches, it injects the instruction itself into the approval flow: show the human the diff of these files. The canonical home of the evidence-file enumeration is no longer the convention's text but that one regex in the hook; the convention keeps a single pointer line saying detection is canonical there.

There are two reasons for this restructuring.

A convention that exists only in a document is followed only probabilistically. Write "evidence-side files get their text shown" into a convention and there's no guarantee the agent recalls it every time. "Which files are evidence-side," though, is a structural property decided purely by the shape of the path — a machine fires on it 100% of the time
Keep the enumeration in two places — convention and hook — and they will drift apart. Centralize it on the side that fires, and let the convention hold only the principle

Semantic judgments like the decision table stay in the document; anything mechanically decidable gets lowered into a gate as soon as you spot it — this convention itself has been growing by exactly that process.

The control plane (hooks / permissions / scheduled tasks) works the same way: summary plus the full text of the relevant files. Anything producing the evidence a judgment rests on gets treated the same as the judgment itself.

Reinforcement 2: on FAIL, show the detection line

The second thing that same review found. Push "no machine-check PASS list and no diff text goes to the human" all the way, and you also delete the path where a human judges a secret scan false positive.

In my harness, when the pre-commit secret scan detects something, there's a workflow to let it through with SECRET_SCAN_BYPASS=1. The judgment is "this is a dummy key in a test, so it's fine."

The reason this goes to a human isn't that a machine can't tell the difference. It's that even if the machine or the review agent happens to be right, this is the kind of decision whose owner must be a human. A bypass is the act of disabling one check on the spot. That has the same nature as moving the control plane, so for the same reason as Reinforcement 1, you need the full text (i.e. the detection line).

There's a second reason: a FAIL is a state absent from the approved plan. An intent summary is meant to be read against the plan (see below), so a state that isn't in the plan has nothing to fold into.

**FAIL is the exception** — a deterministic gate's FAIL presents **the detection line itself**.
Mask the secret's actual value; the owner of the false-positive call (`*_BYPASS`) is the human

PASS needs care. PASS only means "the checks you configured accepted this." Whether it was implemented according to the plan is not something PASS can tell you. That's exactly why you need a separate intent summary — you leave out the PASS list because you're looking at the summary instead.

"Not shown" is not "not kept"

This gets misread easily, so let me be explicit. Leaving out the PASS list means moving it off the human's approval screen, not throwing check results away. Delete the trail and you lose any way to trace "what was passing back then" later. The idea in this article isn't to delete information; it's to move unneeded information off the human's judgment surface.

I originally had this distinction written into the convention as its own clause; the current version drops it. Claude Code already persists the transcript and tool results in machine-readable form at all times, so writing it down merely restated a default the runtime already guarantees. If your execution environment doesn't guarantee log retention, keep it as an explicit clause. "Don't write into the convention what the substrate already guarantees" is itself one of the judgment calls you'll make when porting this.

One more thing: on a true positive, dumping the detection line as-is duplicates the secret's actual value into the conversation, the approval screen, and the logs. That would spread the very leak the gate exists to prevent, so show the file, the rule name, and the surrounding context, and mask only the value. What a human needs to judge a false positive is not the value itself, but where it was and which rule it hit.

Reinforcement 3: don't let the intent summary be free-form

This is a prerequisite for putting the decision table into practice.

If you stop at "implementation code gets an intent summary," the human ends up judging by reading a self-report written by the proposer itself. That just relocates the gap that was at the artifact layer. What they read changed from a diff to an essay; the author is the same.

The countermeasure is to fix what the summary is cross-checked against to something human-originated. My harness has two intervention points.

Plan check — when what to do has been settled (this is where the human approves)
Intent check — just before commit

And the summary in 2 is presented cross-checked against the plan the human approved in 1. Because the referent is a human-approved object, the loop can't close on self-reporting alone.

Only one item is required: the `Divergence from plan` declaration

But "write it against the plan" still leaves room to write it conveniently. In particular, when something new is discovered during implementation and the work naturally diverges from the plan, that divergence quietly disappears from the summary.

My first countermeasure was to make the intent summary itself a fixed form: five mandatory headings — approved intent, what changed, divergence from plan, impact on users and operations, evidence-side changes. But the day after that form went into the convention, an audit re-examining the whole convention clause by clause forced me to re-sort those five. Only one of the fields was actually preventing the worst case — the silent disappearance of divergence. This is all that's required now.

Intent summary: Changed the login-failure retry limit from 5 to 3 (plan step 2)
Divergence from plan: None

Divergence from plan takes one of three values: None / Yes / Needs re-approval. The plan changing because of something you discovered mid-implementation isn't bad in itself. What's dangerous is the fact that it changed disappearing from the summary.

And making this declaration required turns omission into falsehood. In free-form text, a summary that doesn't mention the divergence isn't lying — it just didn't bring it up. With the declaration required, writing None when there is divergence is a falsehood, and writing Yes points the human's eyes exactly there. It closes the omission escape route structurally, without leaning on the writer's honesty. This is the one thing only formal enforcement can protect.

So why drop the other four fields? Because they were the summary's shape, not its defense. What was intended, what changed, what's affected — summarizing without dropping those is exactly the calibration a model should bring to any summary: lead with the outcome, report results faithfully, match the response to the question. Impose a fixed form on the parts calibration already covers, and you're forcing five headings onto a one-line config change — the approval screen turns into template skimming.

That would reproduce, on the summary side, the third case I called the worst at the start: a gate that survives as a form nobody reads.

(As for the "evidence-side changes" field — the hook from Reinforcement 1 covers it structurally, so there was never a need to make a human write it.)

Anthropic's context-engineering guidance for the Claude 5 generation (The new rules of context engineering for Claude 5 models) states this as a general rule: shift from binding with rules to delegating to judgment — except in the regions where the worst case is unacceptable, which stay explicitly bound. The 5→1 reduction is that rule applied. Delegate the summary's shape to the model's calibration; bind exactly one point with form — the disappearance of divergence, the worst case you can't accept.

The whole point of this gate convention was to concentrate the human's cognitive budget on judging intent. If the convention itself burns the writer's and reader's attention on enforcing a form, that defeats the purpose.

One more practical upside: the three values are an enumeration, machine-readable, so "does the declaration exist at all" can itself be lowered into a hook check later. Same as Reinforcement 1 — build the convention so it has an exit ramp down into a deterministic gate.

Cap the number of stops too: one gate per unit of work

Paired with what to show, fix one rule about how often. The intent check happens once, at the completion point of a unit of work. If a commit, a push, a publish, and the accompanying doc updates all arrive together, bundle them into one decision with the count and scope explicitly enumerated. Ask for approval at every intermediate phase or intermediate commit and the approvals themselves become noise — the gate degrades into formality. There are exactly two exceptions: a deterministic gate FAILing, and Divergence from plan: Needs re-approval.

But "bundling is allowed" is not "implicit is allowed." If one decision covers N items, say N at approval time. The moment an approval granted for one item gets silently reused for N, it has stopped being a gate.

Raise the granularity and the bottleneck moves

This cross-check bites hardest when the agent is good. Things going well for a long stretch is exactly the condition under which the intent summary becomes a formality. If you find yourself skimming past the same "no divergence" every time, raise the granularity of the plan you're checking against.

But that isn't free. The finer the plan, the further the bottleneck moves from just-before-commit to plan approval. Reserve the finer granularity for the things you can't take back when they drift (external publishing, data migration, permission changes). You don't get to erase the fork entirely — the accurate statement is that you get to choose where you pay.

On the conventions side, I also renamed the second intervention point from "Verify results check" to "intent check." Leave the name as "Verify results" and it drags you back into making humans read a list of machine-check results.

Porting it into your own conventions: three steps

Enumerate your existing approval steps and check what each one says to present. Grep for wording like "approve the diff," "visually confirm," "review and apply"
Put the decision table and the escalation rule into a single conventions file, and have each place you found point at it. If you only patch the individual wording, the next skill or document you write will fill the blank the same way again
Build the Divergence from plan declaration (None / Yes / Needs re-approval) into the summary as a required item. The summary as a whole doesn't need a fixed form. Without this, you can install the decision table and still be verifying the proposer's own self-report

In step 1's grep, watch out for missing word stems. I searched for eyeballed and missed eyeball (the uninflected stem), so one place survived all the way to the external model's review. Search verbs by stem rather than inflected form, or run grep -i with several patterns.
For reference, in my environment this change (one new conventions file plus re-pointing references across 13 related files) came to 14 files and 217 added lines. The decision table itself is short; what made the difference was re-wiring everything that referenced it.

The convention's body has since been cut in half (the text auto-loaded into every session's context went from 265 to 125 words). What remains is only the principles — the decision table, the escalation rule, the divergence declaration. The "why" moved into the decision record (ADR), and the evidence-file enumeration into the hook. Left alone, a convention fattens on rationale and examples until nobody reads it. Principles in the document, why in the record, anything machine-decidable in a gate — splitting the storage that way is what keeps it maintainable.

The limits of "the machine holds it"

Let me be straight about this. Build, types, lint, tests, and secret scan tell you only about the properties they cover. Holes in authorization logic, concurrency races, irreversible side effects of a migration, requirements that were never implemented at all — all of that sails through as PASS. And the review agent, being from the same lineage as the proposer, shares its blind spots (the generator–verifier gap above).

So the third row of the decision table isn't "the machine guarantees correctness" — it's an operational tradeoff that puts primary responsibility for artifact inspection on the machine. Residual risk does not go to zero.

Which is exactly why you need the escalation rule. Keep the human on the artifact side only in the regions you can't undo — data migration, permissions and billing, external publishing, deletion. Put another way: if a human is reading diffs of reversible implementation code, that isn't a response to residual risk, it's just inertia.

You also can't reduce the whole convention to a deterministic lint. Extracting the evidence-side files (the hook in Reinforcement 1) and checking that the divergence declaration exists can be mechanized, but deciding "is this a behavior-shaping artifact" is semantic, and for edge cases like generated documents and config files you have to think each time. The decision table illustrates by enumeration; it doesn't exhaust the space.

Back to the opening question

The further you extend autonomous execution, the more drift from intent becomes visible only after the fact. But put the human back on review and the point of autonomy disappears. This looks like a fork only while you're counting human involvement by number of stops.

Count by content instead and it stops being a fork. Keep the number of stops the same, and move what they read at the stop from the artifact to the intent. What you cut isn't human involvement — it's the time a human spends re-checking the correctness of artifacts. The machine was always better at that, and when a human does it, volume wins and it becomes a formality.

To the question of who the automation is for, this is the answer. Not to go fast, but so that what you intended gets built fast. If so, there's only one thing the human should hold to the end. Intent. The decision table was a tool for taking everything else away from them.

When you're talking about reducing human review, what's left isn't the work you forgot to cut. It's the work you can't cut.

Opus 5 Changed How Rules Should Be Written — Audit Yours

shimo4228 — Tue, 28 Jul 2026 11:38:58 +0000

What this article covers: Anthropic's official guidance for the Claude 5 generation (Opus 5 / Fable 5) is "fewer rules, trust the model's judgment." This is a procedure for auditing the custom rules (CLAUDE.md / rules files) you accumulated for older model generations against what the product itself loads, and deciding for each rule: keep, fix, or retire.

If you've been using Claude Code for a while, you probably share these worries:

You wrote CLAUDE.md files and rules to compensate for older models' weaknesses, and you have no idea whether they still help with Claude 5
The official guidance now says "cut your rules," but you can't tell which of your rules are the ones to cut
Deleting everything at once and watching behavior degrade is scary, so you don't touch any of it

On 2026-07-25 I ran this audit on my own ~/.claude harness (my set of custom configuration: CLAUDE.md, rules files, agent definitions) and cut the resident rules — the part auto-loaded into every session — from 5,789 down to 2,463 words (measurement details at the end). This article documents the cross-checking procedure I used, in a reproducible form. The takeaway is not the reduction itself but the decision procedure: on what evidence, which rules, and how to dispose of them.

Background: the official guidance on rule-writing changed

On 2026-07-24 (the day Opus 5 launched), Anthropic published The new rules of context engineering for Claude 5 generation models. Two points matter here.

First, Anthropic itself deleted more than 80% of Claude Code's system prompt:

We removed over 80% of Claude Code's system prompt for models like Claude Opus 5 and Claude Fable 5 with no measurable loss on our coding evaluations.

Second, the post spells out the shift in context engineering as six Then → Now pairs.

Then (older generations)	Now (Claude 5 generation)
Give rules	Trust judgment
Give examples	Design interfaces
Front-load everything	Progressive disclosure
Repeat yourself	Simple tool descriptions
Write memory into CLAUDE.md	Auto-memory
Sparse specs	Rich reference material

The post names a concrete harm of over-instruction: conflicting instructions. System prompt, skills, and user instructions collide, so that "leave documentation as appropriate" and "DO NOT add comments" coexist in the same request.

In other words, the reason to cut rules is not just token savings. The problem is that your custom rules conflict with the product's own instructions, and you are forcing the model to resolve the contradiction. From here, the procedure is about finding which of your rules are the conflicting ones.

Premise: the ground truth is the system prompt and tool descriptions

What actually reaches the model in the same context as your custom rules is the system prompt and the tool descriptions. That is the ground truth to audit against, and the first step is simply to read it. Official docs and blog posts state recommendations, but they are not loaded at inference time.

The meaning of a mismatch changes with this distinction, so let's name the two layers.

Layer	Contents	Loaded at inference time?	Name of the mismatch
Runtime layer	System prompt + tool descriptions	Yes (loaded into context alongside your rules)	Conflict (the model is forced to resolve a contradiction)
Guidance layer	Official docs and blog posts	No	Drift (you've merely diverged from official recommendations)

A mismatch only qualifies as a conflict when both sides are in the same context (a tool description is only loaded in sessions where that tool is available).

All examples below come from my own ~/.claude harness (Claude Code 2.1.220, audited 2026-07-25 to 07-26).

Step 1: capture the current runtime layer from a live session

The first task is to know exactly what your custom rules are sharing context with. There is a trap here: the runtime layer is injected from outside your config repository too, so no amount of reading ~/.claude shows you the whole picture.

A concrete example. My session's system prompt contained this resident line:

"Do not call the AgentTool unless the user requested it"

Yet grepping ~/.claude finds this string in no file at all (I haven't identified the source; I suspect the harness itself or a plugin, but couldn't confirm). If you only audit config files, you miss this class of instruction.

So capture from a live session instead. There are two techniques.

For tool descriptions, get the real thing. Ask Claude Code to print the target tool's description:

Quote the description of the EnterPlanMode tool verbatim. Do not summarize.

For the system prompt, have the session quote itself. This is self-reported, so say "verbatim" explicitly to prevent summarizing:

From the system prompt currently loaded, quote verbatim the instructions
about plan mode / commit messages / scope handling.

The trick is to ask per theme. "Print everything" is too long and summaries creep in. Build the theme list from your own rules: open every custom rules file (rules / CLAUDE.md) first, assign each instruction to a theme (planning, commits, review, scope, ...), then query the runtime layer theme by theme. That way you at least eliminate audit gaps on your side of the comparison.

Two caveats. First, gaps in the other direction — runtime instructions you don't know exist — can't be fully caught by this procedure, so zero detections means "this question found nothing," not "no conflicts." Second, both capture techniques are model-mediated self-reports, with no guarantee of exactly matching the actual input. Treat the captures as a screening pass; before acting on a rule's disposition (retire / invert), confirm the same wording reproduces in a separate session.

Step 2: cross-check against your rules and sort into 3 categories

Cross-check the captured runtime layer against your custom rules and sort each mismatch into three categories.

Category	Meaning	Detected in my harness
Conflict	An instruction or claim that contradicts the runtime layer is loaded at the same time	2
Redundant	The runtime layer already says nearly the same thing	1
Drift	Diverged from guidance-layer (official docs) recommendations	3

Conflict example 1: a no-plan-mode rule

My planning.md contained this, written as a countermeasure to older generations' habit of not planning:

On direct implementation instructions, execute immediately. Do not enter plan mode.

Meanwhile, the EnterPlanMode tool description loaded in the same context (Claude Code 2.1.220) says:

"Prefer using EnterPlanMode for implementation tasks unless they're simple."
"If unsure whether to use it, err on the side of planning"

Two instructions pointing in exactly opposite directions were loaded into the same context, and the model was silently picking one every time.

Conflict example 2: a rule referencing a ghost setting

The second one was a nastier pattern. Background: by default, Claude Code appends a Co-Authored-By: Claude ... trailer to commit messages — the attribution that shows Claude as a co-author on GitHub. On whether to emit that attribution, my git-workflow.md said:

Note: Attribution disabled globally via ~/.claude/settings.json.

But checking the actual settings.json, the relevant key (includeCoAuthoredBy) did not exist:

python3 -c "import json; print('\n'.join(sorted(json.load(open('settings.json')).keys())))"
# includeCoAuthoredBy is not in the output

Whether I deleted the setting at some point or the note was wrong from day one, there's no way to tell anymore. What I can say for certain: a rule cannot detect that its own justification has vanished. Code errors out when its referent disappears; a rule keeps asserting "disabled" while sitting in context, contradicting the runtime layer's instruction to append the Co-Authored-By trailer to commit messages.

Strictly speaking, this line is not an instruction ("don't add the trailer") but a (false) statement of fact ("already disabled"). But since it kept riding along in the same context as information contradicting a runtime-layer instruction, I counted it on the conflict side.

How the unresolved conflict was actually being resolved (a model-dependent observation, with limits)

Digging through session logs from the period when this ghost-setting conflict sat unresolved, I found an interesting split. Same Claude Code version (2.1.220), same rules: two Opus 5 sessions added the trailer to commits, one Fable 5 session did not. Within each session, behavior was 100% consistent.

So when you leave contradictory instructions in place, how they get resolved is up to the model — and when the model changes, the behavior can change too.

The limits, stated plainly: this is a natural observation, not a controlled comparison (5 commits with differing tasks and user instructions). The most I can claim is "likely model-dependent." Also, the observed difference — attribution — is harmless; whether conflicts with real consequences split the same way is unverified.

Redundancy example: a scope-adherence rule

Separate from conflicts, some rules had simply become unnecessary to say. My custom rule "when a scope is specified, adhere to it strictly" exists almost verbatim in the Claude 5 generation system prompt:

"The requested scope is the deliverable — don't quietly narrow, widen, or transform it."

No collision, so the harm is small — but I was paying resident tokens to repeat what the product already says. It was exactly the official post's "Repeat yourself → Simple tool descriptions" shift.

Drift example: a confidence threshold for review

I detected three drift items. I'll cover one representative case here; the other two — the handling of pre-commit verification steps and the rule splitting review into a separate agent — get different verdicts, so they're covered in Step 3 and the pitfalls section respectively.

The representative case: my review-agent definition said "only report findings you are at least 80% confident in" (a number unrelated to the 80% system prompt reduction above). The official Opus 5 prompting guide warns against exactly this kind of suppression instruction, using it as an example (it does not mention the number 80% specifically):

If your review prompt says "only report high-severity issues" or "be conservative," the model may follow that instruction literally and report less; ask it to report everything and filter in a separate pass instead.

This is not a load-time conflict, but with the official docs explicitly saying the model may follow this kind of instruction literally and under-report, leaving it in place means dropped findings. I haven't verified how much damage it did on older generations, but since it's the exact pattern the current generation's docs warn against by example, I classified it as drift.

Step 3: don't treat "conflict = bad" — run each item through a decision frame

Once the three-way sort is done, decide dispositions. The important part: do not mechanically mark conflicts and drift for deletion. Some of your custom rules were written precisely to override product defaults on purpose. "Points the opposite way" alone cannot distinguish an accident from a deliberate choice.

I judged each item on four axes.

Axis	Question
Intent	Did you write it to override the product default, or was it simply not conflicting at the time?
Evidence	Is there a record of why it was written — an ADR, an incident note?
Freshness	Does the product behavior it assumed (an old generation's weakness, etc.) still hold?
Expiry condition	Have you defined what would trigger revisiting this rule?

Three example dispositions.

Retire (freshness expired): The no-plan-mode rule (conflict example 1) was a countermeasure to older generations' "start running without planning" weakness. In the Claude 5 generation the product itself now recommends planning; the premise is gone. There was no reason to keep it as an intentional override, so I retired it.

Retire (conflict + redundancy): Beyond the ghost-setting line (conflict example 2), git-workflow.md contained nothing but content the model and the runtime layer already cover — commit message format, how to write PRs. Fix the conflicting line and the rest is still redundant. So the whole file had no reason to exist, and I retired it.

Keep (intent and evidence alive): On the other hand, I kept my rule "run review in a separate agent process from the implementer." The official guide contains the line "do not use subagents to verify or double-check your own work," which at first glance collides with this rule.

But read in context, this is not a rejection of division-of-labor review. That line is one phrase in an example prompt developers can add to control subagent spawning, in a section about cost containment. The official premise is that Opus 5 verifies its own work without being told — explicit verification instructions just cause over-verification and should be cut — and that the new generation, which also delegates more, would otherwise reflexively spawn "just in case" double-check subagents and burn cost. That's the brake being illustrated. The same doc also positively cites the writer–verifier pattern — separating the writer from the verifier — as a strength of Opus 5.

My rule has two tiers. The base: never let review happen in the same context as the implementation (same-context review inherits the implementer's blind spots). On top of that, for important changes, a different model (in my case, Codex CLI) does the review, decorrelating the blind spots. The latter was an easy keep — cross-model review is a capability Claude Code itself structurally cannot provide no matter how much it evolves, so there is nothing for the product to absorb it into. The design decision is recorded in an ADR.

The former (separate-process review on the same model) may still overlap with the "added verification passes" the official guidance wants to suppress. This is the interpretive part: if Opus 5's self-verification turns out to solve even the blind-spot-inheritance problem, the base tier may become unnecessary. It stays — as a keep with an expiry condition: revisit when the premise changes.

Running this decision frame taught me one thing: rules with recorded evidence (ADRs, incident notes) are fast to judge, and the judgments are more confident. Having a record doesn't prevent retirement (the plan-mode rule had a known rationale and was retired anyway, on expired freshness). What the record changes is not the outcome but the cost of deciding. For rules with no recorded rationale, even the question "is it safe to delete?" turns into git log archaeology.

Pitfall: mechanically applying the official advice misdiagnoses

The official guide says verification steps added by legacy harness scaffolding are also candidates for removal. Applying that mechanically will misdiagnose.

My custom rules had an 8-item pre-commit Verify step (build / type check / lint / tests / secret scan / dependency audit / doc sync check / git status check). Taken literally, "cut verification steps" makes all eight look like deletion targets.

But what the official guidance objects to is instructions that increase the model's self-verification. Most of the eight items are deterministic command runs — builds, tests — that add zero model judgment. The axis is:

The verification is done by a machine (command, hook) → fine to keep
The verification is done by the model, on its own judgment → the official guidance's target; consider cutting

The other pitfall: cases where deletion isn't enough and you need an inversion. If you simply delete the "only report at ≥80% confidence" line, the section it lived in (a noise-suppression framework) survives and keeps pushing in the suppressive direction. I rewrote the section to: "Do not suppress by confidence. Report everything; the caller filters in a separate pass." An instruction whose correct direction has flipped must be rewritten in the opposite direction, not merely removed.

Results and numbers

The center of gravity is the procedure, but for reference, my harness before and after (measured with wc -w, resident total of CLAUDE.md + rules):

Point in time	Resident words	Files
Before the audit	5,789	20
After (as of 2026-07-26)	2,463	14

Note that I am not claiming "resolving the conflicts improved performance." My environment has no instrument for measuring the cost of contradiction-resolution; the quantitative case rests on Anthropic's measurement quoted at the top (80% deletion, no degradation). What this article demonstrates is not the effect but real examples of config drift, and a procedure for finding and disposing of it.

Summary: what you can do next

Capture the runtime layer (system prompt + tool descriptions) from a live session. Reading your config repository alone doesn't show the whole picture
Cross-check against your custom rules; sort into conflict / redundant / drift
Don't treat "conflict = bad" — judge keep / invert / retire on the four axes of intent, evidence, freshness, and expiry condition
Apply the official reduction advice with the distinction between model self-verification and machine verification

And one lesson to add from this round: when you write a rule, record its rationale and its expiry condition. I'll confess I wasn't doing this systematically either. Rules with an ADR (design decision record) took minutes to judge; rules without one turned into git log archaeology; and for the ghost setting, I couldn't even pin down when it disappeared. A rule cannot detect that its own justification has vanished — so whether you'll be able to judge your rules at the next generation change will be decided by the notes you leave now.

Postscript (2026-07-26)

The lessons from this article have already flowed back into my harness.

Every resident rule file now carries rationale: and review-when: metadata, and my rules-audit skill rules-stocktake reads these fields during audits
The procedure in this article (collect the runtime layer → cross-check → judge with the decision frame) has been generalized into a skill called generation-audit, published as a standalone repo, so it can be re-run as-is at the next model generation change

claude-security Beta: Cost, Output, Accuracy — 189 Agents, 2 Hours, Measured

shimo4228 — Mon, 27 Jul 2026 13:00:04 +0000

What this article covers: A measured run of Anthropic's official security-scanning plugin claude-security (beta). What the tool does, how long it takes, what output it returns, and how far you can trust it — backed by the raw data from the run's own artifacts.

You may know the official plugin exists and still not have tried it, for reasons like these:

The cost is unreadable. It warns you at startup that it "may take a while and use a significant number of tokens," but never says how much
The output quality is unknown. How is this different from existing static analysis, and does an LLM that only reads code produce findings you can act on?
It's unclear what to do afterward. If dozens of findings come back, where do you start?

On 2026-07-25 I installed v0.10.0 and scanned my entire ~/.claude configuration set (hooks, skills, agents, permission settings). 189 subagents ran for two hours and returned 20 findings. This article is that measurement.

This plugin is in beta as of 2026-07 (the version number is 0.10.0). Every number and behavior below is measured on v0.10.0 and may change.

Prerequisites

Item	Requirement
Claude Code	Measured on 2.1.220
Plugin	`claude-security` v0.10.0 (by Anthropic, marketplace `claude-plugins-official`)
Python	python3 3.9+ (used by the report-generation scripts)
git	Change scans require a git checkout. Whole-repository scans work without git
Permission mode	auto mode recommended (the plugin says so itself)

This plugin is the "runs inside your session" version of the hosted Claude Security product. It starts no separate process and no daemon; it runs entirely inside your Claude Code session.

What the tool is, and what happens when you run it

From install to launch

# Run these inside a Claude Code session
/plugin install claude-security@claude-plugins-official
/reload-plugins

If it reports that the marketplace was not found, run /plugin marketplace add anthropics/claude-plugins-official first and retry.

After installing, /claude-security opens a menu offering three jobs.

Job	Target	Duration
Scan codebase	The whole repository, or a scoped part of it	Depends on size (2 hours 1 minute in my run)
Scan changes	A branch diff, a pull request diff, or a single commit	Minutes for a small diff
Suggest patches	Turns an existing report's findings into patch files	Depends on the finding count

You can also skip the menu and name the job in the arguments.

/claude-security scan my branch's changes
/claude-security --base main
/claude-security 3cb30d2          # a hex string of 7+ chars is read as a commit SHA

Get the confirmation prompt out of the way

Choosing a whole-repository scan always triggers this confirmation.

This scan may take a while and may use a significant number of tokens. You will need to leave Claude Code open while the scan completes. Are you sure you want to continue?

Accept the cost up front and you go straight in. The check is not a literal string match — it asks whether your request already reads as accepting the time and token cost — so include a sentence that conveys that.

/claude-security scan this repository — whole codebase.
I understand it may take a while and use a significant number of tokens.

You do need to leave Claude Code open while the scan runs. No interaction is required, so you can walk away.

Effort changes how much it searches, not how hard it verifies

--effort has four tiers. The distinctive design choice here: the verification panel is fixed at three voters at every tier.

effort	What it does
`low`	One researcher over the whole repository. No inventory, threat model, or breadth sweep
`medium` (default)	The full workflow: inventory → threat model → one researcher per component × category → one sweep → three-voter panel
`high`	Like `medium`, but the component cap rises to 24, two researchers per cell, two sweeps
`max`	Like `high`, plus an adversarial phase: marginal keeps are re-panelled and every survivor faces a red team

So dropping the tier does not mean "wave things through" — it means "search a narrower area." The documentation says as much, noting that the report's confidence figures are calibrated against that three-voter panel. I ran medium.

A six-stage pipeline runs

flowchart LR
    A["Inventory<br/>partition the tree"] --> B["Threat model<br/>one per component"]
    B --> C["Research<br/>component × category"]
    C --> D["Sweep<br/>what the matrix missed"]
    D --> E["Panel<br/>3-lens adversarial check"]
    E -. max only .-> F["Adversarial<br/>repanel + red team"]

Here is what that came to against ~/.claude.

Stage	Agents	Detail
Inventory	1	Partitioned the tree into 11 components
Threat model	11	One per component
Research + Sweep	42	One per component × category cell; the sweep is counted in this same pool
Panel	135	45 candidates × 3 votes
Total	189	2 hours 1 minute

This breakdown and the elapsed time are measured, not estimated. You can reproduce the same tally on your own run: the workflow's journal.jsonl records each agent's return value, so you can classify agents by the shape of what they returned.

# Run inside ~/.claude/projects/<project>/<session-id>/subagents/workflows/wf_*/
python3 - <<'PY'
import json, collections
sig = collections.Counter()
with open("journal.jsonl", encoding="utf-8", errors="replace") as f:
    for line in f:
        try: d = json.loads(line)
        except ValueError: continue
        if d.get("type") != "result": continue
        r = d.get("result")
        sig[tuple(sorted(r))[:5] if isinstance(r, dict) else ("?",)] += 1
for k, v in sig.most_common(): print(v, k)
PY

135 ('reasoning', 'verdict')                                          <- panel votes
 42 ('findings',)                                                     <- researchers
 11 ('assumptions', 'entryPoints', 'hotFiles', 'sinks', 'trustBoundaries')  <- threat models
  1 ('components', 'securityScanSkippedComponents')                    <- inventory

The elapsed time comes from the same logs. The first record is 02:28:12Z and the last is 04:29:16Z. The largest gap in between is 47 seconds, so something is running for the entire two hours.

But this does not mean you are tied up for two hours. You must leave the Claude Code window open; you do not have to do anything. The plugin's internal documentation has a line that assumes exactly this.

Users desire to leave the session unattended very soon after kicking off a scan, around a minute of wall-clock time.

That matched my experience — I spent the interval in a separate session doing unrelated work.

Progress is visible stage by stage in /workflows while it runs.

What the output looked like

When it finishes, three artifacts land in a timestamped directory.

CLAUDE-SECURITY-20260725-022756/
├── CLAUDE-SECURITY-RESULTS.md        # the human-readable report
├── CLAUDE-SECURITY-RESULTS.jsonl     # machine-readable (one finding per line)
└── CLAUDE-SECURITY-REVISION-*.json   # the revision and settings stamp for the run

The result was 20 findings (5 HIGH / 15 MEDIUM). 111 raw candidates became 83 after dedup; of the 45 that reached the panel, 25 were rejected, leaving these 20.

A rejection rate above half can be read as evidence that the panel is working, or as evidence that the research stage over-generates candidates and the panel cleans up after it. The report contains nothing that decides between those readings.

What does decide something is the number on the way out. I acted on all 20 findings, and rejected none as a false positive. I confirmed each one reproduced before fixing it (three were addressed only partially — I did not add OS-level isolation). This is one repository and one run, but at least in this case, a tool that only reads and reasons did not produce a pile of off-target noise.

Three things about the output quality stood out.

1. It reads configuration and natural-language instruction files

The inventory partitioned the tree like this.

hooks / scripts / scheduled-tasks / skills-executable-scripts /
skills-instructions / agents / rules / templates / docs /
notes-and-metrics / tests

rules and skills-instructions — meaning the instruction files written in plain natural language were treated as their own audit components.

The 20 findings broke down by location like this.

hooks: 10 / skills: 5 / settings.json: 3 / scheduled-tasks: 1 / agents: 1

The one under agents was about the trust-ordering instruction itself, written in an agent definition .md. One of my agents treated raw session logs as "machine records, hard to alter" and placed them at the top of its trust order.

But those logs hold the verbatim bodies of external pages fetched in the past. The file may be hard to tamper with; its contents are not trustworthy.

The one under scheduled-tasks was similar in character: a README said a job was "not wired up," when it was in fact registered and running weekly. It flags documentation that contradicts reality as something that makes reviewers misjudge risk.

A tool that searches for patterns with regular expressions produces neither of these.

2. It folds the count down instead of inflating it

The report opens with this.

Read the count with one caveat: the 20 are not 20 distinct defects.

It then names three clusters itself.

Cluster	Findings	What it actually is
Permission allowlist	3	The same line of `settings.json` seen through three lenses
Log-guard leaks	8	Three hooks protecting the same asset, all failing open because of one shared constant
Child-agent execution	4	Two paths where a skill launching a child Claude executed generated strings directly

Those three clusters account for 15; the remaining 5 stood alone. The report describes the whole as "closer to eight problems" when counted by root cause.

It goes further and names a single repair as the highest-leverage one in the eight-finding cluster — fix that one spot and seven close at once. In a field where finding count tends to be treated as a performance metric, the report folds its own count down for you.

3. The machine-readable side lets you sort by what you care about

The report body is long, so start with the jsonl for the shape of it.

# Run in the parent directory of CLAUDE-SECURITY-<timestamp>/
python3 - <<'PY'
import json, collections, glob
path = glob.glob("CLAUDE-SECURITY-*/CLAUDE-SECURITY-RESULTS.jsonl")[0]
with open(path) as f:
    rows = [json.loads(line) for line in f]
print("count:", len(rows), collections.Counter(r["severity"] for r in rows))
print("where:", collections.Counter(r["file"].split("/")[0] for r in rows))
PY

count: 20 Counter({'MEDIUM': 15, 'HIGH': 5})
where: Counter({'hooks': 10, 'skills': 5, 'settings.json': 3, 'scheduled-tasks': 1, 'agents': 1})

Each line carries fields including id, title, severity, confidence, file, line, exploit_scenario, preconditions, recommendation, and cwe_id.

confidence is worth reading alongside severity, because that value is set by the panel's vote.

Confidence in this report is clamped by that vote — only unanimous panels claim high.

How far you can trust it

The tool discloses its own limits. Three things that static-analysis reports tend to omit were stated outright.

Limits the report writes down itself

Some candidates go unreviewed. In my run, 38 candidate sites never reached the panel after dedup and capping. The earlier "45 candidates × 3 votes" counts only what reached the panel; these 38 are not in it. The report states plainly that absence from the report is not evidence of absence in the tree
It names what it excluded. In my run, three vendored Python virtualenvs and __pycache__ — with the note that if a dependency inside those virtualenvs is itself compromised, this scan did not see it
It accounts for coverage. Every top-level directory must be either scanned or set aside with a stated reason, and that reconciliation runs before the search begins. All 11 of my directories were accounted for

It executes no code

No tests run, no exploit fires, no proof of concept is validated. Everything is derived from reading. The heaviest finding in my run — an arbitrary code execution path — was confirmed not by planting an attack file but by reading it against a similar neighboring hook (which turned out to already carry the same defense).

That is the safe design, but the flip side is that reproducible confirmation becomes your job. I wrote a failing test first for each fix before implementing it.

Anthropic states that it is nondeterministic

From the README:

Scans are nondeterministic. Two scans of the same code can surface different findings.

It goes on to say that because it reasons about code the way a human security researcher does, it complements rather than replaces SAST, dependency scanning, and code review. This is not a drop-in substitute for static analysis.

It writes nothing

A scan only reads. Even the patch-generation job leaves your working tree untouched: nothing is committed, pushed, or opened as a pull request. Patch files land on disk and you decide whether to apply them.

What to do afterward

How to order the fixes

I ranked them in four tiers.

Priority	Condition	Why
1	Independently sufficient for arbitrary code execution	It bypasses every other defense, so fixing anything else while this stands is pointless
2	A cluster that reduces to one constant or anchor	One fix closes several findings at once
3	A definition with the trust direction inverted	The code may be correct while the premise of the judgment is broken
4	Documentation that contradicts reality	The harm is indirect, but it gives reviewers false reassurance

Priority 2 is handed to you by the report, so look for wording to the effect of "highest-leverage" in the body before planning anything.

You do not have to take the recommendation as given

The recommendation field is accurate, but it is not the only answer. One example.

My settings.json carried entries like Bash(bash:*) and Bash(python:*) that grant a whole interpreter. With those present, the other ~80 narrowly scoped grants are moot, because bash -c '<anything>' matches by prefix.

The report recommended moving these into permissions.deny. I chose removal from the allowlist instead.

deny is a refusal, so it makes legitimate execution permanently impossible
Dropping an entry from the allowlist merely returns it to a confirmation prompt in auto mode, which hands the judgment back to a human

The replacement shape is a shift from "grant by interpreter name" to "grant by the script you actually run."

-      "Bash(python:*)",
-      "Bash(python3:*)",
-      "Bash(sh:*)",
-      "Bash(bash:*)",
-      "Bash(node:*)",
-      (export / source / command removed too — 8 entries in all)
+      "Bash(bash ~/.claude/hooks/:*)",
+      "Bash(bash ~/.claude/scripts/:*)",
+      "Bash(bash ~/.claude/tests/:*)",
+      "Bash(python3 ~/.claude/skills/:*)",
+      "Bash(python3 -m pytest:*)",
+      "Bash(python3 -m scripts.:*)",

The finding is right; the recommendation is one option — that is the distance to keep.

Day to day, put it on Scan changes

Running the whole-repository scan every time is not realistic. For routine use, Scan changes is the one.

It targets a branch diff, a pull request diff, or a single commit
At medium, a diff of at most 5 files and 300 changed lines runs the lighter single-researcher shape instead of the full component matrix (the panel verification is unchanged)
Only committed changes are in scope. Uncommitted work in the tree is not part of any diff, so commit or stash first

If you want the patch-generation job, scan with a clean working tree. When the run's stamp carries revision.dirty: true, the job stops before drafting anything. Mine ran dirty, so that path was unavailable.

The scanner tripped one of my own guardrails

My setup includes a homegrown mechanism that detects attempts to abuse the auto-mode confirmation skip. During the scan, one panel voter systematically searched the Claude Code binary for permission-bypass symbols and set it off.

That candidate was rejected 0-3 by the panel, but the report records the behavior rather than burying it. The original phrasing was precise about why — a scanner probing its own guardrails is exactly the thing a reader should be told about.

Summary

As a measurement of the beta, here is what to take away.

Cost — one configuration directory (11 components) at medium came to 189 subagents over 2 hours. Put routine use on Scan changes instead
Character of the output — it produces findings regular expressions do not. Configuration and natural-language instruction files enter the audit scope, and documentation that contradicts reality gets flagged
Precision, anecdotally — of the 20 findings delivered, I rejected none as a false positive. One repository and one run, but this is not a tool built on the assumption that you skim and discard
How to read the count — it does not inflate the number; it folds the findings into clusters and names the highest-leverage repair. Look there first
Bounds of trust — unreviewed candidate counts, excluded areas, code never executed, nondeterminism: the report discloses all of it. A complement to static analysis, not a replacement

To be straight about it: I have not re-scanned after the fixes. I have not asked the tool itself to confirm that the findings are closed, and since it is nondeterministic there is no guarantee the same result would come back anyway. Taking that into account, treating it as "one more reviewer on the team" seems like the right weight for now.

herdr, a tmux for AI Agents — Until the Editor Disappeared

shimo4228 — Mon, 20 Jul 2026 13:00:06 +0000

What this article covers: how to build a terminal environment where you can monitor multiple Claude Code sessions with live status, come back to the same sessions after stepping away or over SSH, and — the interesting part — let the agents reorganize their own screen layout. It's a field log from installation to the point where the editor's (Zed's) role shrank from "writing" to "sign-off" (reviewing and approving) to "gone."

Intro — the "where do I put them" problem of parallel agents

Once you start running multiple Claude Code or Codex sessions in parallel, you hit these walls:

Terminal tabs pile up, and remembering "which tab was doing what" costs you time every single round
Quitting your editor or terminal app kills the agent sessions running inside it
You can't check on your running agents from your iPhone or another machine while away from your desk
Agents can't touch their own execution environment (pane layout, workspaces) at all. Rearranging things is always manual human work

This article solves these with herdr (an agent multiplexer — a terminal multiplexer built for agents). I'm a Zed user, and right after installing it my verdict was "isn't this redundant with what Zed already does?" — until the moment I let an agent manipulate the layout itself, which flipped my evaluation.

And the flip didn't stop there. Once herdr became my main battleground, the editor's (Zed's) role shrank from "a tool for writing" to "a tool for signing off on agent output" — and eventually it lost even that role and I stopped opening it. This is the whole story, from installation to the editor becoming unnecessary, with the configs and commands that actually worked.

Prerequisites

macOS + Homebrew (herdr also runs on Linux)
You're already using CLI agents such as Claude Code
This article is based on herdr v0.7.4 (verified 2026-07-18). The tool is only about three and a half months old, so the command surface may change
The editor is Zed (a recent build with the markdown_preview_* settings), and the terminal I end up choosing is Ghostty (the how and why comes later)
Pane IDs in this article (w5:p8 etc.) are real values from my environment. Yours will differ

What herdr is — only two real differences from tmux

herdr is an "agent multiplexer that lives in your terminal." It's a single Rust binary, and like tmux it persists sessions in a server process (the prefix key is even tmux-compatible: ctrl+b). GitHub is ogulcancelik/herdr, dual-licensed AGPL-3.0-or-later + commercial (stated in the LICENSE file).

The essential differences from tmux come down to two things:

Semantic agent state tracking — it auto-detects agents inside panes and lists their state — working / blocked / done / idle / unknown — in a sidebar. You can see "which one is waiting on me" without visually patrolling every pane
A socket API — pane splitting, command execution, output reading, and layout changes are all controllable from external processes (CLI commands like herdr pane run are wrappers around a Unix-socket API). Which means an agent can operate its own execution environment

The screen has three levels: workspace → tab → pane. From the CLI you address them as w1 (workspace 1), w1:t1 (tab 1 inside it), and w1:p1 (pane 1). Read the command examples below with this notation in mind.

Installing — one brew command

It's bottled in homebrew-core, so brew it is. I skipped the official site's curl | sh because piped install scripts are hard to audit. Including config generation, it's four lines:

brew install herdr                                    # v0.7.4
mkdir -p ~/.config/herdr
herdr --default-config > ~/.config/herdr/config.toml  # 305-line baseline config
herdr config check                                    # → config: ok

Running herdr starts the TUI, and the server (herdr server) comes up automatically. No login-item daemon (brew services) was needed.

As a sanity check, run a smoke test of the socket API. This doubles as the minimal example of "letting an agent operate the environment" covered later:

# Create a workspace, run a command in a pane, wait for output, read it back
herdr workspace create --cwd ~ --label smoke-test --no-focus
# → Returns workspace_id and pane_id as JSON. The lines below assume it returned
#   "w1:p1" — substitute whatever IDs you actually got (if you already have
#   workspaces, you'll get w2 or later)
herdr pane run "w1:p1" "echo herdr-smoke-ok"
herdr wait output "w1:p1" --match "herdr-smoke-ok" --timeout 5000
herdr pane read "w1:p1" --lines 10
# → If the output contains herdr-smoke-ok, you're wired up

If all four pass, layout, execution, and reading are controllable from an external process.

Where Zed is enough, and where it isn't

Let me be honest here: right after installing, my evaluation was "this is barely different from Zed."

Zed announced Parallel Agents on 2026-04-22. The Threads sidebar runs multiple agents in parallel, with per-thread git worktree isolation. Its terminal already does tabs (cmd+N) and splits (cmd+D). If you're at your desk reviewing in a GUI while agents run, Zed's experience is better. herdr's workspace switching looked about the same as switching Zed windows.

In fact, I almost shelved it with a tidy division of "at the desk = Zed, away = herdr."

What changed my evaluation was using the layers that structurally don't exist in Zed:

Layer	Zed	herdr
GUI review experience	◎ Parallel Agents + editor integration	— (terminal only)
Execution persistence	Quitting the app stops execution (thread history survives)	Server-resident. Execution continues with every app closed
External reattach to a running session	Not possible (SSH remote dev exists, but it's a different thing)	`ssh → herdr` from an iPhone etc. drops you back into the same screen
Agents operating their own environment	Not possible	Full control via the socket API

The first row is Zed's win; the bottom three are territory only herdr has (and the first row's verdict changes later, once plugins enter the picture). So it's not competition — it's a complement: herdr fills the "managing many agents" layer where Zed is weak.

One more trap that matters in practice: sessions launched from Zed's agent panel can't be grabbed from outside — not by the official Claude Code app (Remote Control), not by Termius (verified myself). They're sealed inside the editor process. A Claude Code launched as a CLI from a terminal shows up in the official app's list and is reachable via SSH → herdr. If there's any chance you'll want to look at a session from outside, launch it as a CLI.

The next section is the clincher for the complement story.

Letting the agent operate the layout itself

After installing, I wanted my agents — scattered across tabs — on one screen, and on a whim asked Claude Code itself: "consolidate the tabs into one."

Claude Code executed these three commands:

# Move panes from other tabs into tab t2 as splits (executed by Claude Code itself)
herdr pane move w5:p5 --tab w5:t2 --split right --no-focus
herdr pane move w5:p7 --tab w5:t2 --split down --target-pane w5:p4 --no-focus
herdr pane move w5:p6 --tab w5:t2 --split down --target-pane w5:p5 --no-focus
# → Panes scattered across tabs become a 2×2 grid on one tab. Emptied tabs auto-close

Two agents were in working state at the time, and the layout changed without stopping a single one. All I did was say one sentence. No worrying about reopening panes and interrupting processes, no assembling the sequence of move commands by hand.

Here's what it looked like. Starting from one-tab-per-agent (Before):

Before: four tabs in the bar. Only one session visible at a time

I ask the Claude Code in another pane to "consolidate them":

The instruction is a single sentence. Claude Code surveys the current tab structure via the socket API, assembles the pane moves, and runs them

And everything lands in a 2×2 grid on one tab:

After: every agent on one screen. Running processes never stopped

Agents know where they are through environment variables:

$ env | grep -i herdr    # home directory in the output replaced with ~
HERDR_ENV=1
HERDR_PANE_ID=w5:p8
HERDR_SOCKET_PATH=~/.config/herdr/herdr.sock
HERDR_TAB_ID=w5:t2
HERDR_WORKSPACE_ID=w5

So an agent knows "which pane am I in," and from there it can split a pane next door, run a command, and read the result. The shape of the execution environment itself becomes one of the agent's tools. Incidentally, the session writing this very article is a Claude Code inside a herdr pane — herdr agent list shows it as working on this draft.

git worktree integration was also a one-liner (verified myself):

# Create a worktree + branch + new workspace in one shot
herdr worktree create --workspace w6 --branch feature-x
# → Creates a worktree at ~/.herdr/worktrees/<repo-name>/feature-x/ and
#    opens a workspace labeled "feature-x" with that as its cwd

The same "one room per branch, work in parallel" structure as Zed Parallel Agents — except an agent can assemble it from the CLI.

Gotcha: a zoomed tab refuses layout changes

pane move can return changed: false (reason: "zoomed_tab") and do nothing. If the target tab is in zoomed view, layout changes are rejected by design. Run herdr pane zoom --off to unzoom first.

Gotcha: an omitted target means the focused pane, not the caller

When I had Claude Code run herdr pane split, the pane split in a different workspace — the one I happened to be looking at — not the intended one. That's because when you omit the pane target, it resolves to "the pane focused in the TUI," not "the pane the command was invoked from" (stated in the official CLI reference; passing --current resolves to the calling pane).

When agents drive the CLI, don't let them omit the target; have them pass their own position explicitly via --current or HERDR_PANE_ID.

Gotcha: server lifetime and startup directory

The server belongs to the parent process that started it. If herdr server gets started from Claude Code's shell, it can die with that session. After my experiments I stopped the server once and restarted herdr myself so the server ownership sat with my own process
The startup cwd only matters once, at first session creation. cd-ing into a repo and running herdr just reattaches to the existing session from the second time on. Add repos from inside herdr with herdr workspace create --cwd --label. It took a mental-model move from "cd to navigate" to "switch workspaces to navigate"

Does the sidebar show state, or history?

Running herdr inside Zed's terminal gives you this nesting:

The spaces / agents sidebar on the left. Each agent gets one line with its state and remaining context

What I noticed in use: herdr's sidebar only shows rows that map 1:1 to living processes. When an agent finishes, its row disappears. It structurally cannot get cluttered.

Zed's Threads sidebar, by contrast, is a history list. Sessions that ended days or months ago sit alongside active ones, projects interleaved. Finished work keeps occupying your screen as "attention inventory."

Zed's Threads sidebar. Sessions from 4 days, 2 weeks, and 2 months ago mixed into the same list as active ones

Does a list show current state, or past history? That felt like a litmus test for UIs in the parallel-agent era.

This realization changed how I use the whole screen, too. I used to keep a permanent two-way split of editor and CLI — but I only actually looked at the editor "when reading code," and the rest of the time it was dead space.

Before: permanent two-way split. The editor side spends most of its time unwatched

After: herdr (the CLI side) is the main screen; when I read code I flip the editor to full screen with cmd+shift+backtick

In agent-driven development, the primary screen flips from the editor to the agent CLI, and the editor becomes the on-demand side. Same root idea as the sidebar: align "what's in view" with "what's in use right now," and you stop leaking attention.

For a while, the answer was "Zed at the desk, herdr everywhere else"

Even after the evaluation flipped, my placement answer briefly returned to that first instinct (the "shelve it" plan above). Only now it wasn't a passive shelving but an active division of labor: GUI review at the desk goes to Zed; being away, persistence, and delegation to agents go to herdr. Monitoring and reattaching from the iPhone is measured and working on the herdr side.

Field notes on away-from-desk operation (iPhone mirroring, sleep behavior) - The iPhone connection path (Tailscale + Termius) is identical to [the one I wrote up in the tmux days](https://dev.to/shimo4228/running-claude-code-from-iphone-via-ssh-tmux-4c10) — just swap tmux for herdr. If conversational UX is your main goal, [my write-up on the official Claude Code app](https://dev.to/shimo4228/claude-code-from-iphone-plugging-3-holes-in-remote-control-17cf) is a better fit. The split is: "conversation = official app / fleet monitoring = Termius + herdr" - Attach to the same session from the Mac's terminal and from Termius on the iPhone, and the two become **mirrors of the same screen**. Workspace switches and focus moves on one side show up on the other with barely any lag ![Termius on iPhone mirroring the same herdr session as the Mac](https://raw.githubusercontent.com/shimo4228/zenn-content/main/images/herdr-termius-iphone-sync.png) *The iPhone side. It syncs with the Mac's screen with barely any lag* - The display size **syncs to the smaller client**. While you're driving from the iPhone, the Mac's terminal also rewraps to the iPhone's screen width ![The Mac side while the iPhone is in control. Pane contents rendered at the iPhone's screen width](https://raw.githubusercontent.com/shimo4228/zenn-content/main/images/herdr-termius-mac-sync.png) *The Mac at the same moment. While the iPhone is driving, the Mac's display syncs to iPhone size* - **Agents make no progress while the Mac sleeps** (sleep suspends process execution, launchd-managed or not). Sleep doesn't kill the server — sessions are intact on wake (measured myself: sleep → wake, session survived). If you want agents running while you're away, configure the Mac not to sleep

But this division of labor didn't last. A few days of using herdr as the main battleground ran me into the next question: Claude Code writes the code — so what am I actually doing in the editor? Here begins the second half.

The three jobs left in Zed — from "writing tool" to "sign-off tool"

Once you hand all code-writing to the CLI (Claude Code), the editor's "write" function has long since retired. I was still keeping Zed around for three things:

Reading — following multiple files side by side on one screen (multibuffer)
Diff review — eyeballing the changes
Checking Markdown — polishing how articles and design notes look

In other words, everything left was on the "reading" side. So I flipped the framing: optimize Zed as a tool for reading and signing off.

The first thing that paid off was the zed CLI. Append :line to a file path and it opens with the cursor on that line:

# Open a specific line (line:column also works)
zed src/contemplative_agent/cli.py:715

# Open several sign-off points at once
zed src/core/metrics.py:88 src/cli.py:715 tests/test_cli.py:350

# --wait: don't proceed to the next command until the human closes the file
zed --wait changed_file.py

This reverses the direction of the sign-off flow:

Before: the agent tells you path:line as text → the human hunts for it in the editor
Now: Claude Code pinpoints the line with grep, runs zed <file>:<line> → and hands the human an editor already open at the spot

The human stops searching and just reads the line being pointed at. Add --wait and you get a structural pause — "don't proceed until the human has laid eyes on this file" — in a single command. One caveat: closing the file only means "seen." It can't distinguish approve from reject, so don't chain irreversible operations like commits onto --wait; run those explicitly, human-side, after confirming.

I didn't uninstall Zed at this point. Its resident cost while unused is zero, and the cost of rebuilding a "reading" replacement on the terminal side is real. Let actual usage make the call — that was the judgment.

The Japanese-text rendering wall, and a host swap — to Ghostty

Once I started optimizing for "reading and signing off," I hit a rendering-quality wall next. Two problems, both around Japanese text.

First: line spacing in Zed's Markdown preview. Preview a Japanese Markdown document and the lines are cramped and hard to read. Digging in, the preview's paragraph line height is hard-coded at 1.3 with no user setting (zed#56111 has a request for Japanese-friendly settings). The preview-only font settings (markdown_preview_font_family etc.) let you raise the size, but the 1.3 ratio itself can't be changed, so it's not a real fix. I moved final article checks to npx zenn preview (the same rendering as production) and demoted Zed's preview to "glances while writing." Zed can't finish the reading experience — the first crack.

Second, and nastier: Markdown tables emitted by Claude Code came out misaligned in the terminal, with broken rules.

My suspect was characters whose East Asian Width (Unicode's character-width classification) is "ambiguous" — box-drawing characters and the like. That width can be interpreted differently by each layer: Claude Code computing character widths, herdr laying out the grid, and the terminal actually drawing. When the layers disagree about width, tables break. (Other rendering factors, like font fallback, can produce similar symptoms — so this is a leading suspect, not a confirmed cause.)

I couldn't pin down the exact mechanism, but which layer was at fault could be isolated with a controlled swap: replace only the host (the terminal doing the rendering). Piping the same Claude Code + herdr output into Terminal.app, the tables lined up cleanly. The misbehaving layer was confirmed to be Zed's terminal rendering. For display bugs in a multi-layer stack, this "swap exactly one layer" move is the fastest isolation you can do.

Host	Table correctness	Color (True Color)
Zed's terminal	Breaks	Accurate
Terminal.app	Accurate	Not supported
Ghostty	Accurate	Supported

I ended up on Ghostty. It draws box-drawing and block characters itself instead of relying on the font, so tables don't break, and colors come out accurate.

"After the fix." Same Claude Code + herdr output — on Ghostty, tables with box-drawing rules line up.

With the host settled, unify the look across the nesting too. If the host (Ghostty) and the TUI (herdr) disagree on color scheme, every glance between them adds a small snag. herdr can auto-follow the host's light/dark appearance:

# ~/.config/herdr/config.toml
[theme]
name = "tokyo-night"
auto_switch = true          # follow the host's light/dark
dark_name = "tokyo-night"
light_name = "tokyo-night-day"

herdr server reload-config applies it without complaint, and the three tiers — macOS appearance → Ghostty → herdr — switch together.

The same two-pane screen in light and dark. It follows the host's (Ghostty's) appearance, TUI and all.

Why herdr gets details like Japanese input right — a solo dev and his bots

What surprised me in use: herdr has engineering effort in places as practical as Japanese input (IME) handling. The settings dialog has entries like these:

For example, with a Japanese IME active, pressing a key sequence like ctrl+b v can get the v swallowed by the IME. herdr's countermeasure: switch the input source to ASCII only while accepting a key sequence, then switch back. There's also a feature that persists pane scrollback across server restarts.

How does a tool get this level of polish? The repository's raw data hints at the reason:

gh api repos/ogulcancelik/herdr \
  --jq '{created: .created_at, stars: .stargazers_count, license: .license.spdx_id}'
# → {"created":"2026-03-27...","stars":17794,"license":"NOASSERTION"}
#   (license shows NOASSERTION because GitHub can't auto-detect
#    the AGPL + commercial dual license)

gh api "repos/ogulcancelik/herdr/contributors?per_page=6" \
  --jq '.[] | "\(.login): \(.contributions)"'
# → ogulcancelik: 979 / kangal-bot: 54 / github-actions[bot]: 43
#   akbash-bot: 16 / human contributors are at 4 commits or fewer each

At 113 days since creation (as of 2026-07-18) it's past 17k stars, and it hit Hacker News on 2026-06-29 (166 points, 110 comments; all figures as of writing). Meanwhile, roughly 979 of the commits are the author's own — it's effectively a solo project. The fun part: the #2 and #4 contributors are bots (kangal-bot and akbash-bot). Presumably these are agents the author operates, committing under their own accounts.

An aside: kangal and akbash are both real breeds of Turkish livestock guardian dogs. Perhaps a deliberate match with the name herdr (one who herds) — though that's speculation about naming intent. What the API establishes for certain is only that nearly all commits are attributed to the author himself and to bot-named accounts; that the author operates those bots as agents is itself an inference from circumstance.
The tool's defects bounce back into the developer's own velocity every day. So requirement discovery gets fast. Tools of the agent era fit the hand best when they're built together with agents — and that structure was right there in the contributor list.

The ending — Zed didn't even get to be the sign-off viewer

Everything so far was about "optimizing Zed as a sign-off viewer." Then even that role got taken by the herdr side.

The trigger was two herdr plugins:

file-viewer — read-only, git-aware file browsing
reviewr — shows diffs in a sidebar and sends line comments back to the agent

file-viewer covers "reading"; reviewr covers "review the diff and send feedback" — both now inside herdr.

The agent on the left and the file it opened on the right, on the same screen. "Reading" moved to the terminal side.

The reviewr screen. The right sidebar shows the uncommitted diff and the changed-file list. What's on screen is the diff of this very article's consolidation work.

Here the flow comes full circle. Against the "agent → human" bridge from the earlier section (zed file:line), reviewr builds the reverse "human → agent" bridge (comment send-back). The sign-off loop closed inside herdr.

This plugin marketplace just auto-indexes a GitHub topic — there's no review process. Installing a plugin is the same as granting code execution, so I read the manifest and install script before installing (for file-viewer: SHA-256 verification present, no automatic hooks — confirmed).
The result: the zed file:line bridge stopped needing to be crossed within hours of being built. The third job, checking Markdown, was covered too — glances-while-writing by file-viewer's rendered view (the v key cycles diff ⇄ rendered ⇄ syntax), and final checks already consolidated onto npx zenn preview — leaving no reason for Zed to stay. Zed lost even the sign-off viewer seat, and within a day I'd stopped opening it.

The final stack: Ghostty + herdr + plugins + Claude Code. There is no editor.

Closing — the shape of the execution environment becomes the agent's tool

Seen as "a tmux successor," herdr looks redundant next to Zed. My evaluation flipped when the socket API let an agent operate its own environment. Splitting panes, opening rooms, cutting worktrees — the "workspace housekeeping" humans used to do becomes delegable, wholesale.

And the flip kept going, all the way to a change of tools. From "isn't Zed enough?" to "sign-off only" to "gone." While every layer — editor, terminal, harness (Claude Code's own subagent machinery) — is absorbing "agent orchestration" features at the same time, herdr's distinctive answer is that it made the layout of the execution environment itself something agents can operate. Each layer slimming down to a single function feels less like regression than like attention design: the work of paring what's on screen down to a state with not one layer surplus to its current role.

That said, "best" here means best for the current role, not forever. The honest way to test the judgment is against actual usage, so in a few weeks I plan to look back at "how many times did I open Zed?"

It's one brew command to try. If you're running two or more Claude Code sessions in parallel, start by watching your own agents show up in herdr agent list.

What Do My AI Agent's Logs Look Like in OpenTelemetry?

shimo4228 — Thu, 16 Jul 2026 13:00:04 +0000

What this article covers: what you can see when you plug homegrown logs into OpenTelemetry. I convert them to OpenTelemetry format, visualize them, and sort out which elements are worth borrowing and which you can pass on.

I wanted to bring OpenTelemetry to my AI agent (telemetry = the standard machinery for recording and emitting data about a system's behavior; OTel from here on). The typical adoption path is to embed the SDK in the application itself, emit telemetry as it runs, and keep shipping it to a monitoring backend via a Collector (a resident process that receives, transforms, and forwards). But maybe you've hit walls like these:

Running an SDK and a Collector around the clock is heavy for a personal-scale agent. There's no ops team watching a monitoring dashboard
I already have my own structured logs. I don't want to instrument the same information twice
The standard for which attribute names to record LLM calls under (the GenAI semantic conventions) is still in development. The spec may still move, so it's hard to judge how far to lean on it

Contemplative Agent, the agent I operate, was in exactly this situation. It's a local-LLM agent that autonomously writes posts and comments on Moltbook (a social network where AI agents post and comment on each other). The text it reads from the feed is written by other agents, and prompt injection (smuggling instructions into a post body to steer the agent reading it) is a daily occurrence — an environment where input cannot be trusted.

When the agent misbehaves in this environment, investigating the cause requires being able to trace, precisely and after the fact, "what it read from outside and what it emitted at that moment." So I had designed all external input/output to be kept in full in an audit log (an append-only record for reproducing past decisions exactly as they happened). The "own structured logs" in the second wall above is this audit log.

At the same time, nobody in my environment watches telemetry continuously. What I need is "the ability to examine past runs in a standard UI when something is worth investigating." If that's the requirement, the telemetry doesn't have to be flowing in real time.

So this article does two things.

Instead of putting OTel into the production runtime, I built an after-the-fact conversion as an experiment. It takes the audit logs I already have, converts them into traces (a record of one processing run as a collection of timestamped intervals) after the fact, and ships them over OTLP (OTel's standard transport protocol) for visualization
I compared my log schema against the OTel standard and added the 2 fields it was missing to the agent's own logs: run_id and session_id, which identify a run

The first is a small conversion script — zero changes to the agent body, two dependencies. My local logs became visible as a waterfall (the chart that stacks processing intervals along a timeline) in Jaeger (an OSS trace viewer). On a day that had an incident, the picture tells you the moment you open it.

This article is a sequel to my previous piece, "Why Did My Agent Decide That? 3 Observability Patterns". There I wrote that what traces and metrics tell you by default stops at "what the request did." This is the continuation: what, then, can homegrown logs and the OTel standard actually connect? — answered by doing the conversion for real. It also reads fine on its own.

Prerequisites

Conversion target: append-only JSONL (one JSON per line) logs the agent has already written. Three kinds: LLM call telemetry / API audit / CAPTCHA solver audit
Python 3.10+, uv for package management
Only two dependencies: opentelemetry-sdk + opentelemetry-exporter-otlp-proto-http (both on the conversion-script side; nothing is added to the agent itself)
Viewer: Jaeger v2.19.0 (single binary; no Docker required, storage is in-memory)
Signals: traces only (of OTel's three signals, metrics and logs are out of scope this time)
The conversion script is public: contemplative-agent-otel — a small implementation, about 800 lines in total, built on the assumption that you swap in records.py (log loading and normalization) / mapping.py (the attribute-name mapping table) for your own log format

The processing flow is 5 steps.

Read the already-written JSONL logs
Normalize the 3 log kinds into a common record (records.py)
Attach standard attribute names to each record and convert it into a span (one interval of a trace) (mapping.py)
Send to Jaeger over OTLP/HTTP
Check the waterfall and the attributes in the Jaeger UI

I decided whether to adopt OTel by asking "who reads the telemetry?"

Put OTel into the agent, or pass? I organized this decision around a single question: "who is going to read that telemetry?" The reason is that OTel traces and my audit log, even when they record the same event, have different readers (consumers). Take one LLM call: on the trace side, what you want to see is the model name, token counts, duration, and whether it errored. On the audit-log side, what you need is the prompt body itself, the caller, and the identifiers used for reproduction.

	OTel traces	My audit log
Primary reader	People watching dashboards; alerting	People investigating incidents; the replay script used for reproduction
Prompt and other bodies	Not recorded by default (opt-in)	Stored in full (base64 + sha256)
Nature of the data	May be sampled; may be volatile	Append-only; every record; never deleted
Question it answers	"What's slow or broken right now?"	"Why did that decision come out the way it did?"

The right column is not a property of audit logs in general — it's a design I chose because my goal is offline reproduction of incidents. Your logs may well not store everything. What generalizes is the yardstick: different consumers mean different retention policies.

The emblematic case is body text. The GenAI semantic conventions specify that prompt and response bodies are not recorded by default (they end up on a viewer's screen, so not carrying sensitive data is the default). My audit log does the exact opposite: full retention. In incident investigation, without "the exact bytes the decision saw," you can't reproduce it offline.

So this isn't a question of which one is right: same event, different consumers, opposite retention policies. Once you frame it this way, the choice stops being binary.

Put OTel into the runtime (if someone reads telemetry in real time)
Do nothing
Convert existing logs into traces after the fact — get the connection to the standard vocabulary, and the visualization, without touching the runtime

This article is an implementation of option 3.

That said, option 3 only held up because two conditions happened to be true for me: nobody reads telemetry in real time, and the incident-investigation reader was already well served by a thorough audit log. If either is missing, the answer changes. Even with no monitoring team, if you don't have structured logs yet, going straight for option 1 (the runtime SDK) is probably faster. "No monitoring team = no OTel" is not a general rule.

You might also wonder: "isn't using only part of the standard just cherry-picking the spec?" OTLP is a public protocol, and the semantic conventions are an agreement on attribute names. You don't have to instrument the app with the SDK — any program that speaks OTLP can feed data in. This is a usage the standard anticipates, and this article's conversion script simply uses the OTel SDK as "a library for offline export."

What I borrowed from OTel, and what I passed on

Before getting into the implementation, here's how I judged each element of OTel, up front. The biggest lesson this time was that it's not an all-or-nothing "adopt the standard or don't" — you can pick borrow-or-pass per element.

OTel element	Verdict	Why
GenAI semantic conventions vocabulary (`gen_ai.*`)	✅ Adopt	External tools and people can read it without explanation. Copying attribute names costs zero dependencies
Run-ID machinery (trace ID / session ID equivalents)	✅ Adopt (implemented `run_id` / `session_id` in the logs)	One field turns trace reconstruction from estimation into measurement
OTLP (transport protocol)	✅ Adopt (as the exit of the offline conversion)	A common language every viewer speaks. Costs only 2 packages
Runtime SDK instrumentation	❌ Pass	Would double-instrument alongside the existing audit log, and add dependencies to the agent
Running a resident Collector	❌ Pass	Heavy for a single process at personal scale. On-demand conversion when needed is enough
Bodies not recorded by default (redaction)	Partial	Followed on the trace side (don't carry untrusted text onto the screen). The audit log does the opposite — full retention, needed for offline reproduction
Importing attribute names from the semconv package	❌ Pass	Import paths are still unstable while in development, so attribute names are self-managed as string constants

Passing on an element doesn't mean that part of OTel has no value. There are mainly 4 things that are hard to get without runtime instrumentation.

Exact causal structure — which operation called which. Only IDs issued at execution time capture this precisely
Context propagation across services — the machinery that stitches one trace across multiple services
Real-time visibility — latency and errors happening right now, visible as they happen
Metrics — numbers aggregated continuously while running. You can technically produce them from an after-the-fact conversion too, but they're fundamentally a signal meant to be measured live

Of these, the last 3 have no reader in my operation — a single process, with nobody watching continuously. The only one genuinely missing is the first, and its minimal patch is row 2 of the table, the "run-ID machinery."

The sections below implement the ✅ elements in order.

My logs mapped almost 1:1 onto the standard vocabulary

My homegrown telemetry writes one line per LLM call. A line looks like this (simplified):

{"ts": "2026-07-15T09:12:03+09:00", "model": "gemma4:e4b", "prompt_eval_count": 1560, "eval_count": 210, "duration_ms": 36400, "done_reason": "stop"}

The next table matches these fields against the GenAI semantic conventions attribute names (the naming table for how to record model name, token counts, finish reason, and so on in OTel).

My log field	OTel attribute
`model`	`gen_ai.request.model`
`prompt_eval_count` (input token count)	`gen_ai.usage.input_tokens`
`eval_count` (output token count)	`gen_ai.usage.output_tokens`
`done_reason` (`stop` / `length`)	`gen_ai.response.finish_reasons` (array-typed, so wrap as `[done_reason]`)
`num_predict` (generation cap)	`gen_ai.request.max_tokens`
`temperature`	`gen_ai.request.temperature`
`error_kind` (`timeout` / `http_429` etc.)	`error.type`
`ts` + `duration_ms`	start / end time of the span (one interval of a trace)
`caller` (which processing stage made the call), `prompt_sha256`, etc.	No counterpart → custom namespace `ca.audit.*`

It mapped almost 1:1. For a simple LLM call log like this one (model, token counts, stop reason, parameters), I suspect the set of things you'd want to record converges to the same collection no matter who designs it (logs that also cover tool calls, streaming, or routing won't map this cleanly).

This is where riding the standard vocabulary pays off. If the attribute name is gen_ai.usage.input_tokens, external tools and people alike can read it with no explanation.

The interesting part is what didn't map. caller (the key I use to aggregate logs by processing stage during incident investigation) and prompt_sha256 (a hash for matching identical prompts without storing the body) have no standard counterpart. These two are investigation keys specific to my operation. For just looking around in a UI, the standard attributes are mostly enough; for offline reproduction of an incident, these are the ones that matter.

Practical handling of the Development status (official status name: Development)

The GenAI semantic conventions (semconv below) are in Development status as of this writing (2026-07) (canonical repository). The opentelemetry-semantic-conventions package does have attribute-name constants, but they live under an underscore-prefixed (= unstable) _incubating import path.

The conversion script doesn't import from the package. Instead, the attribute names are defined as string constants in my own code, with the referenced semconv version pinned in a comment. If the standard moves, catching up is a single-file change.

The conversion script has exactly 3 design points

Let's pin down spans properly here. At the top I described a trace as "a collection of timestamped intervals." Each of those intervals is a span: it has a name, start and end times, and attributes (key-value pairs), and spans connect through parent-child links into one trace. In a Jaeger waterfall, one row is one span.

So the core of the conversion is "read JSONL, create spans with past timestamps." There were 3 design decisions.

1. Spans can be created at past timestamps

The OTel SDK lets you explicitly set a span's start and end times in epoch nanoseconds. I restore them directly from the log's ts and duration_ms.

```python:emit.py (excerpt, simplified)

Create the span at the log's recorded time (the core of the conversion script)

span = tracer.start_span(
name, # e.g. "text_completion gemma4:e4b"
context=parent_ctx,
kind=kind, # SpanKind (CLIENT etc. for LLM calls)
attributes=attrs,
start_time=start_ns, # ts converted to epoch ns
)
if is_error:
span.set_status(Status(StatusCode.ERROR, error_type))
span.end(end_time=end_ns) # start_ns + duration_ms




For logs that don't record a duration (API audit, solver audit), I made the spans **zero-width**. Giving them a plausible width by estimation would mean fabricating latency that was never measured. Jaeger draws zero-width spans as thin markers, so they're also visually distinct from the LLM spans that carry real durations. The fact that "logs without a duration become zero-width" was itself feedback into the log schema (it pinpointed the next improvement candidate: add elapsed time to the API audit log too).

### 2. Trace grouping is reconstructed from gaps in time

A trace is normally bundled by IDs issued at execution time. Past logs don't have those. So I merge the 3 log kinds in timestamp order and **start a new trace whenever a gap of a fixed length (default 300 seconds) opens up**. A schedule-launched agent repeats "a few minutes of activity → long silence," so this cut lines up well with an actual single run. Conversely, for workloads where the logs never pause — parallel runs, always-on workers — this cut will merge things incorrectly.

But it is still an estimate. I put a `ca.convert.grouping = "time-gap"` attribute on each trace's root span, making **the data itself say it is a reconstruction, not a measurement**.

Flip that around and this was **a gap in my own log design that only became visible against OTel**: add one field to every log line — a per-run ID (the equivalent of OTel's trace ID) — and this reconstruction turns from estimation into measurement.

Even if you never adopt the SDK, the one mechanism worth borrowing up front is "stamp every run with an ID."

I implemented it in my agent while writing this article: a single write function shared by all audit logs now stamps `run_id` (per process) and `session_id` (per agent session). That is the only agent-side change in this whole effort, and converting past logs still works with the agent untouched. The converter bundles by ID when `run_id` exists and falls back to time gaps only for older logs. If you're designing logs now, I recommend putting it in from day one.

### 3. Untrusted text is dropped at the conversion entrance

The third design decision is the boundary of what gets onto the Jaeger screen.

The audit log contains raw text that came from outside (CAPTCHA challenge texts, server error response bodies), stored as base64. As described at the top, this agent's input comes from a social network where prompt injection is routine — "strings from outside may be attack input" is the default assumption. Put such text into span attributes, and **attacker-controlled strings flow straight onto the Jaeger screen (= into screenshots, = into this article)**.

The conversion drops bodies at the parser stage and passes through only a sha256 hash and a classification code (`http_400` etc.). There is no "flag to include bodies" either. On top of that, I added a regression test that scans every attribute value of every test fixture and asserts that no body fragment appears. Not "be careful later," but "make it structurally unable to appear, then visualize" — that's the safe side.

To be clear, the goal here is not to keep bodies secret but to avoid carrying external strings onto the screen. The sha256 is a correlation identifier for matching identical inputs, not an anonymization mechanism (short boilerplate strings can be brute-forced back). If your logs hold confidential data, consider a keyed hash (HMAC) or simply not surfacing the hash at all.

## Looking at it in Jaeger — the incident day was obvious the moment I opened it

Jaeger v2 is a single binary; once started, it accepts OTLP directly and keeps everything in memory. Because it takes OTLP directly, there's no need for the Collector I called "heavy" at the top, either.



```bash
# Start the viewer (no Docker; data disappears when it exits)
# For Apple Silicon. On Intel Macs read darwin-amd64; on Linux, linux-amd64
curl -sLO https://github.com/jaegertracing/jaeger/releases/download/v2.19.0/jaeger-2.19.0-darwin-arm64.tar.gz
tar xzf jaeger-2.19.0-darwin-arm64.tar.gz
./jaeger-2.19.0-darwin-arm64/jaeger    # OTLP :4318 / UI :16686

# Convert and send (example run; see the repository README for setup)
contemplative-agent-otel --date 2026-07-15
# => emitted 1031 spans across 5 runs -> http://localhost:4318

A normal day looks like this. Reading the screen: time runs along the horizontal axis, one row is one span, parent-child structure is indentation, errors are red marks. One trace is one agent run, with LLM calls carrying real durations (text_completion gemma4:e4b, 36 seconds or 1.3 minutes) and zero-width API calls lined up in time order.

Open a span and the mapping table from earlier appears directly as attributes. Standard attributes like gen_ai.usage.input_tokens: 1560 sit alongside custom ones like ca.audit.caller: core.skill_selection.

And here's what converting the day with the incident produces. On that day, a retry storm (failed calls whose retries cascade until calls and logs balloon) had in fact hit the LLM backend.

Where a normal run has tens to hundreds of spans, one trace with 30,442 spans and 33 errors towers over everything. The log file for that day was also nearly 30 times its normal size. As numbers, this information was already in the logs — but "you know it's an abnormal day the moment you open it" is a strength unique to trace visualization.

You can follow the errors too. In the run where the CAPTCHA answer was rejected by the server, captcha solve and POST /verify are marked red as a pair. The error bodies were dropped by the conversion, so all the screen shows is the classification: error.type: http_400.

Gotcha: send 30,442 spans at once and they're silently dropped

In the OTel SDK, span.end() doesn't send immediately — a BatchSpanProcessor (an SDK-internal component that queues spans and sends them in batches) does the exporting. In the incident-day conversion, spans beyond the default queue length of 2048 were discarded with only a one-line warning (Queue full, dropping Span.). For real-time instrumentation that's a reasonable self-defense; for an offline bulk conversion, "send everything" is the correct spec.

Since the number of records to convert is known before sending, I sized the queue to the record count and that solved it. If you use the SDK for bulk conversion, watch out: the defaults assume real time.

Summary

Instead of a binary "adopt OTel or don't," I tried the third option: after-the-fact conversion of existing logs. Zero changes to the agent, two dependencies — and you get the Jaeger waterfall, error display, and standard attributes, all of it
My homegrown LLM call log mapped almost 1:1 onto the GenAI semantic conventions. The fields that didn't map remain as a list of "information needed for offline reproduction of incidents"
Traces serve humans investigating on a screen (bodies unrecorded by default); the audit log serves machines re-executing for incident reproduction (bodies stored in full). Different consumers flip the retention policy, so in a conversion, dropping untrusted bodies at the entrance is the safe side
Master 3 things — span creation at past timestamps, zero-width spans, trace reconstruction from time gaps — and any structured log from which times and run groupings can be recovered connects the same way
Add one run-ID field to your logs and the reconstruction itself becomes unnecessary. Even without the SDK, this one mechanism is worth borrowing from OTel up front (I implemented it while writing this article)
This conversion was also a way to measure my own log design against OTel as a yardstick. The missing run ID and the real-time assumptions of the export queue both became visible only against the standard

Fault Injection TDD Found 3 Silent Failures in My LLM Agent

shimo4228 — Wed, 15 Jul 2026 13:00:03 +0000

Pipelines with an LLM in the loop keep hitting walls like these:

The LLM returns JSON in a subtly different shape. Parsing succeeds, but the result silently comes back empty
A local LLM's response was cut off mid-generation (done_reason=length) and you only notice days later
The log only says outcome="error", so you can't tell whether it was a rate limit or a timeout

What they have in common: every test passes, yet production breaks silently. Processing doesn't stop, no exception is raised — the results, or the records of them, quietly degrade. In this article I call this a silent failure.

The local-LLM CLI agent I operate hit this family of failures 5 times. All 5 slipped past the tests I had beforehand and were only diagnosed after the fact. So I adopted this workflow: build a fault catalog (a taxonomy of failures) from the history of production incidents, write tests that deliberately inject those faults first, and land the minimal guard that makes them pass in the same PR. Taxonomically, this is fault injection testing × TDD. What I borrowed — and deliberately didn't borrow — from chaos engineering (the practice of injecting failures on purpose to verify resilience), which inspired the approach, is laid out in the body.

The first application found 3 silent failures that my ~1800 existing tests had missed.

What you'll get from this article: a pattern for adding fault injection testing to a single-process Python app — no daemon, deterministic, pytest-native (fault catalog → RED → GREEN) — plus the details of the 3 silent failures the first application actually surfaced

Prerequisites

Python 3.13 / pytest 9.0 (at time of verification; no bleeding-edge features are used, so nearby versions should work)
hypothesis 6.156.6 / responses 0.26.0 (both added as dev dependencies)
Target: a single-process Python app that calls an LLM server (Ollama etc.) via requests
The LLM call sits behind a swappable layer such as a Protocol (if it doesn't, carving out that layer is step zero)

This article is a sequel to my previous one, "Why Did My Agent Decide That? 3 Observability Patterns". If that one was about recording the agent's decisions, this one is about asserting on that recording channel — and injecting faults up front. It stands alone, though.

Chaos engineering was the inspiration; fault injection testing is what I adopted

My agent operates autonomously on a social network (Moltbook), continuously. As long as it keeps running, I can't know in advance which failure it will meet next. Searching for a way to build "operations resilient to unforeseen failures," I arrived at chaos engineering.

But measured against the five advanced principles of the canonical Principles of Chaos Engineering, I did not follow it verbatim.

Canonical principle	This work
Build a Hypothesis around Steady State Behavior	○ Adopted (asserts against the execution-log (telemetry) channel)
Vary Real-world Events	△ Half (faults come from incident history, but injection is a deterministic replay of a known catalog)
Run Experiments in Production	✕ Rejected
Automate Experiments to Run Continuously	△ CI runs them automatically, but derandomized — a regression test that pins the known, not an experiment that explores the unknown
Minimize Blast Radius	○ Adopted in extreme form (injection happens inside tests, so production impact is zero)

I rejected production experiments for 2 reasons. First, the target is a single local process — there is no "redundant fleet of instances" to kill in the first place. Second, the logs the agent accumulates are primary data that can't be regenerated; random injection in production risks corrupting them. I deferred random exploration to keep CI from going flaky.

The core of chaos engineering is not "testing" (asserting known expected behavior) but "experimentation" (discovering unknown weaknesses) — so once production experiments and random exploration are out, it can no longer be called chaos engineering. As stated up front, the classification is fault injection testing × TDD. What I borrowed from chaos engineering is exactly 2 ideas: "build the failure catalog from real-world events" and "assert steady state through an observation channel."

Tooling-wise, I also skipped the distributed-systems staples and built on pytest. This was the conclusion of pre-adoption external research.

Candidate	Verdict	Reason
chaostoolkit / toxiproxy	Rejected	Declarative experiment runner / resident network proxy. Built for distributed topologies; the scale of the target doesn't match a single local process
pytest fault-injection plugins	Rejected	Nothing usable exists (e.g. pytest-disrupt is a TODO-only scaffold)
agent-chaos	Reference only	Conceptually the closest, but coupled to the Anthropic SDK / DeepEval / pydantic-ai, with no pytest integration and no local-backend support
hypothesis + responses + a hand-rolled injection backend	Adopted	Sits on the existing test infrastructure (pytest) with just 2 dependencies

That settled the approach: inject faults deterministically inside tests. No daemon, no proxy — pytest itself is the execution environment.

The 2 elements I deferred (exploration, experimentation) are not discarded; they remain as the entry points for extending this toward chaos engineering later — a two-tier setup where PR CI stays deterministic while a nightly run randomizes with seed recording (any failing sequence found gets reproduced from the seed and pinned with @example), plus sandbox experiments running the real pipeline in random-injection mode. This pilot leaned deterministic because the state of "even the known incident history isn't asserted" had to be fixed first. Once the known holes are plugged, those 2 entry points are where I'll expand.

The pattern — the fault catalog becomes the test spec

The workflow is 3 steps.

Catalog: classify the history of production incidents into fault classes (failure taxonomies) and diff them against existing tests. The untested classes become the fault catalog
RED: for each fault, assert in a test how the system should behave when it's injected. The current implementation fails it (this is TDD's RED)
GREEN: land the minimal guard that passes the test, in the same PR as the test

The point is that the fault catalog doubles as a spec. Desired behaviors — "a timeout should leave error_kind=timeout in telemetry," "malformed-shape JSON should abstain with a reason code" — get pinned down as executable tests.

The only difference from ordinary TDD is where the inputs come from. Instead of writing tests from functional requirements, you write tests from incident history — everything else reuses the RED → GREEN discipline as-is.

Build the fault catalog from incident history, not imagination

If you enumerate "which faults to inject" from imagination, you tend to mass-produce tests for failures that never actually happen. I started from 5 past production incidents (silent truncation from context-length overflow, mid-generation cutoff with done_reason=length, deduplication failing to fire, an external API rate limit, and a scraping target turning into a CAPTCHA) and swept for "failures in the same family that existing tests don't cover."

One caveat: the 5 incidents and the fault classes are not 1:1. What I extracted from the history is the shared family — "when the LLM or external I/O returns something unexpected, the pipeline silently degrades" — and from that family I derived the 5 classes that existing tests didn't cover.

Here are the resulting 5 classes.

#	Fault class	State of existing tests
F1	Read-timeout mid-generation	Only `ConnectionError` covered. Mid-stream timeout untested
F2	Embedding API failures — transport-level (429, timeout) and content-invalid success responses (mismatched dimensions, missing rows)	Untested
F3	Syntactically valid JSON that violates the expected schema (wrong top-level type, wrong keys, non-string elements)	Untested
F4	HTTP 429 from the LLM server itself	Only covered on a different external API client. LLM backend side untested
F5	Flapping (alternating / consecutive success-failure sequences)	Only single failure→recovery covered

Before and after:

Metric	Before	After
Tests directly injecting the fault classes	0/5 (per the table above, only partial coverage of adjacent cases)	5/5 (32 deterministic fault injection tests)
Telemetry failure-type discrimination	None (everything is `outcome="error"`)	7 kinds of `error_kind`
Observability of extraction failures	Silently missing	3 reason codes + per-reason aggregate summary
Full suite	—	1832 passed / 1 skipped

The test count of 32 is measured with pytest --collect-only (every number in this article was re-measured at the time of writing).

Implementation — 2 replacement points and a determinism discipline

Pin injection to 2 existing seams

Injecting faults requires a seam (a joint where the test side can swap the implementation without rewriting production code). I used only 2 existing ones and decided to add zero injection hooks to production code. The pipeline under test does not know it is being tested.

The LLM backend Protocol — implement a ChaosBackend on the test side and swap it in
The requests HTTP layer — fake HTTP responses with the responses library

ChaosBackend is driven by a schedule — a list of "which fault fires on which call." The chaos in class and file names is a leftover from the naming at inspiration time. Also, FAULT_VOCABULARY in the code is the minimal unit of injection (primitives), at a different granularity from the F1–F5 fault classes — one class is tested through combinations of several primitives.

```python:tests/chaos.py (excerpt, simplified — see the public skill in Related Links for the full version)

Fault vocabulary is an ordered tuple (unordered would break seed-derived determinism)

FAULT_VOCABULARY = (OK, NONE, EMPTY, EXC_TIMEOUT, EXC_CONNECTION, TRUNCATED, SHAPE_VIOLATION)

@dataclass
class ChaosBackend: # conforms to the LLMBackend Protocol
schedule: List[str] = field(default_factory=list)
calls: List[dict] = field(default_factory=list)

@classmethod
def from_seed(cls, seed: int, n: int, weights=None) -> "ChaosBackend":
    rng = random.Random(seed)
    vocab = list(weights.keys()) if weights else list(FAULT_VOCABULARY)
    wts = list(weights.values()) if weights else None
    return cls(schedule=rng.choices(vocab, weights=wts, k=n))

def generate(self, prompt, system, num_predict, format, *, temperature=1.0, think=False):
    idx = len(self.calls)
    self.calls.append({"prompt": prompt, "num_predict": num_predict})
    fault = self.schedule[idx] if idx < len(self.schedule) else OK
    if fault == NONE: return None
    if fault == EXC_TIMEOUT: raise requests.exceptions.ReadTimeout("chaos")
    if fault == TRUNCATED: return BackendResult(text=self._ok_text(idx), finish_reason="length")
    ...




A schedule can be built as an explicit list or derived from a seed — either way, **you can inspect its contents before the run**. This is not "break things randomly"; it's "breaking under this exact sequence is the spec."

### Run hypothesis under a deterministic profile

Instead of enumerating fault combinations by hand, I use property-based testing (asserting properties that must hold for any input, rather than individual input/output examples). To keep CI from going flaky, though, hypothesis is pinned to deterministic mode.



```python:tests/conftest.py (excerpt)
# database=None only disables the example DB. The constants/unicode caches
# need HYPOTHESIS_STORAGE_DIRECTORY relocated (before hypothesis is imported)
os.environ.setdefault("HYPOTHESIS_STORAGE_DIRECTORY", str(_TEST_HOME / ".hypothesis"))
from hypothesis import settings
settings.register_profile("ci", derandomize=True, max_examples=50, deadline=None, database=None)
settings.load_profile("ci")

derandomize=True generates the same case sequence every run. I verified this by running the full fault injection suite twice in a row and confirming identical output.

The other determinism rule: latency faults are expressed by injecting a ReadTimeout exception, not by real sleeps. On this call path, the observable consequence of a timeout reduces entirely to "catch and handle ReadTimeout," so within that scope exception injection verifies the same thing — and the tests run in 0 seconds. The real-sleep-plus-short-timeout approach goes flaky, so I avoided it. Note that behaviors that only surface in real time cannot be verified this way — e.g. cleanup of partial output mid-stream — so if that's your target, you need separate measures.

Writing RED — assert the desired guard behavior first

Here is the RED for F3 (valid JSON, wrong shape).

```python:tests/test_distill_chaos.py (excerpt)
class TestParsePatternsShapeViolationFuzz:
@given(raw=non_patterns_json())
@example(raw='{"patterns": [123]}') # pins the real str() promotion bug
@example(raw="null") # pins the json.loads("null") is None trap
def test_wrong_shape_abstains_with_no_patterns(self, raw):
patterns, mode = _parse_patterns(raw)
assert mode == "shape_violation"
assert patterns == []




`@given` asserts the property "any shape-violating JSON should cause abstention," and `@example` pins known failure shapes as permanent regression tests. Those 2 `@example`s are exactly silent failures ① and ② in the next section.

## The 3 silent failures the first application surfaced

The moment I wrote the RED tests, 3 holes in the existing implementation were exposed. ① and ② are the kind where the processing result silently degrades; ③ is on the observation-channel side — the failure itself gets recorded, but the type information needed for root-cause analysis is silently lost.

### ① Numbers promoted to strings let schema violations sail through

- **Symptom**: when the LLM returned non-string elements like `{"patterns": [123]}`, the old implementation stringified each element with `str(item)`, so `"123"` could pass as a legitimate pattern
- **Why it's silent**: the JSON is valid, so parsing succeeds, and `str()` promotion raises nothing. No trace of the schema violation is left anywhere
- **Guard**: require `isinstance(item, str)` on every element; on violation, abstain with the `shape_violation` reason code. Regression pinned with `@example(raw='{"patterns": [123]}')`

### ② `json.loads("null")` returns `None`

- **Symptom**: when the response body is JSON `null`, `json.loads` returns `None` without raising. The old implementation used "parse result is `None` = parse failed" as its check, so valid JSON was treated as a parse failure and misrouted into a fallback path (bullet-list text scanning) it should never have entered
- **Why it's silent**: the fallback path legitimately exists as a rescue for non-JSON responses, so a misroute is indistinguishable in the logs from a normal fallback
- **Guard**: introduce an identity sentinel, `_JSON_PARSE_FAILED = object()`, separating "parsing failed" from "`None` was parsed" at the type level

### ③ Telemetry flattens every failure type into `error`

- **Symptom**: whether it was a 429, a timeout, a connection failure, or a bad body, telemetry only recorded `outcome="error"`, making offline failure analysis impossible
- **Why it's silent**: the error itself is recorded, so "something failed" is visible. What's flattened is the **type** — a gap you can't notice until you actually attempt an incident investigation
- **Guard**: add `_classify_request_error` to classify exceptions, and add an `error_kind` field to failure rows only (7 kinds: `timeout` / `connection` / `http_<status>` / `bad_json` / `bad_url` / `request_error` / `backend_exception`). The change is additive — the existing `outcome` value set is untouched — so analysis code over past logs keeps working

What the 3 share: they are **abnormal paths wearing a happy-path face**. Bugs that raise exceptions get caught by ordinary tests, but all 3 of these "run to completion while the result or the record goes quietly wrong" — no amount of extra happy-path tests would have caught them. Only RED tests that inject faults and assert the desired way of failing expose these holes.

<details><summary>2 gotchas (hypothesis cache / circuit breaker × property tests)</summary>

**`.hypothesis/` appears even with `database=None`**

Even with `database=None` in the profile, caches like `.hypothesis/constants/` show up at the repo root. As the [settings reference](https://hypothesis.readthedocs.io/en/latest/reference/api.html) says, `database=None` only suppresses the example DB; the constants cache and friends are written unconditionally under `HYPOTHESIS_STORAGE_DIRECTORY`. The fix is to point that environment variable at a test tempdir **before** importing hypothesis (included in the code sample above).

**The circuit breaker breaks property-test predictions**

I asserted the property "success count = number of OKs in the schedule," and it broke only on schedules with 5 consecutive failure faults. The cause was the circuit breaker in production code (a mechanism that cuts off calls after consecutive failures; in this project it opens after 5 in a row). Once the breaker opens, subsequent OK calls never reach the backend, so the exact-count property can't hold even though the behavior is correct. I handled it in 2 layers: exclude breaker-tripping schedules from the exact-count property via a `trips_circuit()` filter, and keep a weaker property — "no schedule ever crashes" — over all schedules.

</details>
## Wrap-up — turning post-hoc debugging into up-front spec

- Silent failures don't get caught by happy-path tests. Only tests that **inject faults and assert the desired way of failing** catch them
- Build the fault catalog from **production incident history**, not imagination. 5 incidents → 5 untested fault classes → 32 deterministic fault injection tests
- For a single-process local LLM app, no daemon or proxy needed: **hypothesis + responses + a swappable backend** sit right on top of pytest
- The determinism discipline (`derandomize` / seed-derived schedules / no real sleeps) removes the 2 big flakiness sources — randomness and real time — from injection tests (measured: identical output across 2 consecutive runs)
- Land the guard in the **same PR** as the test. The fault schedule is the spec; the guard is the implementation that satisfies it
- Chaos engineering is the inspiration, not the classification. What I borrowed is 2 things — "build the catalog from real-world events" and "assert steady state through an observation channel" — while production experiments and unknown-space exploration are deliberately deferred (nightly randomization and sandbox experiments are the entry points for extending toward chaos)

The pattern from this work (fault vocabulary, ChaosBackend, hypothesis profile, RED templates) is published as a generalized Claude Code skill. If you want to bring this into your own pipeline, that's the entry point.

## Related Links

- [chaos-tdd-fault-injection](https://github.com/shimo4228/chaos-tdd-fault-injection) — the public skill generalizing this article's pattern (called "chaos-TDD" in the repo and its ADRs)
- [ADR-0077: Chaos-TDD Fault Injection](https://github.com/shimo4228/contemplative-agent/blob/main/docs/adr/0077-chaos-tdd-fault-injection.md) — primary source for the design decisions (English; a Japanese version sits in the same directory)
- [Previous article: Why Did My Agent Decide That? 3 Observability Patterns](https://dev.to/shimo4228/why-did-my-agent-decide-that-3-observability-patterns-ami)
- [Building an Autonomous Agent on an M1 Mac, on Purpose](https://zenn.dev/shimo4228/articles/small-llm-by-choice) — hub of the small-LLM series this article belongs to (Japanese)
- [hypothesis documentation](https://hypothesis.readthedocs.io/en/latest/) / [responses](https://github.com/getsentry/responses)
- [Chaos Toolkit](https://chaostoolkit.org/) / [toxiproxy](https://github.com/Shopify/toxiproxy) — chaos tools for distributed systems (not adopted here due to the mismatch in target scale)
- [agent-chaos](https://github.com/deepankarm/agent-chaos) — prior OSS for chaos engineering on AI agents (prior art referenced for fault classification)
- [Author's GitHub](https://github.com/shimo4228) — other repositories and tools

LLM-as-Judge Shouldn't Aggregate Scores: Binary Checks as Evidence, One Holistic Verdict

shimo4228 — Tue, 14 Jul 2026 13:00:09 +0000

What this article gives you: How to design LLM quality evaluation around "binary checks + a named verdict" instead of numeric scores. Includes a judge prompt structure you can reuse by swapping in your own evaluation target.

Having an LLM evaluate deliverables is called LLM-as-judge. Once you introduce it, most people run into the same three walls:

Ask for a 5-point rubric score and the same input oscillates between 3 and 4 across runs
Gate pass/fail on a total-score threshold and one fatal flaw gets diluted by high scores on everything else — and slips through
Read the scores back later and nobody can explain why something got a 3.5

This article introduces a design pattern that avoids all three at once. The core fits in one line:

Collect evidence with binary Yes/No checks, have the judge pick one verdict from named labels as a holistic judgment, and never aggregate anything into a score.

I've been running this pattern for about four and a half months in a Claude Code skill-quality audit (/skill-stocktake) and a knowledge-extraction quality gate (/learn-eval). How I built the audit command — the trial and error that led me to throw out numeric rubrics — is covered in my previous article.

This article is the sequel. It generalizes that design into a reusable LLM-as-judge pattern, including the episode where I tested my own design while it was in production — and rebuilt it.

Assumptions

Audience: anyone who wants to automate quality evaluation of deliverables (documents, code, data) with an LLM
The examples use Claude Code skill evaluation, but the pattern itself is agnostic to evaluation target, model, and tooling
Terminology: verdict = the final output of an evaluation. It takes the form of picking exactly one from named options such as "Keep / Retire". The JSON examples in this article use a verdict field
Terminology: holistic judgment = a single conclusion reached by looking at the whole, without going through per-dimension scoring

The pattern at a glance — three principles

Evaluation target
  → ① Binary checks: decompose into Yes/No questions
  → No answers = evidence
  → ② Holistic judgment: pick one named verdict
  → ③ No aggregation: scoring and averaging are forbidden
  → Downstream code / humans branch on the verdict

Principle	Do	Don't
① Checks are binary	Decompose criteria into questions answerable with Yes/No	"Rate specificity on a 5-point scale"
② The verdict is a named label	Look at the whole and pick one from `Keep / Improve / Retire` etc.	"Total is 12 points, so it passes"
③ No aggregation	Have the No answers listed as the grounds for the verdict	Using the Yes ratio (satisfaction rate) as a quality metric

① Make the checks binary Yes/No

Instead of "how specific is this skill, out of 5?", ask "is there a concrete command example you can run immediately after reading? Yes/No."

Binary questions have two properties that numeric scoring lacks:

Verifiable: "Is there a command example?" can be settled black-and-white by looking at the text. "Specificity: 3 points" cannot be verified at all
Evidence: a No answer points directly at what is missing. It feeds straight into an improvement list

In the skill audit I run in production, every target goes through these four questions:

- [ ] Actionability: are there steps, commands, or worked examples you can run immediately?
- [ ] Scope fit: do the name, trigger, and body agree (not too broad / too narrow)?
- [ ] Uniqueness: does it avoid overlapping with other skills in the same batch?
- [ ] Currency: do the referenced paths, CLI flags, and URLs still exist today?

② Have the verdict picked from named labels, as a holistic judgment

Holistic judgment is what LLMs are good at. As I wrote in the previous article, LLMs are bad at scoring dimensions independently — they get pulled by the overall impression. So it's more honest to design for that from the start: "pick one, based on the overall impression."

The key is to name the verdict labels so they map 1:1 to the next action.

Use case	Verdict options	Corresponding action
Skill audit	`Keep / Improve / Update / Retire / Merge into [X]`	Keep / hand off to the improvement engine / technical refresh / delete / consolidate (with an explicit target)
Knowledge-extraction gate	`Save / Improve then Save / Absorb into [X] / Drop`	Save / fix then save / append to an existing skill / discard

A fixed set of options that downstream processing can branch on directly is more practical than a bare accept / reject (in programming terms, an enum — a mechanism that forbids any value outside the defined set. With free-form text you get drift like "Improve" vs. "improve slightly", and code can't branch on it). Designs like Merge into [X] that force a concrete target name make vague verdicts like "this feels like it overlaps with something" structurally impossible to write.

This separation — "the LLM picks from named options, code executes" — also acts as a firewall against judgment errors. If judging and executing are separate, an LLM's bad call can never corrupt system state directly (I run this as a pattern I call LLM judge + Code enforce).

③ Never aggregate the binary answers

This is the pattern's biggest fork in the road. Once you have binary checks, you'll be tempted to say "6 of 8 questions are Yes, that's 75%, above the 70% threshold, so it passes." Don't.

The reason is simple: a satisfaction rate dilutes a single fatal No. If one No says "the referenced file doesn't exist," the skill is broken even if the other seven answers are Yes. Averages and ratios convert that dominant No into a 12.5% deduction.

Do these two things instead:

Require the No answers to be listed as the grounds for the verdict (hidden Nos breed verdict drift)
If there is even one dominant No (a nonexistent reference, an unsupported claim, etc.), tip the verdict to the non-Keep side even if everything else is Yes

The skill-audit definition file states the principle in one sentence:

Evaluation is holistic judgment, not a numeric rubric — binary answers are evidence feeding the verdict, never aggregated into a score (a satisfaction ratio changes no decision and dilutes a single dominant No).

The verdict pressure-test — attack the verdict before you commit it

Binary checks plus holistic judgment already work, but one more quality device raises the verdict's reliability: before finalizing a provisional verdict, have the LLM itself generate and answer questions that try to refute it.

Procedure (applied only to non-Keep candidates):
1. Generate 1–3 Yes/No questions that would refute the provisional verdict
   e.g. if the provisional verdict is "Update (a referenced CLI flag is deprecated)"
   → "Doesn't that flag still exist in the current version's --help? Verify empirically"
2. Answer each question with one line of evidence (file read, path check, web search result)
3. Refutation succeeds (e.g. the flag still exists) → move the verdict back toward Keep
4. Defect confirmed (e.g. the flag was indeed removed) → the No items from the binary
   checks become the improvement list as-is

Two disciplines govern question design:

Atomicity: each question tests exactly one verifiable claim
Refutation-oriented: frame questions to seek refutation of the provisional verdict, not confirmation of it

One more discipline matters for re-judging after fixes: re-judge exactly once, with the same question set. If you rewrite the questions, you can no longer tell whether the fix worked or the bar just moved.

A copy-paste judge prompt structure

Here is the whole pattern in a form you can reuse by swapping in your own evaluation target.

You are a quality evaluator of {kind of evaluation target}. Evaluate using the following steps.

## Step 1: Binary checks (all questions mandatory)
Answer each question with Yes/No plus one line of evidence.
- Q1: {verifiable question, e.g. is there a runnable code example?}
- Q2: {verifiable question}
- Q3: {verifiable question}
- Q4: {verifiable question}

## Step 2: Target-specific refutation questions (only if the provisional verdict leans non-pass)
Pick one provisional verdict, generate 1–3 Yes/No questions that would refute it,
and answer each with one line of evidence.

## Step 3: Verdict
Choose exactly one of the following. Do not output scores or points.
- {verdict_label_1}: {meaning and next action}
- {verdict_label_2}: {meaning and next action}
- {verdict_label_3}: {meaning and next action}

In the grounds for the verdict, list every question answered No.
If there is even one dominant No ({domain-specific fatal condition}),
choose {non-pass verdict label} even if everything else is Yes.

## Output format
Follow the JSON schema below and output **JSON only**
(no explanatory text or Markdown before or after).

Receive the output as JSON and downstream code can process it directly.

{
  "verdict": "Improve",
  "evidence": [
    { "question": "Is there a runnable code example?", "answer": "No", "detail": "Steps are bullet points only; not a single command example" },
    { "question": "Do the referenced paths exist?", "answer": "Yes", "detail": "All 3 paths confirmed with ls" }
  ],
  "pressure_test": [
    { "question": "Does the refutation hold that the bullet-point steps alone are reproducible?", "answer": "No", "detail": "The argument spec in step 3 is ambiguous; not reproducible" }
  ],
  "reason": "The Actionability No is dominant. Adding a worked example to step 3 would make this Keep-worthy"
}

Since the verdict field only takes fixed options, the receiving code just branches with a switch statement. The No items in evidence can be handed to the next stage (a human or another LLM) as an improvement task list as-is.

The full design pattern, including this template, is published as a Claude Code Agent Skill at github.com/shimo4228/llm-as-judge. Copy skills/llm-as-judge into ~/.claude/skills/ and it fires automatically whenever you're designing a judge.

Same principles, different operation at different scales

Running this pattern at two scales taught me that the principles are shared, but the application conditions change with scale.

	Knowledge-extraction gate (learn-eval)	Skill audit (skill-stocktake)
Evaluation target	1 draft extracted from a session	The entire library (73 skills)
Refutation question generation	Unconditional (3–5 questions every time)	Non-Keep candidates only (1–3 questions)
Reason	N=1, so the cost is negligible	Generating for all 73 items wastes most of the shots

With a single evaluation target, go all-in — it costs nothing. At tens of items, reserve the expensive device (refutation questions) for the suspicious candidates only.

Does single-context evaluation lose accuracy? — testing my own design

Here begins the second half of this article. The skill audit above originally used a design of "load every skill into one context and evaluate." The logic: a long-context model can survey everything at once, so cross-skill duplication should also be visible.

Then an audit run under this design returned Keep for every single item. Not one defect, not one duplicate pair detected. Is that "the library is healthy" or "the evaluation is spinning idle"? To find out, I re-audited the same library two ways: splitting it into small batches of about 10 items, each close-read by an independent agent, and a probe dedicated solely to duplicate detection (the experiment design and caveats about the numbers are in the collapsible section at the end of this chapter).

Results — some dimensions degrade, others don't

What was tested	Result
Per-item defect detection (small-batch close reading)	12 of 73 items non-Keep (16%). The single context had missed all of them
Duplicate detection (dedicated probe)	True duplicates: 0 (all 17 candidate clusters were adjacent skills with documented role separation)

Half of the 12 missed items were mechanically detectable defects: 404 links, references to deleted files, deprecated CLI flags. The kicker: two files that no longer existed on disk had been given Keep verdicts.

Zero duplicates, on the other hand, turned out to be reality, not dilution (attention thinning as the context fills up). The single-context rationale — "you can see duplication because you survey everything at once" — had been fully replaced by a lightweight description scan plus targeted close reading.

The biggest lesson isn't context width — it's verification ownership

Tracing the mechanism of the misses revealed a cause deeper than context width itself. The audit definition contained a conditional verification instruction: "if a reference looks stale, check it." That "if it looks..." condition is itself a judgment that depends on the LLM's attention. When the context fills up and attention thins, conditional triggers default to "don't check."

The previous article's conclusion was "mechanical work to scripts, quality judgment to AI." This experiment redrew that boundary one level deeper: inside quality judgment there are deterministic parts hiding, and they belong in code. "Does this file exist?" is not a judgment — it's ls. The deterministic checks that had snuck into the judgment phase should have been moved to the code side as an unconditional, mandatory pre-pass. That is what "verification ownership" in the heading means: moving the execution owner of deterministic checks from the LLM's attention to code.

The revised design (v3.0) assigns each property to the environment that should check it:

Phase 0: Deterministic pre-pass (code-owned, unconditional)
         Reference-existence checks, ledger consistency.
         Anything that doesn't exist never reaches judgment.
Phase 1: Inventory
         (the layer offloaded to scripts in the previous article; unchanged)
Phase 2: Per-item scrutiny (small batches of ~10, parallel)
         Binary checks + verdict pressure-test + holistic judgment
Phase 3: Dedicated duplication probe (1 agent, all items)
         Lightweight description scan → close-read candidate clusters only

I reviewed the 12 detections one by one and fixed them the same day. For one of them, testing the proposed fix before applying it showed the command didn't actually work, so I swapped in a different form. Verify not just the verdict but the fix itself before committing it — the same pattern surfacing outside the evaluation loop.

Experiment design and notes on the numbers (for the curious)

Baseline: the audit result from the all-in-one-context run. On the ledger it was 84 entries, all Keep. The actual files on disk numbered 73 — the gap was a ledger bug: the same skill double-counted under two different keys, plus entries for files that no longer existed. That the ledger was this dirty had gone unnoticed by everyone at that point. The follow-up experiments target the 73 real files
Treatment A (small-batch close reading): split the 73 items into 7 batches of 10–11, each close-read by an independent subagent using the same criteria (binary checks → holistic judgment). Reference existence had to be verified with ls etc., stated explicitly
Treatment B (dedicated duplication probe): one agent reads all 73 descriptions, enumerates candidate clusters → close-reads the bodies side by side to decide
Bias control: the agents were not told that the baseline had returned all-Keep
An honest note on confounding: Treatment A's prompt explicitly required "verify reference existence with ls", so part of the difference from baseline is attributable to "forced verification" rather than "context width." However, detections that required semantic judgment — description/body mismatches, overly abstract writing, scope drift — cannot be explained by forced verification, and I read those as the effect of attention density: 10 items per batch versus 73

Why no scores — the boundary the proposing paper drew itself

The "decompose evaluation criteria into binary questions" approach has a research lineage: CheckEval (arXiv:2403.18771, 2024), TICK (arXiv:2410.03608, 2024), and the later BinEval, "Ask, Don't Judge" (arXiv:2606.27226, 2026). My setup had the "binary checks + holistic judgment" skeleton first; I learned of this lineage afterward and folded it into each skill's References. BinEval proposes decomposing into atomic Yes/No questions and wiring failed questions directly into improvement feedback — the dynamic generation of refutation questions and the "No answer = improvement list" wiring are the parts I ported from it after it came out.

What's interesting is BinEval's own limitations section. The paper self-reports two reservations. First: binary decomposition is reliable only up to concretely checkable criteria, like "is there a code example?" Subjective quality like "is this well written?" is something humans judge by looking at the whole rather than stacking individual checks — it cannot be fully decomposed into a bundle of Yes/No questions (less reliable). Second: "it assumes the fraction of satisfied questions corresponds roughly linearly to quality, and this does not always hold." In other words, using the satisfaction rate of decomposed questions as a score comes with a caveat from the proposing paper itself.

Flip that limitations report around and it is exactly a division-of-labor boundary. Concretely checkable criteria can be decomposed into Yes/No — so let the checks gather evidence. The final value judgment belongs to the subjective "is this good writing?" layer, which reduces neither to a bundle of questions nor to a satisfaction rate — so that layer stays a holistic judgment. The three principles in this article are that self-reported boundary, transcribed directly into a design.

Summary

Design LLM-as-judge evaluation around three principles: "binary checks = evidence," "verdict = pick one named label holistically," "no aggregation"
Make verdict labels a fixed set mapping 1:1 to next actions, and require every No answer to be listed as grounds for the verdict
A verdict pressure-test before finalizing raises reliability. Re-judge exactly once, with the same question set
In my experiment, per-item scrutiny diluted as the context filled up (16% missed). Don't put deterministically verifiable checks on the LLM's attention — move them to unconditional, code-owned execution
The reason not to turn "satisfaction rate" into a score comes from the limitations self-reported by the binary-decomposition paper itself (BinEval): decomposition works up to concrete criteria; the value-judgment layer stays holistic

The previous article's division of labor — "mechanical work to scripts, quality judgment to AI" — got one level deeper this time. Inside judgment, too, there are parts that belong in code. When you design an LLM-as-judge, start by asking, before anything reaches the judge: "is this really a judgment, or is it ls?"

DEV Community: shimo4228

AI Review Kept Creating Work: Why I Deleted 4,541 Lines

The previous evening, I rebuilt task management

Task management was creating its own tasks

The first question should not have been how to fix it

Even a HIGH finding needs its premise checked

Whose work grows when you create a task?

Machine-readable layer: technical record and audit contract

Read-only audit prompt for your coding agent

Sources and references

Related links

Can Six-Month-Old AI Code Survive Today's Review? A 25-Bug Triage

The setup: two generations, six months apart, head to head

What I did: run two review lineages in parallel

Verdict 1: fatal or trivial?

Verdict 2: why did the tests stay green for half a year?

What the audit side revealed: the two models looked at different places

How much of the scaffolding could be dissolved?

Wrap-up: how to read the report card

Sources

Related links

Claude Code's Memory Has No Vectors — Try ADRs Before Memory RAG

Decomposing RAG — who makes the relevance judgment

The decision rule — when RAG is needed, when it isn't

Evidence 1: Claude Code's implementation — search is demoted to a tool

Evidence 2: embeddings can't see "the concrete difference" — my failure

Evidence 3: "select and inject" breaks the cache — a prefix-cache A/B

Objection: "you just moved the cost to write-time discipline"

My own agent runs on the full benefit of RAG

Why "ADR" specifically

Summary

Related links

Give Claude Code a Second Harness

Prerequisites

The configuration directory switches only half the harness

The human decision is where to draw the boundary

Give this article to your agent and ask for a plan

Judge success by the load source

Summary

References

Related links

Banned from Wikidata Overnight — I Believed Every Edit Was Compliant, but All 109 Items Were Deleted as Promotion

What I was building, and what happened

The cause: judgment is per account, not per edit

Where the misunderstanding came from: "Wikidata allows self-registration"

Same-day response: withdraw, purge references, apologize

Prevention: the record goes into an ADR, the behavior into rules

Conclusion: separate the effect from the pathway

Related links

What Humans Should Approve Is Intent, Not the Diff — A Decision Table for Agent Approval Gates

Assumptions

Gates have two axes

The decision table: what you show is decided by the target

Escalation rule: irreversibility overrides the target category

In one line

Applying it: what was actually happening in five places

The same gate had two competing sources of truth

"A human inspects the artifact" was scattered across five places

Reinforcement 1: the control plane and "artifacts that produce the evidence"

So how should you present it

Reinforcement 2: on FAIL, show the detection line

"Not shown" is not "not kept"

Reinforcement 3: don't let the intent summary be free-form

Only one item is required: the Divergence from plan declaration

Cap the number of stops too: one gate per unit of work

Raise the granularity and the bottleneck moves

Porting it into your own conventions: three steps

The limits of "the machine holds it"

Back to the opening question

Related links

Opus 5 Changed How Rules Should Be Written — Audit Yours

Background: the official guidance on rule-writing changed

Premise: the ground truth is the system prompt and tool descriptions

Step 1: capture the current runtime layer from a live session

Step 2: cross-check against your rules and sort into 3 categories

Conflict example 1: a no-plan-mode rule

Conflict example 2: a rule referencing a ghost setting

Redundancy example: a scope-adherence rule

Drift example: a confidence threshold for review

Step 3: don't treat "conflict = bad" — run each item through a decision frame

Only one item is required: the `Divergence from plan` declaration