DEV Community

Lily
Lily

Posted on • Originally published at dev.to

When Claude Detects "Why Isn't This Fixed Yet?" and Repairs Its Own CLAUDE.md: A Self-Improvement Loop Driven by Frustration Signals in Transcripts

#ai #automation #shell #devops

In a previous post, I extracted skill usage stats from my conversation logs. This time I'm going the other way: a weekly loop that reads the transcripts to detect whether my Claude Code environment is degrading, and when a threshold is crossed, lets claude -p fix things itself.

~/.claude/scripts/cc-self-audit.sh (177 lines) runs every Sunday at 8:30 via launchd. It collects five numbers, compares them against thresholds, and if everything is green it sends a one-line ✅; if anything is red, it self-repairs, independently re-measures, and reports to Discord. About 70% of the implementation effort went into "why a naive grep doesn't work here."

The problem: nobody was detecting environment degradation

A performance audit on 2026-07-11 uncovered a triple whammy: agents that were supposedly moved out of the way were still being injected — 99 of them; CLAUDE.md had silently bloated to 228KB; and a Stop hook was firing dozens of times per week.

Each of those is fixable on its own. The real problem was that there was no mechanism to notice the "fixed it, but it broke again" cycle. A human doing a weekly visual check doesn't last, so I made a script do it.

Overall design: 5 metrics × thresholds × independent re-measurement 

launchd（日曜 8:30）
  → cc-self-audit.sh
      ① collect()  →  5数値（静的2 + 動的3）
      ② breaches() →  閾値超過を列挙
      ③ 緑 → ✅通知・終了
      ④ 赤 → claude -p（sonnet・MAX枠）が ~/.claude 内を修正
      ⑤ collect() 独立再実行 → 数値改善を確認（自己申告は信じない）
      ⑥ Discord #01_alerts へ報告
Enter fullscreen mode Exit fullscreen mode

(The flow: launchd on Sunday 8:30 → ① collect() gathers 5 numbers (2 static + 3 dynamic) → ② breaches() lists threshold violations → ③ green: notify ✅ and exit → ④ red: claude -p (sonnet, on my MAX plan) fixes things inside ~/.claude → ⑤ collect() re-runs independently to confirm the numbers improved (self-reports are not trusted) → ⑥ report to Discord #01_alerts.)

Thresholds (overridable via env vars):

Metric Threshold Variable
inject_bytes 40,000 B SELF_AUDIT_TH_INJECT
agents_loaded 60 files SELF_AUDIT_TH_AGENTS
stopspam 15 /week SELF_AUDIT_TH_STOPSPAM
frustration 8 /week SELF_AUDIT_TH_FRUST
toolerr 400 /week SELF_AUDIT_TH_TOOLERR

Static metrics: injected bytes and agent count 

inject_bytes=$(( \
  $(find "$HOME/.claude/rules" -name '*.md' -print0 2>/dev/null | xargs -0 cat 2>/dev/null | wc -c) + \
  $(cat "$HOME/.claude/CLAUDE.md" 2>/dev/null | wc -c) + \
  $(cat "$HOME/CLAUDE.md" 2>/dev/null | wc -c) + \
  $(cat "$HOME/.claude/projects/-Users-matsubara/memory/MEMORY.md" 2>/dev/null | wc -c) ))
agents_loaded=$(find "$HOME/.claude/agents" -name '*.md' 2>/dev/null | wc -l | tr -d ' ')
Enter fullscreen mode Exit fullscreen mode

The find ~/.claude/agents -name '*.md' is deliberately recursive so it re-detects a failure mode I've already been burned by: agents that were "archived" into dot-prefixed subdirectories (like .backup/) getting loaded again. If agents_loaded exceeds 60, I treat it as "the archiving has leaked."

Dynamic metrics: reading only human-typed messages out of the transcript JSONL

This is the core of the design. Targeting "up to 200 JSONL files over 100KB that were updated since the last run," it counts stopspam (actual firings of the audit hook) and frustration (the user's frustration words).

STOP_MARKER = 'Stop hook feedback:\n[~/.claude/hooks/self_audit_stop.sh]: '
FRUST_RE = re.compile(r'何回も言|いい加減にし|嘘つ|舐めんな|なんで治らん|最悪やろ|頭悪い')

for line in fh:
    if STOP_MARKER in line:
        stopspam += 1
    try:
        o = json.loads(line)
    except Exception:
        continue
    if o.get('type') != 'user' or o.get('isMeta'):
        continue                          # ← ここが肝
    content = (o.get('message') or {}).get('content')
    texts = [content] if isinstance(content, str) else \
        [b.get('text', '') for b in content if isinstance(b, dict) and b.get('type') == 'text'] \
        if isinstance(content, list) else []
    if any(FRUST_RE.search(t) for t in texts):
        frustration += 1
Enter fullscreen mode Exit fullscreen mode

(The regex matches Japanese phrases like "how many times do I have to say this," "give me a break," "liar," "why isn't this fixed yet," and so on; the # ← ここが肝 comment marks the crucial line.)

Only blocks with type=user that are not isMeta are examined — in other words, only messages a human actually typed. Why that matters is the next section.

The false-positive incident of 2026-07-12

The first implementation looked like this.

# 旧: 素朴なgrep
stopspam=$(echo "$files" | xargs grep -h -c 'self_audit_stop' 2>/dev/null | awk '{s+=$1} END{print s+0}')
frustration=$(echo "$files" | xargs grep -hE -c 'なんで治らん|いい加減' 2>/dev/null | awk '{s+=$1} END{print s+0}')
Enter fullscreen mode Exit fullscreen mode

history.jsonl still holds the measured values from back then.

{"ts":"2026-07-11 22:37:28","inject_bytes":21798,"agents_loaded":48,"stopspam":6,"frustration":4,"toolerr":2}
{"ts":"2026-07-12 08:30:06","inject_bytes":21966,"agents_loaded":48,"stopspam":40,"frustration":18,"toolerr":67}
{"ts":"2026-07-12 08:40:03","inject_bytes":21966,"agents_loaded":48,"stopspam":58,"frustration":32,"toolerr":68}
Enter fullscreen mode Exit fullscreen mode

The evening of 07-11 was normal (GREEN). The morning of 07-12, stopspam jumped from 6 to 40 and frustration from 4 to 18 — RED. Then claude -p attempted a fix, the independent re-measurement ran, and the numbers had climbed further, to 58/32.

The cause: grep was doing a string search over the entire JSONL. The transcripts contained:

  • Diffs from the repair work itself (the hook script's source landing wholesale in old_string/new_string)
  • Contents of files opened with the Read tool (files that happened to contain frustration words)
  • Echoed-back tool_results (the previous audit's output being quoted by the next session)

All of it was matching. Of the 40 stopspam hits, actual hook firings were near zero; of the 18 frustration hits, only a handful were frustration words a human had actually typed.

A naive string grep cannot be used against transcripts. Unless you first determine "which role, which block" within the JSONL, quotations, tool output, and echoes of past logs all get counted. The resolution of your measurement design determines the quality of your audit.

The fix was switching to a JSONL parser. Only text blocks with type=user and isMeta: false (messages the user actually sent) are examined, and stopspam counts only lines matching the exact fixed format the harness actually injects. That excluded quotations, tool output, and echoes of past logs entirely.

Threshold breach → self-repair → independent re-measurement

The core of the prompt passed to claude -p when things go red:

PROMPT="あなたはClaude Code環境の自己監査・自己修復エージェント。

## 検知した違反
$BREACH

## 指示
1. ~/.claude 内だけを調査・修正する（プロジェクトコード・secret・plist削除は禁止）
2. 変更は1件ずつ ${CHANGELOG} に「日時/対象/理由/戻し方」を追記
3. 修正後に必ず 'bash ~/.claude/scripts/cc-self-audit.sh --collect-only' を実行し改善を数値で確認
4. 自分で安全に直せない項目は無理せず、最後に『要人間: 理由』と1行で出力
5. 最終出力は3行以内の要約のみ"
Enter fullscreen mode Exit fullscreen mode

(In English, the prompt says: "You are a self-audit / self-repair agent for the Claude Code environment. ## Detected violations: $BREACH. ## Instructions: 1. Investigate and fix only inside ~/.claude (project code, secrets, and plist deletion are forbidden). 2. For each change, append date/target/reason/how-to-revert to ${CHANGELOG}. 3. After fixing, always run bash ~/.claude/scripts/cc-self-audit.sh --collect-only and confirm improvement numerically. 4. Don't force items you can't safely fix yourself — end with a one-line 'Needs human: reason.' 5. Final output is a summary of at most 3 lines.")

The important part is instruction 3 — or rather, what happens outside it. Instead of only having claude -p re-measure itself, the harness also runs collect() independently and checks the numbers (from line 165):

# 独立再計測（自己申告は信じない）
AFTER=$(collect)
log "after: $AFTER"
echo "$AFTER" >> "$HISTORY"
STILL=$(echo "$AFTER" | breaches)
Enter fullscreen mode Exit fullscreen mode

If the AI says "fixed it" but the numbers haven't improved, a 🚨 goes out. When you use a Claude-built tool to repair Claude's own environment, trusting only the self-report is how you get hurt.

What the recorded numbers show

What the current history.jsonl tells us:

  • inject_bytes: 21,798–21,966 B (less than half the 40,000 threshold, GREEN)
  • agents_loaded: 48 (under the threshold of 60, GREEN)
  • The 07-12 red was a false positive (after the grep fix, values returned to normal)

The first two entries are the before/after of the manual --collect-only on 07-11; entries 3–4 are the 07-12 grep misdetection and the subsequent re-measurement. Production operation as a weekly job starts from 08-30.

launchd configuration 

<key>StartCalendarInterval</key><dict>
  <key>Weekday</key><integer>0</integer>  <!-- 日曜 -->
  <key>Hour</key><integer>8</integer>
  <key>Minute</key><integer>30</integer>
</dict>
Enter fullscreen mode Exit fullscreen mode

(Weekday 0 with the 日曜 comment means Sunday.)

RunAtLoad is false. If the job ran immediately on registration, the first measurement would happen in the "empty" state right after launchd starts. Instead, the first baseline measurement is done by manually running --collect-only to put one line into history.jsonl before registering the job.

Pitfalls I hit

  • Naive grep misdetected across the entire transcript → use a JSONL parser and target only type=user && !isMeta
  • The self_audit_stop match for stopspam also hit quotations of the hook script itself → count only the fixed marker string the harness actually injects (exact format match)
  • The AI reported "fixed" but the numbers hadn't changed → re-run collect() independently of the claude -p output, append to HISTORY, and only then judge
  • claude not found under launchd's minimal PATH → declare PATH explicitly in the plist's EnvironmentVariables (/opt/homebrew/bin + ~/.local/bin are required)
  • frustration pinned at zero can look like "nothing is happening" → "zero frustration" is a good sign, but check monthly that the thresholds and regex still track the real-world word set

Summary

  • Degradation of a Claude Code environment (agent overload, injection bloat, hook spam, user frustration) is measured weekly across 5 metrics, and when a threshold is crossed, claude -p self-repairs
  • Dynamic measurement without JSONL role checks (type=user && !isMeta) is riddled with false positives — naive grep against transcripts does not work
  • After a self-repair, the harness re-measures independently of the AI's self-report and confirms numerical improvement before notifying
  • "The resolution of your measurement design determines the quality of your audit" — the 2026-07-12 incident demonstrated this with real code

Next time, I'll walk through what claude -p actually changed when this audit fired, using real entries from the change log (self-audit-changes.log).

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Top comments (0)