Let Your AI Fix Its Own Broken Automation: Building an Unattended Watchdog

If you ship enough side projects, something in your automation is always quietly on fire — and you usually find out days too late. This is the next installment in my "automation foundation for mass-producing side projects" series, following "How I split Claude Code's memory into four layers" and "Running Claude Code and Codex together on one machine." This time I'll walk through the design and implementation of an unattended watchdog that lets an AI detect, repair, and verify broken automation scripts, and pushes changes to production without approval only when verification passes. Using real code as the anchor, I'll explain how to stack guardrails so the AI runs safely instead of "rampaging without limits."

Why "fix itself when it breaks" became necessary

When you mass-produce side projects, the number of always-on automation scripts keeps growing. Auto-posting affiliate articles, scrapers, delivering briefs to Discord, checking the review status of iOS apps. Each runs under launchd or cron, so when one breaks, days can pass without anyone noticing.

Monitoring and fixing everything by hand hits its limit quickly. But just "letting the AI fix it" carries risks: unintended code rewrites, committing secrets, or reporting a repair as done when it's actually still broken and pushing that to production.

So I designed self-repair.sh. The idea is simple.

Detect → Claude identifies the cause and fixes it → the watchdog independently runs verify
     → commit/push/deploy only when it passes
     → roll back the edits and notify a human if it fails

Don't trust Claude's self-report is the key point. Even if Claude says "I fixed it," the watchdog runs the verification script itself, and reflects the change only when it exits 0.

Overall structure of the watchdog

The files are laid out like this.

~/.claude/self-repair/
├── registry.tsv          # list of monitored projects
├── checks/
│   ├── <slug>-health.sh  # health check (exit 0=healthy, non-0=broken)
│   └── <slug>-verify.sh  # post-repair verification (exit 0=pass, non-0=fail)
├── state/
│   └── <slug>-YYYYMMDD.count  # today's attempt count
└── (the main script lives at ~/.claude/scripts/self-repair.sh)

~/.claude/logs/
└── self-repair.log

The main loop simply reads registry.tsv and calls process() for each project.

while IFS=$'\t' read -r slug dir mode _rest; do
  case "$slug" in ''|\#*) continue ;; esac
  [ -n "$ONLY" ] && [ "$slug" != "$ONLY" ] && continue
  dir="${dir/#\~/$HOME}"
  process "$slug" "$dir" "${mode:-full}"
done < "$REG"

The registry.tsv format is three columns: slug\tdir\tmode.

# slug          dir                    mode
affiliate       ~/dev/affiliate-factory full
brief-discord   ~/dev/brief             detect
article-pub     ~/dev/article-publisher full

mode can be full (detect + repair) or detect (detect and notify only). Projects that aren't under git management, or where auto-repair is dangerous, should be set to detect.

How to write the health check and verification scripts

The key is to completely separate health.sh and verify.sh.

health.sh decides "is it broken right now?" When it detects a problem it exits non-0 and prints the error context to stdout (this output is used in the repair request to Claude).

# Example: affiliate-factory-health.sh
#!/usr/bin/env bash
# Broken if no article was generated in the last 24 hours
LAST=$(find ~/dev/affiliate-factory/output -name '*.md' -newer ~/dev/affiliate-factory/output -mmin -1440 | wc -l | tr -d ' ')
if [ "$LAST" -eq 0 ]; then
  echo "過去24時間の記事生成数=0。generate.sh が失敗している可能性"
  exit 1
fi

verify.sh decides "does it actually work after the repair?" It can be stricter than health.sh, or identical. What matters is that a process independent of Claude's repair runs it.

# Example: affiliate-factory-verify.sh
#!/usr/bin/env bash
# Pass if it actually generates one article and finishes cleanly
cd ~/dev/affiliate-factory
timeout 120 bash generate.sh --dry-run

Seven guardrails: why unattended repair holds up safely

A comment at the top of the script says "the real safety is the guardrails." By stacking all seven guards, unattended repair becomes viable.

Guard 1: Independent verification (don't trust Claude's self-report)

The most important one. Inside process(), after calling Claude, the watchdog itself runs bash "$verify".

# After calling Claude...
local vrc=0
run_capped "$VERIFY_TIMEOUT" bash "$verify" >/tmp/sr-$slug-verify.log 2>&1 || vrc=$?

if [ "$vrc" -eq 0 ]; then
  # fixed → secret scan → commit → push
else
  # ❌ verification failed → roll back the edits
  restore "$dir" "$baseline"
fi

run_capped is a wrapper that runs with a timeout if gtimeout is available. It prevents verification from running forever.

run_capped() { # run_capped <timeout_sec> <cmd...>
  local t="$1"; shift
  if [ -n "$TIMEOUT_BIN" ]; then "$TIMEOUT_BIN" "$t" "$@"; else "$@"; fi
}

Guard 2: Immediate rollback on failure

If verify is non-0, restore() returns to the baseline.

restore() { # restore <dir> <baseline_sha>
  local dir="$1" base="$2"
  (cd "$dir" && {
     git reset -q --hard "${base:-HEAD}" 2>/dev/null || true
     git clean -fdq 2>/dev/null || true
     git stash list 2>/dev/null | grep -q . && git stash drop -q 2>/dev/null || true
   }) || true
}

git reset --hard returns to the pre-commit state, git clean -fdq removes files Claude newly added, and anything stashed away is dropped too. This assumes a git repo, so non-git projects need to be in detect mode (the script also checks with git rev-parse and skips auto-repair if it's not a git repo).

Guard 3: Secret scanning

Before pushing, it scans the staged diff with regular expressions.

secret_in_staged() { # secret_in_staged <repo>
  local repo="$1"
  if (cd "$repo" && git ls-files --cached | grep -qE '(^|/)\.env$'); then
    echo ".env がコミット対象"; return 0
  fi
  local hits
  hits=$(cd "$repo" && git diff --cached | grep -nE \
    'pk_[A-Za-z0-9]{6,}|sk_live_[A-Za-z0-9]|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{20}|xox[baprs]-[A-Za-z0-9]|-----BEGIN [A-Z ]*PRIVATE KEY-----|AIza[0-9A-Za-z_-]{20}' \
    2>/dev/null | head -3)
  if [ -n "$hits" ]; then echo "$hits"; return 0; fi
  return 1
}

If .env is under tracking, it's an immediate fail. Beyond that it detects Stripe's pk_/sk_live_, AWS's AKIA, GitHub Personal Access Tokens (ghp_), Slack tokens (xox), private key headers, and Google API keys (AIza).

This secret scan is built into the script itself because Claude's hooks don't fire when running under launchd. The point is to scan yourself instead of relying on hooks.

What actually scans before commit is secret_in_staged_after_add(). It runs git add -A, then scans for secrets, and if clean, commits as-is.

secret_in_staged_after_add() {
  local dir="$1" reason
  (cd "$dir" && git add -A) || true
  if reason=$(secret_in_staged "$dir"); then echo "$reason"; return 0; fi
  (cd "$dir" && git commit -q -m "fix(self-repair): $(date +%F) 自己修復による自動修正" 2>>"$LOG") || true
  return 1
}

Guard 4: Attempt cap

The same slug can be attempted up to MAX_ATTEMPTS (default 2) times per day. The state file is used as a counter.

MAX_ATTEMPTS="${SELF_REPAIR_MAX_ATTEMPTS:-2}"

attempts_today() { cat "$STATE/$1-$TODAY.count" 2>/dev/null || echo 0; }
bump_attempts() {
  local n; n=$(attempts_today "$1"); n=$((n+1))
  echo "$n" > "$STATE/$1-$TODAY.count"; echo "$n"
}

When the cap is reached, it skips repair and notifies via Discord that "a human needs to check this." This prevents a repair loop from spinning forever and melting tokens.

Guard 5: Cost cap

If the 5-hour block's output tokens exceed the threshold, it skips the entire run cycle.

BUDGET_CAP_TOK="${SELF_REPAIR_BUDGET_CAP:-380000}"

budget_ok() {
  local tok
  tok=$("$BUDGET_ADVISOR" 2>/dev/null | python3 -c 'import sys,json
try: print(int(json.load(sys.stdin).get("5h_output_tokens",0)))
except Exception: print(0)' 2>/dev/null)
  tok="${tok:-0}"
  if [ "$tok" -gt "$BUDGET_CAP_TOK" ]; then
    log "SKIP(budget): 5h_output_tokens=$tok > cap=$BUDGET_CAP_TOK"
    return 1
  fi
  return 0
}

BUDGET_ADVISOR is a separate script that returns Claude Code's usage in JSON format. It's the last line of defense against a billing runaway.

Guard 6: Scope restriction (build a wall with allowedTools)

Claude is restricted with --allowedTools on which tools it can touch, and made to edit only within the project directory.

out=$(cd "$dir" && printf '%s' "$PROMPT" | run_capped "$REPAIR_TIMEOUT" "$CLAUDE" -p \
      --model "$MODEL" --output-format text \
      --allowedTools "Read,Write,Edit,Bash,Grep,Glob" \
      --max-turns 40 2>&1) || rc=$?

I don't use --skip-permissions. The wall is built by narrowing the tools with --allowedTools and making the scope explicit in the prompt.

Guard 7: Push destination owner verification

It won't push unless the destination GitHub remote is your own account. This prevents unintentionally writing to a third party's fork.

ALLOWED_OWNERS="bokuwalily"

push_if_safe() {
  local repo="$1" url owner
  url=$(cd "$repo" && git remote get-url origin 2>/dev/null || true)
  [ -z "$url" ] && { log "  push skip: origin未設定（ローカルcommitのみ）"; return 0; }
  owner=$(printf '%s' "$url" | sed -E 's#.*github.com[:/]+([^/]+)/.*#\1#')
  case " $ALLOWED_OWNERS " in
    *" $owner "*) ;;
    *) log "  push skip: owner=$owner は許可外（第三者repo保護）"; return 0 ;;
  esac
  if (cd "$repo" && git push -q origin HEAD 2>>"$LOG"); then
    log "  pushed → $owner"
  else
    log "  push失敗（commitは保持）"
  fi
}

How to craft the repair request to the AI

The prompt to Claude is assembled dynamically inside process(). The key is to pass health.sh's output and recent logs as context, and explicitly state what must not be done as constraints.

local PROMPT="あなたは自律修復エージェントです。自動化『${slug}』(ディレクトリ: ${dir})が壊れています。原因を特定し、**最小の変更**で直してください。

# 制約（厳守）
- 触ってよいのは $dir 配下のファイルのみ。スコープを広げない。新機能を足さない。
- .env や秘密情報を読み出して出力・コミットしない。
- 直したら必ず自分で検証コマンド \`bash $verify\` を実行し、exit 0 になることを確認してから終了する。
- 直せない/原因が $dir 外にあるなら、推測で書き換えず『UNFIXABLE: <理由>』とだけ述べて終了する。

# 健全性チェックの出力
$hctx

# 最近のログ（末尾）
$rlog

直して、検証まで通してください。"

The important part is the UNFIXABLE: exit keyword. In cases that can't or shouldn't be fixed, it prevents the runaway behavior of "just rewrite something anyway." When the cause is outside $dir (a dependency API outage, a cron config, etc.), the design keeps Claude from touching it.

The health check output is trimmed to head -40, and the logs to tail -60, before being passed. If the context is too large, the judgment wobbles.

Recording the baseline and stashing uncommitted changes

Before repairing, it records the current HEAD (the baseline). Because uncommitted changes would make the restore dirty, it first stashes them with git stash -u.

local baseline; baseline=$(cd "$dir" && git rev-parse HEAD 2>/dev/null || echo "")
(cd "$dir" && git stash -u -q 2>/dev/null) || true

-u is the option that also stashes untracked files. If Claude fails to repair, restore() returns to git reset --hard baseline and then drops the stash too.

"Detect only" mode

Setting mode=detect makes it "just detect that something is broken and notify," without calling Claude. To notify only once per day, it creates a flag file under state/.

if [ "$mode" = detect ]; then
  if [ ! -f "$STATE/$slug-$TODAY.detected" ]; then
    local hd; hd=$(head -3 /tmp/sr-$slug-health.log 2>/dev/null | tr '\n' ' ' | cut -c1-200)
    notify alerts "🩺 $slug がサイレント失敗（本日成功の形跡なし）。自動修復対象外＝人間の確認が要ります。$hd"
    touch "$STATE/$slug-$TODAY.detected"
  fi
  return
fi

Projects not under git management, projects where repair has side effects (requiring writes to external APIs), and projects with no verify.sh in place are safest left as detect. The script also automatically falls back to detect-equivalent behavior (notify only) when verify.sh doesn't exist.

[ -x "$verify" ] || {
  log "$slug: verify無し→無人修復は危険なのでskip(人間へ)"
  notify alerts "🛠 $slug が異常だが verify 未整備のため自動修復せず。要確認。"
  return
}

Running it from launchd

In actual operation, it runs periodically via a launchd plist. Because of the minimal-PATH problem, the script explicitly sets PATH internally.

<!-- ~/Library/LaunchAgents/com.lily.self-repair.plist -->
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.lily.self-repair</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/bash</string>
    <string>/Users/<you>/.claude/scripts/self-repair.sh</string>
  </array>
  <key>StartCalendarInterval</key>
  <dict>
    <key>Minute</key>
    <integer>30</integer>
  </dict>
  <key>StandardOutPath</key>
  <string>/Users/<you>/.claude/logs/self-repair-launchd.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/<you>/.claude/logs/self-repair-launchd.err</string>
</dict>
</plist>

Setting StartCalendarInterval to Minute: 30 runs it at 30 minutes past every hour. A 30-minute cadence is enough in most cases.

Because nvm isn't loaded when running via launchd, you need to either pass the full path to the claude command via the CLAUDE environment variable, or explicitly export PATH at the top of the script.

Pitfalls I hit

Forgetting git stash -u makes the restore dirty

If you let Claude repair while there are uncommitted work-in-progress files, untracked files remain even after git reset --hard. Unless you follow the flow of stash → repair → drop on failure, you'll create manual cleanup work for yourself.

Making verify.sh the same as health.sh is pointless

If you use the same script for health.sh and verify.sh, even a situation where "it happened to pass right after the repair" gets treated as a pass. verify.sh should include a test that actually runs the processing, so that even with --dry-run it exercises the full set of code paths.

Always build the secret-scanning regex yourself

Relying on GitHub's push protection means detection happens after the push. The point is to scan yourself before pushing. Also, since assignments to environment variables don't get caught by ordinary regexes, I scan git diff --cached directly to detect dangerous literals.

Without gtimeout, the timeout doesn't work

macOS's timeout command comes in via Homebrew as GNU coreutils' gtimeout. I check for its existence with command -v gtimeout, and if it's absent I make the wrapper a pass-through. Without gtimeout, if Claude stops responding, launchd keeps piling up the next runs.

TIMEOUT_BIN="$(command -v gtimeout 2>/dev/null || true)"
[ -z "$TIMEOUT_BIN" ] && [ -x /opt/homebrew/bin/gtimeout ] && TIMEOUT_BIN=/opt/homebrew/bin/gtimeout

"Secrets and working dirs" are out of the script's reach

As the comments note, this watchdog can only take care of the ways code breaks in projects managed as git repos.

• A secret in .env expired → Claude can't fix it (out of scope)
• The launchd plist itself broke → out of reach because it's outside the git repo
• The DB schema broke → depends on verify.sh, but migrations are dangerous, so detect mode is recommended

To let Claude correctly judge "out of reach" from the prompt, I explicitly provide the escape hatch: "when you can't fix it, state only UNFIXABLE: and exit."

A mode-detection mistake makes every project full

If you forget the third column in registry.tsv, mode becomes empty and falls back to full via ${mode:-full}. If you forget it on a project you wanted as detect, unintended auto-repair runs, so always state the mode explicitly when writing the tsv.

Summary

• Just register slug, dir, and mode in registry.tsv, and the watchdog monitors everything fully automatically
• Don't trust Claude's self-report; the watchdog independently runs verify.sh
• commit → secret scan → push only when verification passes. On failure, roll back immediately with git reset --hard
• The seven guardrails (independent verification, rollback, secret scanning, attempt cap, cost cap, scope restriction, owner verification) are the real safety of unattended repair
• For cases that are outside git, have no verify in place, or have side effects, stay in detect mode limited to detect + notify
• Putting the escape hatch "when you can't fix it, say UNFIXABLE: and exit" in the prompt prevents unnecessary rewrites

The more automation you have, the more the "cost of detecting and repairing when it breaks" grows in proportion. Once you set up a watchdog, silent failures drop dramatically, and even when things break in the middle of the night, you wake up to a "self-repaired it" message in Discord. The cost of writing health.sh and verify.sh for every project is a one-time thing, and the running cost is nearly zero.

---

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Comments

[Mustafa ERBAY]

I like that the architecture is built around guardrails instead of trust. The strongest design choice, in my opinion, is treating Claude as a repair proposal engine while letting deterministic verification decide whether the change is actually acceptable. One thing I’d be curious about is whether you’ve considered adding a repair planning stage before execution. A machine-readable repair plan (expected files, risk level, intended outcome) could make the process even safer and provide much richer audit trails when something goes wrong.

[FromZeroToShip]

"Don't trust Claude's self-report" is the exact sentence I arrived at this week from the opposite direction, so reading you build a whole loop around it was uncanny. Mine was smaller and dumber: a health-check watcher that had been reading the endpoint's own "database: true" and calling it verified. Someone pointed out that's not an auditor — it's a process grading a report the audited thing wrote about itself — so now the watcher runs its own query instead of reading the claim. Same principle; you just took it all the way to unattended repair, which is the version that actually needs it.

Two of your seven guards are the ones I'd underline for anyone copying this. Independent verification is the obvious one. But "when you can't fix it, say UNFIXABLE and exit" is the quiet hero — the failure mode of a self-healing loop isn't not-fixing, it's confidently fixing the wrong thing forever, and an explicit "I can't" is the only thing that stops the loop from burning budget faking success. Rollback-on-fail is the same instinct: assume the fix is wrong until an artifact you didn't write says otherwise.

The thing I keep coming back to: none of the seven guards is clever. They're all just "don't let the thing vouch for itself." Stack enough of those and unattended becomes safe; skip them and "self-healing" is just a confident story about a broken system. Great, genuinely-run-in-production writeup.

[Mike Czerwinski]

Do not let the thing vouch for itself is the right spine, and you actually built it, the external verify.sh instead of Claude's self-report is the move most of these designs skip. So the push is on the one place the principle can still leak: an independent verifier is only as strong as what it asserts, and an autonomous fix loop optimizes against exactly that.

The loop's objective is make verify.sh green. If verify.sh checks that the automation runs without error, the cheapest green is not always the correct fix, it is the cheapest fix that stops the error, and disabling the failing behavior often is that. A broken scraper repaired into returning empty results passes a liveness check cleanly, and the automation is now silently doing nothing while reporting healthy. You removed self-vouching, which is the big one, but the verifier can still be a shallow oracle, and a repair loop will find the shallowest path to satisfy it, because that is what optimizing against a checker means. So the question the seven guardrails do not yet answer is whether verify.sh asserts the outcome or just the exit code. Liveness is cheap to game, correctness is not, and the more autonomous the loop the more the verifier's shallowness becomes the attack surface, not because anything is adversarial but because the optimizer is doing its job against the weakest true statement you left it.