Is Your 24/7 Automation Actually Alive? Checking It With a Single Command

In my previous autopilot article, I walked through a setup that runs Claude unattended via launchd. This time I want to talk about what happens behind the curtain — the problem where an unattended job quietly dies and nobody ever notices — and how to deal with it.

The truly scary part of unattended automation isn't bugs; it's silent failure. A launchd job can crash with exit 78, a hooks script can lose its execute permission, or agentmemory can run in a "half-alive" state with the port open but no worker behind it — and on the surface, nothing looks wrong. There's a real case documented in the comments of automation-health.sh where exactly this state persisted for six days with nobody noticing.

The problem: dying in silence

Once you build autonomous loops, the number of things to monitor grows. Even in just my own setup:

• launchd jobs (skill-harvest / skill-curate / agentmemory, etc.)
• The actual scripts behind Stop hooks / PreToolUse hooks
• A Stop hook that converts conversation logs into long-term memory
• Automatic updates to my Obsidian Vault
• Weekly and monthly batches (env-audit / plugin-purge / dotfiles-snapshot, etc.)

I have no desire to check by hand whether every one of these is healthy. But hammering launchctl list individually doesn't give me the "big picture" either.

What I need is a single command that inspects all my automation at once and reports back with a three-value verdict: GREEN / WARN / FAIL.

The design: a three-value verdict and OK/WN/NG helpers

The core of ~/.claude/scripts/automation-health.sh is simple.

fail=0; warn=0
ok()  { printf "  ${GRN}✓${RST} %s\n" "$1"; }
wn()  { printf "  ${YEL}⚠${RST} %s\n" "$1"; warn=$((warn+1)); }
ng()  { printf "  ${RED}✗${RST} %s\n" "$1"; fail=$((fail+1)); }

Each check is evaluated with one of ok / wn / ng, and at the very end the verdict is decided like this:

if   [ "$fail" -gt 0 ]; then
  printf "  ${RED}✗ FAIL${RST}  red=%d warn=%d\n\n" "$fail" "$warn"; exit 1
elif [ "$warn" -gt 0 ]; then
  printf "  ${YEL}⚠ WARN${RST}  warn=%d (致命的問題なし)\n\n" "$warn"; exit 0
else
  printf "  ${GRN}✓ ALL GREEN${RST}  全自動化が正常稼働\n\n"; exit 0
fi

• ALL GREEN: every check passed. exit 0
• WARN: not fatal, but worth watching. exit 0
• FAIL: even a single RED means exit 1

By separating the exit codes, you can chain follow-up processing from daily-brief.sh or CI with ||.

The checks: what it's actually looking at

The script is divided into nine sections. Here are the main checks, taken straight from the real code.

1. Are the launchd jobs alive?

for job in com.shun.skill-harvest com.shun.skill-curate; do
  line=$(launchctl list 2>/dev/null | grep -E "\b${job}\b")
  if [ -z "$line" ]; then
    ng "$job: 未ロード (launchctl load し直しが必要)"
  else
    exitc=$(echo "$line" | awk '{print $2}')
    if [ "$exitc" = "0" ] || [ "$exitc" = "-" ]; then
      ok "$job: ロード済 / last exit=$exitc"
    else
      ng "$job: last exit=$exitc (前回失敗)"
    fi
  fi
done

The second column of launchctl list is the last exit code. - means "hasn't run yet (normal)", 0 means success, and anything else means failure.

2. Do the hooks scripts exist, and are they executable?

hooks=(pre_git_guard.sh pre_secrets_check.sh pre_env_guard.sh \
       post_audit_log.sh post_format.sh post_tsc_check.sh \
       stop_notify.sh user_prompt_submit.sh)
for h in "${hooks[@]}"; do
  f="$CLAUDE/hooks/$h"
  if   [ ! -f "$f" ]; then ng "$h: 不在"
  elif [ ! -x "$f" ]; then wn "$h: 実行権限なし (chmod +x 推奨)"
  else ok "$h"
  fi
done

Even if you wire a hook into settings.json, if the underlying script isn't chmod +x'd it gets silently skipped. This detects that case as a WARN.

3. Freshness of the skill-harvest log

hlog="$CLAUDE/skills/auto/.harvest.log"
a=$(age_h "$hlog")   # (now - mtime) / 3600 を返すヘルパ
if [ "$a" -ge 0 ] && [ "$a" -le 48 ]; then
  ok "最終稼働 ${a}h前: ${last##*] }"
else
  wn "最終稼働 ${a}h前 (>48h: cron停止の疑い): ${last##*] }"
fi
nskills=$(find "$CLAUDE/skills/auto" -maxdepth 2 -name SKILL.md | wc -l | tr -d ' ')
ok "生成済 auto-skill: ${nskills} 個"

If the log's mtime is older than 48h, it's a WARN. "A log exists" and "it ran recently" are two different things, so I check the mtime.

4. Survival of conversation logs and freshness of INDEX.md

cnt=$(find "$CONV" -name '*.md' ! -name 'INDEX.md' | wc -l | tr -d ' ')
ok "会話ログ ${cnt} 件 / 最新 ${la}h前"
ia=$(age_h "$idx")
if [ "$ia" -le 24 ]; then ok "INDEX.md 鮮度 ${ia}h前"
else wn "INDEX.md が ${ia}h前 (extract が更新していない可能性)"; fi

5. Auto-update markers in the Obsidian Vault

if [ -f "$hot" ] && grep -q "recent:start auto-updated" "$hot"; then
  ha=$(age_h "$hot"); ok "hot.md 自動更新マーカー有 / ${ha}h前"
else wn "hot.md の自動更新マーカーが見つからない"; fi
# index.md カバレッジ: 実ファイル数 vs 記載ページ数
real=$(find "$VAULT" -name '*.md' -not -path '*/.*' | wc -l | tr -d ' ')
stated=$(grep -oE '総ページ数：[0-9]+' "$vidx" | grep -oE '[0-9]+' | head -1)
if [ "$stated" = "$real" ]; then ok "index.md カバレッジ一致 (${real}p)"
else wn "index.md 記載 ${stated:-?}p ≠ 実 ${real}p"; fi

6. Detecting duplicate bursts in the remember memory layer

for f in now.md recent.md archive.md; do
  [ -f "$REMEMBER/$f" ] && ok "$f 存在" || wn "$f 不在 (必須層)"
done
dup=$(grep -vE '^\s*$|^##|^#' "$nowf" | sort | uniq -c | sort -rn | head -1 | awk '{print $1}')
if [ "${dup:-0}" -ge 3 ]; then
  wn "now.md に同一要約が ${dup} 回 (consolidate 遅延の兆候)"
else
  ok "now.md 重複なし (consolidate 健全)"
fi

When consolidation falls behind, now.md gets the same summary appended over and over. Three or more duplicates trigger an early-warning WARN.

7. Disk usage and leftover files

cdir_mb=$(( ${cdir_total_mb:-0} - ${disabled_mb:-0} ))
if   [ "${cdir_mb:-0}" -lt 2000 ]; then ok "~/.claude 実効 ${cdir_mb}MB"
elif [ "${cdir_mb:-0}" -lt 5000 ]; then wn "~/.claude 実効 ${cdir_mb}MB (>2GB: 大きめ)"
else ng "~/.claude 実効 ${cdir_mb}MB (>5GB: 異常肥大)"; fi
orphan=$(find "$CLAUDE" -maxdepth 1 -name 'security_warnings_state_*.json' -mtime +7 | wc -l | tr -d ' ')
if [ "${orphan:-0}" -ge 5 ]; then
  wn "security_warnings_state_*.json が ${orphan} 個 (>7日前: backups/ へ退避推奨)"
fi

The reversible cache (.disabled-cache) is excluded from the effective size before judging.

8. Log mtimes of the weekly and monthly batches

declare -a CRON_JOBS=(
  "weekly cleanup-misc:$CLAUDE/logs/cleanup-misc.log:192"
  "weekly env-audit:$CLAUDE/logs/env-audit-latest.md:192"
  "monthly plugin-purge:$CLAUDE/logs/plugin-purge.log:744"
  "weekly plugin-auto-disable:$CLAUDE/logs/plugin-auto-disable.log:192"
  "weekly dotfiles-snapshot:$CLAUDE/logs/dotfiles-snapshot.log:192"
  "weekly agents-index:$CLAUDE/logs/agents-index.log:192"
  "daily plugin-usage:$CLAUDE/scripts/plugin-audit-latest.md:48"
)
for entry in "${CRON_JOBS[@]}"; do
  IFS=":" read -r name path budget <<< "$entry"
  a=$(age_h "$path")
  if [ "$a" -le "$budget" ]; then ok "$name: ${a}h前 (budget ${budget}h)"
  else wn "$name: ${a}h前 (budget ${budget}h 超過 — launchd 配送失敗の可能性)"; fi
done

For a weekly job the budget is 192h (8 days), for a monthly one it's 744h (31 days), and anything over budget is a WARN.

The agentmemory "half-alive" problem

Only the agentmemory server check uses a two-stage verdict.

am_line=$(launchctl list 2>/dev/null | awk '$3=="com.shun.agentmemory"{print $1" "$2}')
am_pid=${am_line%% *}; am_exit=${am_line##* }
if [ -z "$am_line" ]; then
  wn "com.shun.agentmemory が launchd に未ロード"
elif [ "$am_pid" = "-" ] && [ "$am_exit" != "0" ]; then
  ng "com.shun.agentmemory 停止中 (last exit=$am_exit) — クラッシュループの可能性"
else
  am_code=$(curl -s -o /dev/null -w '%{http_code}' -m 3 \
            http://localhost:3111/agentmemory/health 2>/dev/null || echo 000)
  if   [ "$am_code" = "200" ]; then ok "稼働中 (pid=$am_pid / health 200)"
  elif [ "$am_code" = "000" ]; then ng "プロセスは居るが port 3111 無応答"
  else ng "port 3111 は開くが /agentmemory/health=$am_code — worker 不在の半生状態"; fi
fi

The reason is written in a comment.

A launchd exit 78 crash loop went unnoticed by anyone for six days, and on top of that the "port is open but there's no worker, so every API returns 404" half-alive state is invisible to liveness monitoring.

A launchd status check alone cannot detect the "the process is there but nothing actually works" case. That's exactly why we confirm HTTP 200 with curl.

Note
Unless you look at both launchd's PID check and the HTTP health endpoint, you'll miss the half-alive state where "it looks like it started but every API returns 404." If you run an MCP server or API server under launchd, always add the extra step of hitting a /health endpoint.

Detecting cron ↔ launchd double execution

An easy-to-miss item is the ninth check: double-execution detection.

cron_sh=$(crontab -l 2>/dev/null | grep -vE '^[[:space:]]*#' | \
          grep -oE '/[^ ]+\.sh' | xargs -n1 basename | sort -u)
launchd_sh=$(grep -hoE '/[^<> ]+\.sh' \
             ~/Library/LaunchAgents/com.shun.*.plist | xargs -n1 basename | sort -u)
dup=$(comm -12 <(printf '%s\n' "$cron_sh") <(printf '%s\n' "$launchd_sh") | \
      grep -vE '^[[:space:]]*$')
if [ -n "$dup" ]; then
  ng "cron と launchd の両方に登録され二重実行されるスクリプト: $(printf '%s' "$dup" | tr '\n' ' ')"
fi

The comment reads: "2026-06-01: while migrating from cron to launchd, I forgot to remove the cron entry, so scripts ended up registered in both = double execution every cycle." It's the classic pattern of forgetting to clear the old crontab after a migration.

Integration with daily-brief

daily-brief.sh is launched by launchd every morning at 8:00, and it embeds a single line about environment health at the top.

echo "## 🏥 環境ヘルス"
run_to 60 ~/.claude/scripts/automation-health.sh 2>&1 | tail -3 | head -1

tail -3 | head -1 slices out just the final summary line (the ✓ ALL GREEN / ⚠ WARN / ✗ FAIL line).

Furthermore, on the daily-brief.sh side, a ## 🚨 Action needed section appears only when a live-connectivity probe detects RED.

RED_FLAGS=""
flag_red() { RED_FLAGS="${RED_FLAGS}\n- ❌ $1"; }

if [ -n "$RED_FLAGS" ]; then
  echo "## 🚨 要対応（ライブ疎通でRED）"
  printf '%b\n' "$RED_FLAGS"
fi

When there's no problem, this section doesn't exist. It's designed as "a section that only appears while something is failing." The brief is written out to ~/Desktop/Daily Brief/today-brief-YYYYMMDD.md, so when you open your desktop, the RED lands right in front of your eyes.

Using automation-health.sh's exit 1, you can add the same pattern to a launchd wrapper.

bash ~/.claude/scripts/automation-health.sh || touch ~/Desktop/AUTOMATION_FAILED.md
bash ~/.claude/scripts/automation-health.sh && rm -f ~/Desktop/AUTOMATION_FAILED.md

The file exists only while something is failing, and disappears once it's fixed — so even without looking at a dashboard, your desktop tells you the state.

Pitfalls I hit

• If you don't treat launchctl list's last exit codes 0 and - the same, you get false WARNs → treat both as normal by rejecting them with an OR condition
• A hooks script that isn't chmod +x'd produces no error from settings.json → I didn't notice until I detected it with a WARN
• agentmemory: once the port is open, launchd considers it "running" → you can't detect the half-alive state until you hit HTTP /health
• When I moved env-audit's output from scripts/ to logs/, the path on the health-check side needed updating too (from the in-script comment "2026-06-11: moved output from scripts/ → logs/")
• In the index.md coverage check, if you don't exclude hidden directories like .backup, you get a permanent WARN → exclude hidden directories with -not -path '*/.*'
• After a cron → launchd migration, forgetting to clear the crontab causes double execution → detect it by comparing the script names of both systems with comm -12

Summary

• Consolidate your liveness check for autonomous systems into a single command with a three-value ALL GREEN / WARN / FAIL verdict
• Use exit 1 as a machine-readable signal, and hook it into a one-line embed in daily-brief or into generating a FAILED file
• By designing "sections and files that only appear while there's a problem," anomalies jump out at you without ever looking at a dashboard
• A launchd PID check alone misses the "half-alive" state. Always hit an HTTP health endpoint alongside it
• Batch liveness can be judged mechanically just by comparing the log's mtime against a budget time

Next time, I plan to write about a weekly-report auto-generation mechanism that integrates this health check with the improvement logs autopilot produces, summarizing what changed over the week.

---

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*