DEV Community

Lily
Lily

Posted on • Originally published at dev.to

It Can Die in Its Sleep — Self-Healing launchd Jobs with Multi-Slot Firing and a Done-Marker

#macos #launchd #automation #claudecode

My previous piece, "Making a launchd Job Unload Itself," built a job that runs exactly once and then unloads itself. This time it's the mirror image: a pattern designed around the assumption that the job will die mid-run — it fires several slots a day and delivers "retry until it succeeds, then quit immediately on success."

Every morning I hand Claude Code the task of updating my Obsidian Vault, and it kept dying partway through — killed by macOS sleep, no network on wake, or a claude timeout. Instead of trying to prevent every failure perfectly, I decided "it can die overnight as long as it's done by the time I wake up" was the more realistic goal, and I redesigned around that.

The problem: "started but didn't finish" piles up silently

There are three ways a launchd job fails to run to completion.

  1. Lid-close sleep (on battery) — caffeinate -s only works on AC power. On battery, the job freezes the instant you close the lid, and gets reaped by timeout after wake.
  2. No network — right after wake, WiFi isn't connected yet. git push and claude's API calls time out.
  3. claude timeout — an ingest that chews through 28 hours of conversation logs doesn't fit in a single slot and times out (this happened three days in a row, 2026-06-11 to 13).

All three can look like "the job started, exit code 0," so you notice late.

Overall design: 4 slots + a done-marker

The fix is simple: stuff multiple StartCalendarInterval entries into the plist, and at the top of the script check "if today's run already succeeded, exit 0 immediately."

<!-- com.shun.vault-auto-ingest.plist (StartCalendarInterval excerpt) -->
<key>StartCalendarInterval</key>
<array>
    <dict>
        <key>Hour</key><integer>4</integer><key>Minute</key><integer>55</integer>
    </dict>
    <dict>
        <key>Hour</key><integer>8</integer><key>Minute</key><integer>20</integer>
    </dict>
    <dict>
        <key>Hour</key><integer>10</integer><key>Minute</key><integer>45</integer>
    </dict>
    <dict>
        <key>Hour</key><integer>12</integer><key>Minute</key><integer>15</integer>
    </dict>
</array>
Enter fullscreen mode Exit fullscreen mode

4:55 is the ideal start time. If that one succeeds, the later three slots (8:20 / 10:45 / 12:15) all see the done-marker and finish in three seconds.

Here's the check at the top of the script.

TODAY=$(date +%Y%m%d)
DONE_MARKER="$HOME/.claude/logs/.vault-ingest-done-${TODAY}"

# 0. If today's run already succeeded, exit immediately (catch-up slot no-op — don't dirty the log either)
[ -f "$DONE_MARKER" ] && exit 0
Enter fullscreen mode Exit fullscreen mode

The done-marker is only touched when the final step succeeds. A run that dies partway never sets the marker, so the next slot redoes the whole thing.

Per-step half-markers: don't do a completed step twice

A single claude ingest can take 25+ minutes. When you retry on the next slot, repeating already-successful steps is wasteful. So I placed a "half-marker" per step.

STEP2A_MARKER="$HOME/.claude/logs/.vault-ingest-step2a-claude-${TODAY}"
STEP2B_MARKER="$HOME/.claude/logs/.vault-ingest-step2b-codex-${TODAY}"
STEP2_MARKER="$HOME/.claude/logs/.vault-ingest-step2-done-${TODAY}"
Enter fullscreen mode Exit fullscreen mode

The function that calls the ingest is written like this.

ingest_src() {
  local marker="$1" src="$2" name="$3" to="$4" extra="$5"
  [ -f "$marker" ] && {
    echo "[$(date '+%F %T')] step2($name) は本日実施済み — skip" >> "$LOG"
    return 0
  }
  cd "$VAULT" && run_to "$to" "$CLAUDE" -p "..." \
    --dangerously-skip-permissions >> "$LOG" 2>&1 \
    && { touch "$marker"; echo "[$(date '+%F %T')] step2($name) 完了" >> "$LOG"; return 0; } \
    || { echo "[$(date '+%F %T')] WARN: step2($name) 失敗/timeout" >> "$LOG"; return 1; }
}

# 2a: Claudeログ（先に処理）。2b: Codexログ（2aの結果も見て統合）
ingest_src "$STEP2A_MARKER" "$KB/raw/conversations/"       "claude" 1500 ""
ingest_src "$STEP2B_MARKER" "$KB/raw/codex-conversations/" "codex"  1500 "..."

# 両サブが済んだ時だけ「本日ingest完了」を立てる
[ -f "$STEP2A_MARKER" ] && [ -f "$STEP2B_MARKER" ] && touch "$STEP2_MARKER"
Enter fullscreen mode Exit fullscreen mode

step2a (Claude logs) succeeds → step2b (Codex logs) times out → next slot fires → step2a is skipped → only step2b is retried. Once both are done, STEP2_MARKER goes up and the pipeline moves to the next stage. The half-markers reset naturally the next day thanks to the ${TODAY} suffix.

Note
The step split came out of a real incident. Trying to process 28h of Claude logs + Codex logs in one shot overran the 40-minute window and timed out three days in a row → hot.md froze (2026-06-11 to 13). Splitting into two per-source runs, each capped at a 25-minute (1500-second) timeout, made it stable.

caffeinate -i exec: block sleep while running

The pattern where the script re-executes itself under caffeinate.

if [ -z "${CAFFEINATED:-}" ]; then
  exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash "$0" "$@"
fi
Enter fullscreen mode Exit fullscreen mode

Because exec replaces the process with itself, caffeinate becomes the process's own parent rather than a child. Putting CAFFEINATED=1 in the environment prevents an infinite re-invocation.

Warning
-s (prevent system sleep on AC power) is inactive on battery. Close the lid on battery and it freezes. That can't be fully prevented, so the design is "a frozen run gets reaped by timeout and the next slot redoes it." -i (prevent idle sleep) works regardless of power source, but doesn't stop lid-close.

90-second network-wait loop

Move right after wake and WiFi isn't up yet. git push and claude's API calls fail immediately.

net_ok=""
for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18; do
  if /usr/bin/nc -z -G 3 1.1.1.1 443 2>/dev/null; then net_ok=1; break; fi
  sleep 5
done
[ -z "$net_ok" ] && echo "[$(date '+%F %T')] WARN: 網未接続のまま続行（失敗時は次スロットが再試行）" >> "$LOG"
Enter fullscreen mode Exit fullscreen mode

18 iterations × 5 seconds = up to 90 seconds of waiting. It breaks as soon as it confirms a connection. If it still isn't connected after 90 seconds, it just emits a warning and continues. It doesn't exit here because in an offline environment step1 (local log conversion) alone can still run. If the later git push or claude fails, the next slot retries.

The -G in nc -z -G 3 specifies macOS's connect timeout (in seconds); note that GNU nc uses different flags.

macOS TCC preflight

Under launchd, ~/Documents/ is protected by TCC (Full Disk Access): unless you grant /bin/bash full disk access from the GUI, you can't read or write it. If it silently ends with exit 0, you won't notice for a week. Detect it explicitly at the top.

if ! ( cd "$VAULT" 2>/dev/null && git rev-parse --git-dir >/dev/null 2>&1 ); then
  echo "[$(date '+%F %T')] ❌ FDA未付与: launchdから '$VAULT' にアクセス不可(TCC保護)。" >> "$LOG"
  echo "    解決: システム設定 > プライバシーとセキュリティ > フルディスクアクセス で /bin/bash を許可。" >> "$LOG"
  notify_fail "FDA未付与: vault にアクセス不可（設定→フルディスクアクセス→/bin/bash）"
  exit 1
fi
Enter fullscreen mode Exit fullscreen mode

Whether git rev-parse --git-dir succeeds tells you whether you have access to the Vault. On failure it exit 1s (without setting the done-marker) → the next slot retries → once FDA is granted it resolves naturally.

Note
iCloud-synced folders (under ~/Documents/) are especially heavily protected. Escaping to ~/ via a symlink is a no-go, because iCloud's conflict handling breaks the symlink (verified 2026-06-01). Keep the Vault directly under ~/Documents/.

Making failure visible: a desktop FAILED marker

A failure you only notice if you go read the log gets ignored. On failure, drop a FAILED-${TODAY}.md on the desktop, and auto-delete it on success.

notify_fail() {
  local reason="$1"
  mkdir -p "$HOME/Desktop/Daily Brief"
  { echo "# Daily Brief 生成失敗 — $(date '+%F %T')"
    echo "- 理由: ${reason}"
    echo "- 詳細ログ: ~/.claude/logs/vault-auto-ingest.log"
    echo "- 自動再試行: 8:20 / 10:45 / 12:15（成功したらこのファイルは自動で消える）"
  } > "$FAILED_FILE"
  /usr/bin/osascript -e "display notification \"${reason}\" with title \"Daily Brief 生成失敗\"" >/dev/null 2>&1
}
Enter fullscreen mode Exit fullscreen mode

Only when step2.6 (archiving the brief) succeeds does it rm -f "$FAILED_FILE". "The file disappears only on a day that succeeded" = "if the file is still there, it's still failing."

Locking: the double-run race

A race where the launchd version and a manual version ran at the same time actually happened (2026-06-10). Exclude it with an mkdir-based lock.

LOCKDIR="$HOME/.claude/locks/vault-auto-ingest.lock"
if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then
  oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true)
  if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then
    echo "[$(date '+%F %T')] 別インスタンス実行中(pid=${oldpid}) — skip" >> "$LOG"
    exit 0
  fi
  rm -rf "$LOCKDIR"
  /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0
fi
echo $$ > "$LOCKDIR/pid"
trap 'rm -rf "$LOCKDIR"' EXIT INT TERM
Enter fullscreen mode Exit fullscreen mode

A stale lock (a lock dir left behind by a dead process) is auto-reclaimed via a pid liveness check (kill -0). Because mkdir is atomic, even if multiple processes hit it simultaneously, only one succeeds.

Pitfalls I hit

  • caffeinate -s inactive on battery → switched to a "don't prevent lid-close freeze; the next slot redoes it" design. -i alone doesn't stop lid-close.
  • claude ingest timed out three days straight → split step2 into two per-source runs (step2a/step2b), each managed with a 1500-second timeout plus a half-marker (fixed 2026-06-13).
  • Silent exit 0 under TCC protection → early detection via a git rev-parse preflight. A silent failure went unnoticed for a week.
  • node/git not found under launchd's minimal PATH → the Vault's commit hook is Node.js-based, so dynamically add the nvm bin to PATH.
  • Log exceeding 5MB → check size with stat -f%z and rotate to .old when it's over (dumping claude's entire output into the log makes it bloat surprisingly fast).
  • timeout doesn't work without GNU timeout → check TIMEOUT_BIN=/opt/homebrew/bin/timeout first, and pass through if absent (brew install coreutils is a prerequisite).

Summary

  • Put multiple StartCalendarInterval entries in the plist, and at the top, check the done-marker and exit 0 immediately if it already succeeded — that alone makes it a self-healing job.
  • Per-step half-markers (step2a/step2b) deliver "no re-running of completed steps — retry only the failed step."
  • caffeinate -i -s isn't a silver bullet. It can't prevent a battery lid-close freeze, so design it so the next slot takes over a frozen run.
  • The network wait is an 18×5-second = up-to-90-second loop. Don't exit immediately even if it can't connect (the design lets the next slot retry).
  • macOS TCC protects ~/Documents/. Confirm access from launchd with a git rev-parse preflight to prevent silent failures.

Combined with the previous piece, "Making a launchd Job Unload Itself," you now have two patterns: "a job you want to run to completion exactly once" and "a job you want to reliably complete once a day." Next time I plan to write about how to efficiently inject the Vault this job keeps building up into Claude Code's context.

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Top comments (0)