DEV Community: Lily

The Morning My cron Jobs Went Silent: A 97-Line Script That Migrated Everything to launchd

Lily — Tue, 01 Sep 2026 00:00:04 +0000

Six months after being laid off, I'd rebuilt my income from zero to ¥1.2M/month on an autonomous setup. Then one morning at 8:00, it just wasn't there — no error, no alert, nothing. The cause: a macOS update had quietly disabled the cron daemon. My fix was a 97-line shell script that parses crontab line by line and auto-generates launchd plists.

Why This Approach Works

"It Should Be Running" Is the Most Dangerous Kind of Confidence

Back when my side business was earning ¥600K/month, nearly every yen of that automation benefit rode on cron jobs. Timing note publications, scheduling social posts, daily data aggregation — all of it lined up in crontab -l. When I was laid off and dropped to zero, rebuilding the environment with Claude Code, I decided carrying the crontab over as-is was the fastest path.

Right after upgrading to macOS Sequoia (15.x), nothing appeared to have changed. Run crontab -l and every entry is still there. But the daemon isn't running. Since macOS Ventura, Apple has been progressively decoupling the cron daemon from the user session, and on Sequoia/Tahoe it's perfectly normal to have /usr/sbin/cron present while launchctl list | grep cron returns nothing at all.

The reason I was slow to notice is that when automation stops, no error appears. My assumption was that if cron isn't running, an error mail lands in /var/mail/<username> — and that assumption had collapsed. On Sequoia it doesn't reach the post office by default. The 8:00 daily brief doesn't arrive, the 11:00 social post doesn't go out, and only then do you notice. That "silent death" is what scares me.

Why launchd Is the Right Answer

On macOS, process launching and management belongs to launchd (PID 1). cron survives only as historical compatibility; what Apple actually recommends is job management via launchd. launchd handles automatic restarts when a daemon crashes, automatic execution after wake for jobs scheduled while the machine was asleep, direct redirection of stdout/stderr to files, and explicit injection of environment variables — all declaratively, in a single plist file.

cron lets you write */5 * * * * cmd on one line; a launchd plist becomes 20–30 lines of XML. That verbosity is the biggest psychological barrier to migrating to launchd. Rewriting ten of them by hand isn't realistic. So you generate them with a script.

Looking at one plist that's actually in production makes the structure click. Here's how ~/Library/LaunchAgents/com.shun.daily-brief.plist is composed (excerpted from the real file, paths converted to ~ notation):

<key>Label</key>
<string>com.shun.daily-brief</string>

<key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>~/.nvm/versions/node/v24.13.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:
          /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.local/bin</string>
</dict>

<key>StartCalendarInterval</key>
<array>
  <dict>
    <key>Hour</key><integer>8</integer>
    <key>Minute</key><integer>0</integer>
  </dict>
  <dict>
    <key>Hour</key><integer>10</integer>
    <key>Minute</key><integer>30</integer>
  </dict>
</array>

<key>ProgramArguments</key>
<array>
  <string>~/.claude/scripts/claude-quota-guard.py</string>
  <string>--job</string>
  <string>com.shun.daily-brief</string>
  <string>--</string>
  <string>/bin/bash</string>
  <string>~/.claude/scripts/daily-brief.sh</string>
</array>

<key>LowPriorityIO</key><true/>
<key>Nice</key><integer>10</integer>
<key>RunAtLoad</key><true/>
<key>StandardOutPath</key>
<string>~/.claude/logs/com.shun.daily-brief.log</string>
<key>StandardErrorPath</key>
<string>~/.claude/logs/com.shun.daily-brief.log</string>

Three things stand out.

Explicit EnvironmentVariables. launchd does not read your shell configuration (.zshrc, .bashrc). A script that uses node installed via nvm has no PATH to it under launchd management and dies with node: command not found. This accounts for 90% of the cases where a job migrated from cron suddenly stops working. Writing PATH explicitly into the plist guarantees the same binary gets called no matter what the shell is.

The array form of StartCalendarInterval. When you want to run multiple times per day, you line up <dict> entries inside an <array>. daily-brief runs twice, at 8:00 and 10:30. In cron you'd write 0 8,10 * * *, but launchd requires a dictionary per time. How far the auto-generation script covers this notational gap ties into the pitfalls described later.

LowPriorityIO and Nice. Background jobs get lowered I/O priority and a CPU scheduler nice value of 10. It's a setting to minimize impact on foreground work (editor, browser), consistent with the "erase your presence" philosophy of an autonomous environment.

What It Means to Invest in the Environment, Not the Work

Of that ¥1.2M/month breakdown, almost none of it is me moving my hands. Most of the note series, social updates, and data aggregation are automated. The maintenance cost of this environment comes down to moving cron onto a foundation that actually runs. The goal of being under launchd management is that a scheduled task you wrote once is still running three years later. Apple's launchd is a stable API unchanged since macOS 10.4 (2005), and it doesn't "die unnoticed" the way cron does. launchctl list com.shun.daily-brief shows you LastExitStatus and the next scheduled run instantly.

The 90 minutes spent setting up the environment is an investment that buys back 5 minutes × 365 days (= 30 hours) of "let me check whether it's actually running" every morning.

The Overall Flow

Map of the Processing

crontab -l
  │  grep -vE '^\s*#' | grep -v '^$'  ← コメント行・空行を除外
  ↓
[1行ごとにループ]
  │  awk '{print $1...$5}' で schedule フィールド抽出
  │  cut -d' ' -f6-           で cmd 部分を切り出し
  │  basename からラベル生成  → com.shun.<script-name>
  ↓
StartCalendarInterval XML 組み立て
  │  ※ */N 形式は非対応（固定値のみ）← ここが落とし穴
  ↓
plist ファイル書き出し
  → [dry]   ~/.claude/scripts/launchd-proposed/*.plist
  → [apply] ~/Library/LaunchAgents/*.plist
               + launchctl unload → launchctl load
  ↓
⚠️  警告: crontab から手動削除しないと二重起動

Dissecting the Script (All 97 Lines)

The script lives at ~/.claude/scripts/cron-to-launchd.sh, and there are two ways to use it.

# 差分確認（ファイルを書くだけ、loadしない）
~/.claude/scripts/cron-to-launchd.sh dry

# 本番反映（LaunchAgentsにコピーしてlaunchctl load）
~/.claude/scripts/cron-to-launchd.sh apply

Call it with no arguments and dry is the default (MODE="${1:-dry}"). The iron rule is to not jump straight to apply — run dry first and eyeball the generated output.

Phase 1: Reading and parsing the crontab (lines 20–28)

CRON_LINES=()
while IFS= read -r line; do
  [ -n "$line" ] && CRON_LINES+=("$line")
done < <(crontab -l 2>/dev/null | grep -vE '^\s*#' | grep -v '^$')

As the comment bash 3.2 互換 indicates, the bash that ships with macOS is version 3.2 (Apple hasn't updated it for GPLv2 reasons). mapfile and readarray aren't available in 3.2, so the array is built with a while IFS= read -r loop. crontab -l 2>/dev/null swallows the error when the crontab is empty, grep -vE '^\s*#' strips comment lines, and grep -v '^$' strips blank lines.

Phase 2: Splitting each line into schedule and cmd (lines 28–38)

minute=$(echo "$line" | awk '{print $1}')
hour=$(echo "$line" | awk '{print $2}')
dom=$(echo "$line"   | awk '{print $3}')
mon=$(echo "$line"   | awk '{print $4}')
dow=$(echo "$line"   | awk '{print $5}')
cmd=$(echo "$line"   | cut -d' ' -f6-)

The cron format min hour dom mon dow cmd... is pulled apart field by field with awk. Since cmd takes everything from the sixth field onward via cut -d' ' -f6-, it picks up the command correctly no matter how many arguments it has.

The label generation logic (lines 38–40):

script=$(echo "$cmd" | grep -oE '~/.claude/scripts/[^ ]+' | head -1 | xargs basename 2>/dev/null)
if [ -z "$script" ]; then
  script="$(echo "$cmd" | awk '{print $1}' | xargs basename 2>/dev/null)-${minute}${hour}"
fi
label="com.shun.$(echo "$script" | sed -E 's/\.[a-z]+$//' | tr '_' '-')"

Scripts under ~/.claude/scripts/ get labeled from the basename with the extension stripped. For example, daily-brief.sh becomes com.shun.daily-brief. Other, general-purpose commands (find, backup-rotate, and so on) secure uniqueness with command name + minute + hour. Underscores are converted to hyphens (launchd Label convention).

Phase 3: Assembling the StartCalendarInterval XML (lines 44–52)

cal_xml="  <key>StartCalendarInterval</key>\n  <dict>\n"
# */N 周期は launchd では複数エントリで再現する必要 — ここでは固定値だけ対応
if [ "$minute" != "*" ]; then cal_xml+="    <key>Minute</key><integer>${minute}</integer>\n"; fi
if [ "$hour"   != "*" ]; then cal_xml+="    <key>Hour</key><integer>${hour}</integer>\n";   fi
if [ "$dom"    != "*" ]; then cal_xml+="    <key>Day</key><integer>${dom}</integer>\n";      fi
if [ "$mon"    != "*" ]; then cal_xml+="    <key>Month</key><integer>${mon}</integer>\n";    fi
if [ "$dow"    != "*" ]; then cal_xml+="    <key>Weekday</key><integer>${dow}</integer>\n";  fi
cal_xml+="  </dict>"

If a field is * (wildcard), the corresponding key is omitted from the XML — that's the semantics of launchd's StartCalendarInterval. For instance, 0 8 * * * (8:00 every day) only needs Hour=8, Minute=0; omitting Day/Month/Weekday is what makes it "every day."

Phase 4: Writing out the plist body (lines 54–76)

cat > "$plist" <<XMLEOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>${label}</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/zsh</string>
    <string>-c</string>
    <string>${cmd//&/&amp;}</string>
  </array>
$(echo -e "${cal_xml}")
  <key>StandardOutPath</key>
  <string>${log}</string>
  <key>StandardErrorPath</key>
  <string>${log}</string>
  <key>ProcessType</key>
  <string>Background</string>
</dict>
</plist>
XMLEOF

The command is wrapped as /bin/zsh -c "cmd". Commands that were running under cron often depend on shell expansion (~ expansion, globbing), and there are cases where passing them directly to ProgramArguments doesn't work. Going through zsh absorbs that difference. ${cmd//&/&} is XML escaping — a command containing & would produce invalid XML, so it's substituted here. Logs send both stdout and stderr together to ~/.claude/logs/${label}.log.

Phase 5: Deployment in apply mode (lines 84–96)

if [ "$MODE" = "apply" ]; then
  for f in "$PROPOSED"/*.plist; do
    cp "$f" "$TARGET_DIR/"
    launchctl unload "$TARGET_DIR/$(basename $f)" 2>/dev/null
    launchctl load   "$TARGET_DIR/$(basename $f)"
    echo "  loaded: $(basename $f)"
  done
  echo ""
  echo "🚨 cron 行は **手動で削除してください**:  crontab -e"
  echo "(誤って cron+launchd 両方走るのを避けるため)"
fi

launchctl unload is called first for idempotency. Trying to load a plist that's already loaded results in an error. Unloading beforehand means running apply any number of times produces the same result. But there's one caveat — the script only prints a warning after apply saying "please delete from crontab manually"; it doesn't automate the deletion. Leave the cron lines in place and, whenever macOS eventually revives the cron daemon, you get double execution from cron + launchd.

Implementation Details

`set -uo pipefail` — Why `-e` Was Left Out

The declaration at the top of the script is set -uo pipefail (line 9 of the real file). Some of you may have noticed -e (exit immediately on error) isn't there. That's an intentional design decision.

Look at the loop in apply mode (lines 84–96).

launchctl unload "$TARGET_DIR/$(basename $f)" 2>/dev/null
launchctl load   "$TARGET_DIR/$(basename $f)"

launchctl unload returns a non-zero exit code if the target plist isn't loaded yet. With -e enabled, the script dies on the very first unload of the first plist. 2>/dev/null suppresses the error output, but the exit code remains. Omitting -e is what delivers the idempotent behavior of "keep the loop going even if unload fails."

For the same reason, crontab -l 2>/dev/null (line 22) is safe. In a user environment with an empty crontab, crontab -l exits non-zero with crontab: no crontab for <username>, but 2>/dev/null swallows it and the loop proceeds. With -e, it would have died right there.

-u (error on undefined variables) and -o pipefail (propagating pipe failures) stay. Those are guards you need — for catching variable name typos and failures partway through a pipe. Only -e gets in the way — and that judgment call is a recurring pattern in shell script error handling.

The Label-Generation Regex and Its Absolute-Path Dependency

Read the label generation logic on line 35 precisely and one important specification becomes visible.

script=$(echo "$cmd" | grep -oE '~/.claude/scripts/[^ ]+' | head -1 | xargs basename 2>/dev/null)

Note that the regex matches on the absolute path, not ~/.claude/scripts/. If the crontab entry was written as ~/.claude/scripts/daily-brief.sh, this regex won't match, because ~ is recorded as a literal string before the shell expands it. If it doesn't match, the script variable ends up empty and falls through to the fallback.

if [ -z "$script" ]; then
  script="$(echo "$cmd" | awk '{print $1}' | xargs basename 2>/dev/null)-${minute}${hour}"
fi

The fallback is "basename of the command + minute + hour." For example, if you registered ~/.claude/scripts/daily-brief.sh with 0 8 * * *, the label becomes com.shun.daily-brief-08. Not daily-brief but daily-brief-08. That discrepancy breeds confusion later when you're chasing logs.

Always write absolute paths when registering in the crontab — that's the only correct way to coexist with this script.

How the `*/N` Format Breaks — Traced Through the Code

Let's confirm the "*/N unsupported" point raised earlier through the actual code flow. Say the crontab has the line */15 * * * * ~/.claude/scripts/health-check.sh. What happens?

minute=$(echo "*/15 * * * * ~/.claude/scripts/health-check.sh" | awk '{print $1}')
# → "*/15"

Then the conditional:

if [ "$minute" != "*" ]; then
  cal_xml+="    <key>Minute</key><integer>${minute}</integer>\n"
fi

"*/15" != "*" is true, so it passes the condition, and the generated XML is:

<key>Minute</key><integer>*/15</integer>

The string */15 ends up inside an <integer> tag. It parses as XML, more or less, but when launchd loads the plist it gets rejected by the validation that "Minute must be an integer from 0 to 59." launchctl load returns a non-zero exit code, loaded: still gets printed, but scheduling was never actually enabled.

This "looks like the load went through but it isn't actually running" state is nasty, and it shows up again in the next section.

The Intent and Limits of the `/bin/zsh -c` Wrapper

The ProgramArguments in the generated plist (lines 54–66):

<key>ProgramArguments</key>
<array>
  <string>/bin/zsh</string>
  <string>-c</string>
  <string>${cmd}</string>
</array>

Wrapping the command in zsh is there to get the ~ expansion, environment variable references, and glob patterns that tend to appear in cron entries interpreted. Pass a command directly to ProgramArguments and execvp is called without shell expansion, so ~ gets passed through as a literal string and you get a file-not-found error.

That said, even with /bin/zsh -c, launchd does not read your .zshrc. That's launchd's design. It starts zsh in non-login script mode rather than interactive mode, so even if you've written source ~/.zshrc, it isn't loaded. As a result, processes start in a state where node managed by nvm, python from pyenv, and the various Homebrew commands have no PATH to them.

Look at the generated plist template and there's no EnvironmentVariables key (not anywhere across lines 54–76). That's exactly why daily-brief.plist has EnvironmentVariables appended by hand. The plists the script auto-generates do not include this PATH injection.

Where I Got Stuck

Stuck 1: `/15 * * *` Failed Silently

Symptom. Running apply printed loaded: com.shun.health-check.plist. But 15 minutes later, and 30 minutes later, nothing was written to ~/.claude/logs/com.shun.health-check.log.

launchctl list com.shun.health-check
# → Could not find service "com.shun.health-check" in domain for port

A service that should be loaded doesn't exist in launchctl's list.

Cause. */15 was written straight into <integer>*/15</integer>, and launchd internally rejected the plist during validation. Because the launchctl load command itself returned exit code 0 (behavior on macOS Sequoia), the script's echo "loaded:" ran anyway. With no error shown, the service simply didn't exist.

Fix. Validating the plist with plutil -lint ~/.claude/scripts/launchd-proposed/com.shun.health-check.plist rejects it immediately. Lines containing */15 need to be manually rewritten in the crontab before migrating. For every 15 minutes, either switch to launchd's StartInterval (interval specified in seconds), or write out the fixed values 00,15,30,45 as an array of four entries.

<key>StartCalendarInterval</key>
<array>
  <dict><key>Minute</key><integer>0</integer></dict>
  <dict><key>Minute</key><integer>15</integer></dict>
  <dict><key>Minute</key><integer>30</integer></dict>
  <dict><key>Minute</key><integer>45</integer></dict>
</array>

Or specifying seconds with StartInterval is simpler:

<key>StartInterval</key>
<integer>900</integer>

900 seconds = 15 minutes. This form is outside the script's auto-generation scope, but it's a single hand-written spot.

Stuck 2: `~` Paths in the crontab Caused Label Collisions and Overwrote Old plists

Symptom. Inside ~/.claude/scripts/launchd-proposed/, which I was checking in dry mode, plists with unfamiliar label names had appeared. Names like com.shun.daily-brief-08.plist and com.shun.note-publish-308.plist — with a time appended to the end.

Cause. Because the crontab was written with ~ as ~/.claude/scripts/daily-brief.sh, it didn't hit the absolute-path match ~/.claude/scripts/[^ ]+ on line 35 and fell into the fallback command-name-minutehour form. On top of that, the com.shun.daily-brief.plist generated by a previous apply was still sitting in ~/Library/LaunchAgents/, so the old plist and the new plist existed in duplicate under different labels.

Running launchctl list | grep com.shun showed two entries calling the same script.

Fix. Open the crontab with crontab -e and rewrite ~ as an absolute path. Then manually unload and delete the old-label plist in ~/Library/LaunchAgents/.

launchctl unload ~/Library/LaunchAgents/com.shun.daily-brief-08.plist
rm ~/Library/LaunchAgents/com.shun.daily-brief-08.plist

You need the habit of always running dry before apply to visually confirm the generated labels and check they're in the expected com.shun.<script-name> form. If fallback-form names (trailing digits) are mixed in, suspect how the crontab is written.

Stuck 3: node and python Were `command not found`

Symptom. After apply, the same error kept appearing every time in ~/.claude/logs/com.shun.note-autolike.log.

/bin/zsh: node: command not found

Running the same command manually from the terminal works fine.

Cause. The generated plist doesn't include EnvironmentVariables. Even started via /bin/zsh -c, .zshrc isn't read, and the ~/.nvm/versions/node/v24.13.0/bin that nvm adds isn't in PATH. Your terminal's shell session and processes under launchd management run in completely different PATH environments.

Fix. Manually edit the generated plist and add EnvironmentVariables before <key>ProgramArguments</key>. daily-brief.plist (quoted from the real file) is the correct model:

<key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>~/.nvm/versions/node/v24.13.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.local/bin</string>
</dict>

Properly, this block should be built into the script's generation template. But "which node version to use" varies by environment, and hardcoding it into the template means rewriting every plist when the environment changes. Perhaps the current script omits it deliberately to avoid that "danger of pinning a version" — at least, that's how I've interpreted it to make peace with it.

In actual operation, I always hand-add EnvironmentVariables to the plists of jobs that use node. The division of labor is: script generation "builds 90% of the skeleton," and the remaining 10% — PATH injection — is manual.

Stuck 4: cron + launchd Were Double-Running the Same Script

Symptom. note auto-posting was supposed to run twice a day, but the logs showed the posting API being called four times a day. It came to light when I hit the rate limit and error responses started appearing.

Cause. I'd forgotten to delete the cron lines with crontab -e after apply. I'd overlooked the warning at the end of the script (lines 94–95).

🚨 cron 行は **手動で削除してください**:  crontab -e
(誤って cron+launchd 両方走るのを避けるため)

I'd convinced myself that "cron lines are safe to leave" because the cron daemon doesn't start in a macOS Sequoia environment. In reality, even on Sequoia there are moments when the cron daemon restarts (mainly after OS updates), and at that point both start running. This time, a macOS minor update was that moment.

Fix. Check which lines have been migrated to launchd with crontab -l and either delete them all or comment out the migrated ones. The safest is crontab -r (delete everything), but if anything hasn't been migrated there's no way back, so I handled it with crontab -e, checking line by line.

Since that failure, I run these two commands as a set to confirm apply is complete.

# launchd側の稼働確認
launchctl list | grep com.shun

# cron側の残骸確認（0行ならOK）
crontab -l 2>/dev/null | grep -vE '^\s*#' | grep -v '^$' | wc -l

If the second command returns 0, no active cron lines exist. That's my criterion for judging the migration complete.

Stuck 5: The plist I Checked in `dry` Was a Different File From the One `apply` Deployed

Symptom. Eyeball the output in dry → no problems → run apply → and somehow the schedule has changed.

Cause. Old plists from a previous dry were still sitting in the PROPOSED directory (~/.claude/scripts/launchd-proposed/). This time's dry generated from a different set of cron lines, so updated plists and old plists were mixed together. Since apply deploys all of PROPOSED/*.plist, unintended older-generation plists also got copied over into ~/Library/LaunchAgents/.

for f in "$PROPOSED"/*.plist; do
  cp "$f" "$TARGET_DIR/"

This copy-everything is the origin of the problem.

Fix. Make it a habit to clear the PROPOSED directory before dry.

rm -f ~/.claude/scripts/launchd-proposed/*.plist
~/.claude/scripts/cron-to-launchd.sh dry

Or check the diff between PROPOSED and LaunchAgents with diff right before apply. Both are chores, but since there's no cleanup handling on the script side, for now manual discipline is the only way to cover it.

To sum up the sticking points so far: the only lines the script automates are the ones that are "fixed schedule, absolute path, no PATH needed." The rest — */N format, ~ paths, nvm/pyenv dependencies — need manual pre- or post-processing. Had I understood that boundary up front, I could have prevented three of the four failures. It's more accurate to read the 97-line script not as something that "fully automates cron migration," but as a tool that "skips 80% of the manual work and throws the remaining 20% into relief."

Pitfalls

The "where I got stuck" section above covered five episodes. Here I organize the pitfalls systematically so the same failures don't repeat. First let's confirm "the scope the script can automate," then line up the easily-missed traps all at once.

What Can and Can't Be Automated

cron-to-launchd.sh (97 lines) only works correctly for cron lines that satisfy all of the following conditions.

The cron expression uses only fixed values — no */N format
The command is written with an absolute path — not a ~ expansion
The command string contains no &, <, or >
The same script is registered at only one time

Lines that fall outside these four conditions either break auto-generation or require mandatory manual fixes after generation. It's accurate to use it not as something that "fully automates migrating every crontab line," but as "a tool that builds 80% of the skeleton for lines meeting the four conditions and throws the remaining 20% of manual work into relief."

Pitfall List (With Real Code)

XML escaping only covers & — plists break on lines containing < and >

Look at line 65 of the script.

<string>${cmd//&/&amp;}</string>

It converts & to &, but there's no conversion for < → < or > → >. If your crontab has a line with a redirect like cmd > /dev/null 2>&1, a > gets mixed into the <string> tag of the generated plist and the XML parser can't read the plist. launchctl load returns an error, but since the apply loop moves on to the next plist, it's a structure where a single broken file is easy to miss. For lines containing > or <, either move the redirect inside the script before migrating, or hand-write the plist.

Generated plists have no RunAtLoad — you can't verify behavior right after apply

The auto-generation template (all of lines 54–76) has no RunAtLoad key. Meanwhile, lines 28–29 of the hand-finished com.shun.daily-brief.plist real file contain <key>RunAtLoad</key><true/>.

A plist without RunAtLoad doesn't execute until the next scheduled time. Checking the log right after apply and finding nothing written isn't a malfunction — it's by design. The problem, though, is that you can't test "does this actually work" on the spot. When you want to check, use launchctl kickstart:

launchctl kickstart -k gui/$(id -u)/com.shun.xxx
tail -f ~/.claude/logs/com.shun.xxx.log

StartCalendarInterval is a bare <dict> — multiple times require manual conversion to <array>

The cal_xml on lines 46–52 of the generation script is complete with a single <dict>. Expressing "twice, at 8:00 and 10:30" like com.shun.daily-brief.plist (lines 33–47 of the real file) requires an array, but the script doesn't generate arrays.

<!-- 自動生成物（単一時刻しか表現できない） -->
<key>StartCalendarInterval</key>
<dict>
  <key>Hour</key><integer>8</integer>
  <key>Minute</key><integer>0</integer>
</dict>

If you want to assign multiple times to the same script, manually rewrite the plist into array form after generation.

Registering the same script at multiple times in the crontab makes the later plist overwrite the earlier one

Suppose you want daily-brief.sh to run at 8:00 and 10:30, so you write two lines in the crontab.

0  8  * * * /path/to/.claude/scripts/daily-brief.sh
30 10 * * * /path/to/.claude/scripts/daily-brief.sh

Because label generation (line 40) strips the extension from the script name, both lines become com.shun.daily-brief. The plist filename is identically com.shun.daily-brief.plist. The line processed later (the 10:30 one) overwrites the earlier one (8:00), and the 8:00 setting disappears. There's no collision detection on the script side. Eyeballing the generated output in dry is the only recourse.

The */N format fails silently — launchctl load looks successful

The crux of the episode detailed on p2, in one line. */15 gets written out as <integer>*/15</integer> and launchd rejects the plist during internal validation. The launchctl load command returns exit code 0 so it looks successful, but if launchctl list com.shun.xxx can't find the service, it was rejected. Manually converting lines containing */N before migration is the only solution.

~ paths fall into the label-generation fallback

The regex on line 35 only matches absolute paths. If you've written ~/.claude/scripts/note-autolike.sh, the fallback (lines 36–39) kicks in and the label gets trailing digits, like com.shun.note-autolike-308. If a com.shun.note-autolike.plist generated earlier from an absolute path is still in ~/Library/LaunchAgents/, you've created a double-execution state where two different labels call the same script. Always write absolute paths in the crontab.

Running apply without clearing PROPOSED mixes in old plists

for f in "$PROPOSED"/*.plist on line 87 copies every file in PROPOSED indiscriminately. If a plist generated by a previous dry for a cron line you've since deleted is still there, the job you thought you deleted comes back to life on apply. Make clearing with rm -f ~/.claude/scripts/launchd-proposed/*.plist before running dry a habit.

Generated plists have no EnvironmentVariables — nvm, pyenv, and Homebrew commands die

The generation template (lines 54–76) doesn't include the EnvironmentVariables key. Since launchd doesn't read your .zshrc, a script calling nvm-managed node falls over immediately at startup with node: command not found. It works fine when run manually from the terminal but dies via launchd — that asymmetry makes diagnosis hard. Using the PATH string on lines 6–9 of com.shun.daily-brief.plist as your model, add it to every plist that uses node or python.

Generated plists have no LowPriorityIO or Nice — automation interferes with the foreground

Lines 12–15 of com.shun.daily-brief.plist have LowPriorityIO and Nice 10, but the generation template doesn't. Without the setting, background jobs run at normal I/O priority. If you've ever had a job doing heavy file reads and writes slow down your editor or browser's responsiveness, check whether these keys are present.

Forgetting to delete cron lines is a time bomb — the next OS update double-runs everything

After the script's apply (lines 94–95) it only warns "please delete the cron lines manually"; the deletion isn't automated. Since the cron daemon doesn't start on Sequoia, it's easy to think "leaving them is safe," but there are real cases where a macOS minor update revives the cron daemon. My note auto-posting running four times a day and hitting the API rate limit came out of this failure. I prevent recurrence by including "zero cron leftovers" in the criteria for migration completion.

Best Practices

A rule set distilled from a 97-line script and six months of operation, usable for both migration work and day-to-day operation.

1. Write crontab entries with absolute paths

Write /home/.../.claude/scripts/xxx.sh instead of ~/.claude/scripts/xxx.sh. It matches the regex on line 35 and the label becomes the intended com.shun.xxx. Rewriting past cron lines takes effort, but it prevents three things at once: label collisions, double execution, and confusion from fallback naming after migration.

2. Manually convert the */N format before migrating

*/15 * * * * (every 15 minutes) converts to one of two things. If the interval is fixed, StartInterval (in seconds) is simplest.

<key>StartInterval</key>
<integer>900</integer>  <!-- 900秒 = 15分 -->

If you need execution at specific minutes, enumerate fixed values in an array (minutes 0, 15, 30, 45). Missed conversions can be caught with plutil -lint.

3. Clear the PROPOSED directory before dry

rm -f ~/.claude/scripts/launchd-proposed/*.plist
~/.claude/scripts/cron-to-launchd.sh dry

Running these two lines as a set prevents the problem of older-generation plists getting mixed into apply.

4. Validate every plist with plutil -lint after dry, before apply

for f in ~/.claude/scripts/launchd-proposed/*.plist; do
  echo "--- $(basename $f)"
  plutil -lint "$f"
done

Catch */N contamination, missed XML escaping, and syntax errors up front with Apple's official tool. Don't apply any plist for which plutil -lint doesn't return OK.

5. Confirm completion with two commands right after apply

# launchd側の稼働確認
launchctl list | grep com.shun

# cron残骸確認（0ならOK）
crontab -l 2>/dev/null | grep -vE '^\s*#' | grep -v '^$' | wc -l

If the second line returns 0 and the launchd entry count matches the number of lines targeted for migration, you can judge the migration complete.

6. Manually add EnvironmentVariables to the plists of jobs that use node

Insert it immediately before <key>ProgramArguments</key> right after generation:

<key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>~/.nvm/versions/node/v24.13.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.local/bin</string>
</dict>

Match the nvm version number to your actual environment. Lines 6–9 of com.shun.daily-brief.plist are the model.

7. Rewrite StartCalendarInterval as an array for multi-time plists

If you want to run the same script at two times, use an array in a single plist (don't write two crontab lines and cause a label collision).

<key>StartCalendarInterval</key>
<array>
  <dict><key>Hour</key><integer>8</integer><key>Minute</key><integer>0</integer></dict>
  <dict><key>Hour</key><integer>10</integer><key>Minute</key><integer>30</integer></dict>
</array>

The description on lines 33–47 of com.shun.daily-brief.plist is a live example.

8. Set LowPriorityIO and Nice 10 on background jobs generally

Add it to every generated plist so it doesn't get in the way of your work:

<key>LowPriorityIO</key><true/>
<key>Nice</key><integer>10</integer>

By having the automation environment "erase its presence," you can design it so it doesn't encroach on human working territory.

9. Hand-write plists for lines whose commands contain &, <, or >

Don't rely on generation; do the XML escaping accurately:

& → &
< → <
> → >

Moving redirects inside the shell script being called is cleanest. Try to handle redirects within the plist's XML and you'll almost always hit this escaping problem.

10. Periodically check LastExitStatus with launchctl list com.shun.xxx

launchctl list com.shun.daily-brief

"LastExitStatus" = 0 is healthy. Anything other than 0, check the log. Weekly bulk check:

launchctl list | grep com.shun | awk '{print $3}' | \
  xargs -I{} sh -c 'launchctl list "{}" 2>/dev/null' | \
  grep -E '"Label"|"LastExitStatus"'

11. Debug with on-demand execution via launchctl kickstart

When you want immediate execution without waiting for the scheduled time:

launchctl kickstart -k gui/$(id -u)/com.shun.xxx

-k is an idempotent option that kills the running instance and restarts it. If nothing appears in the log, it's a PATH problem or a script path problem.

12. Verify every service is alive after a macOS update

Minor updates can change launchd's behavior. If the daily brief doesn't arrive the morning after an update, hit launchctl list | grep com.shun first. If a service is gone, re-apply brings it back.

13. Define three "completion conditions" for the migration

When the "end" of migration work is vague, you tend to skip verification. I set the following as completion conditions:

The entry count of launchctl list | grep com.shun matches the number of cron lines targeted for migration
crontab -l 2>/dev/null | grep -vE '^\s*#' | grep -v '^$' | wc -l returns 0
Every service's LastExitStatus is 0, at least after its first run

Only when all three are satisfied can you say "migration complete."

14. Estimate the total time for the migration work up front

Count the number of cron lines, how many contain the */N format, and how many jobs depend on nvm before you start. With ten or fewer, the whole sequence of dry → plutil validation → manual fixes → apply → completion check finishes within 90 minutes. With 30 or more, a split strategy is realistic: auto-migrate the lines meeting the four conditions first, then hand-migrate the rest on later days.

Summary

The problem of macOS's cron daemon quietly stopping is slow to discover precisely because no error appears. Entries are lined up in crontab -l, yet the 8:00 daily brief doesn't arrive and the 11:00 social post doesn't go out — and it takes hours before that odd feeling registers. It's more accurate to frame migrating to launchd not as "dealing with it after cron breaks," but as "an up-front investment in getting back onto macOS's native mechanism."

What the 97-line cron-to-launchd.sh does is simple. Read the crontab line by line, convert five fields into XML, write it out as a plist. In three steps — dry → plutil validation → apply — you can mass-produce skeletons for lines that are fixed-schedule, absolute-path, and PATH-free. But it's not "fully automatic magic." The */N format, ~ paths, nvm/pyenv dependencies, multiple times, characters requiring XML escaping — these need manual pre- or post-processing. By having the script "build 90% of the skeleton," the target of the manual work becomes clear. Understanding that structure and using it accordingly is the shortest path to not getting stuck after migration.

For jobs you've finished moving to launchd, you can check state instantly with launchctl list com.shun.xxx. LastExitStatus being 0 proves "it is running," not "it should be running." The reliability of an autonomous environment accumulates by eliminating the discovery that "I thought it was running, but it had stopped."

I've written up the full picture of the setup, the ¥1.2M/month breakdown, and the 30-day procedure in a paid note.

📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

28 Hours of Green Logs, Zero Replies: How a Single `echo` Line Swallowed Every Exit Code

Lily — Mon, 31 Aug 2026 05:00:06 +0000

For 24 hours, every dashboard was green. Every launchd job reported exit 0, and the logs lined up neatly with (exit 0) on every row. Meanwhile, DM replies on X (formerly Twitter) were at zero for the entire day, and YouTrust was the same. The failures were happening. The exit codes just never made it back to the caller.

Back in university I stacked freelance gigs up to ¥600k/month, then got laid off and went back to zero. Over the following six months I built an autonomous Claude Code environment, and I'm now at ¥1.2M/month in revenue. This article is about the hook wiring that holds that environment together — more precisely, about the time I thought I'd wired it up and nothing was actually plugged in.

Why This Mechanism Works

Claude Code has two kinds of hook points: PreToolUse and Stop. PreToolUse interrupts right before Claude invokes a tool; Stop runs right before Claude tries to finish a response. If a hook script returns exit 1, Claude blocks that tool call or Stop action — by specification.

When I learned about this, I immediately wanted to use it as a guardrail for my automation environment. "Don't let an implementation be marked complete without an audit." "Stop if a secret is about to slip into a commit." "Cancel the Action if a designated script fails." You write these in code and control Claude's behavior from the outside.

Speaking frankly as someone who runs this environment: with 171 launchd jobs and multiple Claude Code sessions running at once, something breaks every day without hooks. Config mistakes, environment differences, model whims — hooks let you land each of them. Which is exactly why a state where a hook is only pretending to work and isn't actually stopping anything is the same as a cliff with no guardrail.

Scenarios Readers Will Hit

If you use Claude Code, there's a good chance you'll get caught by one of these.

Pattern A: You wrote your hook as a shell script. The script returns exit 1, but Claude doesn't stop. Check the logs and the exit code reads 0 — even though you're sure you wrote the script correctly.

Pattern B: You call your hook through a JS wrapper. The JS wrapper uses child_process.exec() or $() command substitution to call the inner shell script. The inner script returns exit 1, but the outer JS process receives 0.

Pattern C: You call the script inside a pipeline. You pass stdin through a pipe, like cat input.json | ./hook-script.sh. Without set -o pipefail, only the exit code of the right-hand side of the pipe reaches the caller.

In every case: "I wrote the hook," "it's running," "the logs are there" — and the guard still isn't working. Visually green, actually protecting nothing. That's the essence of a silent bug.

Not "No Failure Occurred," but "The Failure Code Never Arrived"

On the day of the incident, every line of the DM system's log read (exit 0). But the problem was in that log line. This is the code that was actually running:

node "$SCRIPT" "$@"
echo "[$(date '+%F %T')] $LANE done (exit $?)"

echo's arguments are evaluated left to right. $(date '+%F %T') spawns a subshell — and succeeds — and returns. At that instant, $? is overwritten with 0. $? is read after that. In other words, whether the preceding node died with exit 3 or exit 4, this line is syntactically incapable of printing anything but (exit 0).

As evidence, I confirmed via a reproduction test that before the fix exit 4 displayed as 0, and after the fix it displayed as 4.

The frightening part is that it's retroactive. As long as that line is in there, not a single (exit 0) in past logs counts as evidence. "It was green last week and last month too" just means you kept running code that prints green.

Scanning the 328 shell scripts under ~/dev and ~/.claude/scripts turned up a total of 3 instances of this trap. One around Claude Code's hooks, one in the note paid-bonus ZIP attachment script, and one in a dotfiles snapshot script. All of them were in the state of "the logs were printing, but the exit code was dead."

The Overall Flow

First, let's confirm the path by which Claude Code's Stop/PreToolUse hooks call shell scripts.

The Full Call Chain

Claude Code（本体プロセス）
    │
    │  hook event (JSON payload を stdin に渡す)
    ▼
JS hook dispatcher（settings.json で指定）
    │
    │  child_process.spawn() または exec()
    ▼
~/.claude/scripts/hooks/run-with-flags-shell.sh
    │
    │  stdin → パイプ経由で渡す
    │  HOOK_ID / REL_SCRIPT_PATH / PROFILES_CSV を引数で受け取る
    ▼
check-hook-enabled.js（このhookが有効か確認）
    │
    │  enabled ならば
    ▼
$SCRIPT_PATH（実際のフックロジック）
    │
    │  exit 0 / exit 1
    ▼
run-with-flags-shell.sh（終了コードを返す）
    │
    ▼
JS dispatcher（終了コードを受け取る → Claude本体へ）
    │
    ▼
Claude Code（exit 1 なら動作をブロック）

The point of this diagram is whether each arrow passes the exit code correctly. If even one link breaks, the terminal exit 1 never reaches Claude.

The Actual Code of `run-with-flags-shell.sh`

Here is the actual wrapper script (~/.claude/scripts/hooks/run-with-flags-shell.sh).

#!/usr/bin/env bash
set -euo pipefail

HOOK_ID="${1:-}"
REL_SCRIPT_PATH="${2:-}"
PROFILES_CSV="${3:-standard,strict}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(cd "${SCRIPT_DIR}/../.." && pwd)}"

# Preserve stdin for passthrough or script execution
INPUT="$(cat)"

if [[ -z "$HOOK_ID" || -z "$REL_SCRIPT_PATH" ]]; then
  printf '%s' "$INPUT"
  exit 0
fi

# Ask Node helper if this hook is enabled
ENABLED="$(node "${PLUGIN_ROOT}/scripts/hooks/check-hook-enabled.js" "$HOOK_ID" "$PROFILES_CSV" 2>/dev/null || echo yes)"
if [[ "$ENABLED" != "yes" ]]; then
  printf '%s' "$INPUT"
  exit 0
fi

SCRIPT_PATH="${PLUGIN_ROOT}/${REL_SCRIPT_PATH}"
if [[ ! -f "$SCRIPT_PATH" ]]; then
  echo "[Hook] Script not found for ${HOOK_ID}: ${SCRIPT_PATH}" >&2
  printf '%s' "$INPUT"
  exit 0
fi

# Extract phase prefix from hook ID (e.g., "pre:observe" -> "pre", "post:observe" -> "post")
HOOK_PHASE="${HOOK_ID%%:*}"

printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE"

set -euo pipefail is at the top of the file. That's the strictest setting: "exit immediately if a command fails, error on undefined variables, catch failures inside pipes too." The last line, printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE", uses a pipe, but thanks to pipefail, if the right-hand $SCRIPT_PATH returns exit 1, run-with-flags-shell.sh itself also ends with exit 1 — within this script alone.

The problem is outside this script.

What Actually Happens Along the Chain

Point of interest ①: the check-hook-enabled.js call on line 19

ENABLED="$(node "${PLUGIN_ROOT}/scripts/hooks/check-hook-enabled.js" "$HOOK_ID" "$PROFILES_CSV" 2>/dev/null || echo yes)"

If the node inside $() dies from some error, || echo yes fires and ENABLED becomes "yes". In other words, even when the hook-check script itself is broken, the hook proceeds down the execution path as "enabled." This is intentional as a fallback design, but it's a structure where errors in the hook-check logic get swallowed silently.

Point of interest ②: the pipe on the last line

printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE"

Because pipefail is on, $SCRIPT_PATH's exit 1 does propagate properly into this script's exit code. But depending on how the JS dispatcher calls this script, whether that code reaches Claude changes.

If the JS uses child_process.exec(), it determines success or failure by whether the callback's first argument (error) is null. Because exec interposes a shell internally, the shell's exit code arrives as error.code — however, when the exec options set shell: true, there are cases where the shell itself handles the exit 1 and it never reaches the parent process.

If it uses child_process.spawn(), you can get the exit code from the code argument of the close event. That's mostly accurate, but if the spawned process ends via SIGTERM or SIGKILL, code becomes null.

And one more: if JS calls it via command substitution like $(run-with-flags-shell.sh ...) — as explained earlier — $? is reliably destroyed.

The Actual Incident Numbers

The following was confirmed for the incident on 2026-08-29.

Channel	Launches	Actual replies	Alerts fired
X (formerly Twitter)	6 (all ABORT)	0	Fired from the 3rd onward (14:33 / 16:15 / 18:15 / 20:15)
YouTrust	6 (all failed to launch)	0	Not a single one

On the X side, the exit code made it through part of the path to the monitoring layer, so from the third run onward a notification appeared in Discord's #01_alerts. On the YouTrust side, a process.exit(3) in the library layer bypassed the caller's catch block, so the read_failures counter was never incremented and not a single notification reached Discord.

Same day, same root cause (Chrome launch failure), same architectural philosophy — one rang and the other was completely silent. The difference is a single line in a library. That one line silenced 24 hours' worth of outreach DMs.

What's Happening at the Code Level

Here's a minimal sample of the structure where a hook's JS wrapper calls a shell script using $().

// ❌ $() 経由では exit code が潰れる
const { execSync } = require('child_process');

function runHook(scriptPath, input) {
  try {
    // execSync はデフォルトで throws on non-zero exit
    // しかし内部で $() を重ねると話が変わる
    const result = execSync(`echo '${input}' | ${scriptPath}`, {
      encoding: 'utf8',
      shell: true,   // ← ここが問題の温床になりやすい
    });
    return { success: true, output: result };
  } catch (e) {
    // e.status が null になるケースがある
    return { success: false, code: e.status };
  }
}

The shell: true option passes the command string to /bin/sh -c "...". Whether that shell wrapper propagates exit 1 as the process's exit code depends on the shell's implementation and how the arguments are assembled. In particular, when you pass a pipe like echo '...' | script.sh with shell: true, pipefail is not inherited into that shell session, so a failure on the left-hand side gets swallowed.

Meanwhile, the pipe that run-with-flags-shell.sh itself uses on its last line —

printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE"

— is under the control of the set -euo pipefail at the top of the script, so $SCRIPT_PATH's exit 1 correctly surfaces as the script's exit code. This script on its own is correct. The problem is in the calling layer above it.

All 3 traps found in the 328-script scan were the same pattern: "an echo line inside the script mixing $(date) and $?." The detection query can be used as-is.

grep -rn 'exit \$?' --include='*.sh' ~/dev ~/.claude/scripts | grep '\$('

This query picks up lines containing exit $? where $( also appears on the same line. Assignment patterns like || rc=$? (x="$(cmd)" || rc=$?) are correct usage, so that distinction alone can't be made mechanically — it needs a human eye.

Implementation Details

The first half traced the structure of "why exit codes don't arrive." From here we'll look concretely, with real code, at "how to rewrite it so they do." There are 3 fix patterns. Each is a change of 2 lines or less, and each covers a different path.

Why `set -euo pipefail` Alone Doesn't Save the "Outside"

The top of ~/.claude/scripts/hooks/run-with-flags-shell.sh is as follows.

#!/usr/bin/env bash
set -euo pipefail

This line guarantees that "pipefail is enabled within this script's execution context." Indeed, the final line

printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE"

is under pipefail's control, so if $SCRIPT_PATH returns exit 1, run-with-flags-shell.sh itself also ends with exit 1. That part is correct.

The problem is "how the JS dispatcher launches this shell script." If the JS passes shell: true to child_process.exec(), the command string is internally wrapped in /bin/sh -c "...". That /bin/sh session does not inherit pipefail. From JS's point of view, the process tree isn't "/bin/sh → run-with-flags-shell.sh" but "shell wrapper → run-with-flags-shell.sh as its child process." The shell wrapper's own exit code is normally 0.

And another: when JS extracts a result string via execSync in a command-substitution-like way —

const out = execSync(`cat payload.json | ${hookScript}`, { shell: true });

— as long as the left-hand cat payload.json succeeds, whatever the right-hand hookScript returns has no effect on execSync's error determination. The combination of shell: true + a pipe + no pipefail quietly discards the right-hand side's exit code.

Pattern 1: Save `$?` Before Any `$(...)`

The lightest fix. It applies broadly to "cases where a log line mixes $(date) and $?."

# ❌ Before：$? が $(date) で上書きされる
node "$SCRIPT" "$@"
echo "[$(date '+%F %T')] $LANE done (exit $?)"

# ✅ After：STATUS に退避してから $(date) を展開する
node "$SCRIPT" "$@"
STATUS=$?
echo "[$(date '+%F %T')] $LANE done (exit $STATUS)"

STATUS=$? is just a variable assignment, so it doesn't spawn a subshell. $? is secured before the next line's $(date) destroys it. That alone eliminates "the divergence between the code the log prints and the actual exit code."

There is a caveat, though. The assignment form x="$(cmd)" has the exit status of the whole assignment become that of $(cmd), so this is correct usage.

# ✅ これは正しい。x の代入ステータスは node のステータスと同じ
x="$(node "$SCRIPT" "$@")"
STATUS=$?

On the other hand, $? on an echo line that contains $() will reliably be zero. The difference between the two forms can't be distinguished mechanically with grep — after a regex hit, one step of visual inspection is required.

Detection query (identical to the one shown in the first half):

grep -rn 'exit \$?' --include='*.sh' ~/dev ~/.claude/scripts | grep '\$('

Among the lines this query returns, exclude the x="$(...)" form. Everything else needs fixing.

Pattern 2: Use `PIPESTATUS` to Capture Every Command's Code

Use this when you can't rewrite the pipe. PIPESTATUS is a bash-specific array that holds the exit codes of each command in the preceding pipeline, in left-to-right order.

printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE"
PIPE_STATUS=("${PIPESTATUS[@]}")

# PIPE_STATUS[0] = printf の終了コード
# PIPE_STATUS[1] = $SCRIPT_PATH の終了コード

if [[ "${PIPE_STATUS[1]}" -ne 0 ]]; then
  exit "${PIPE_STATUS[1]}"
fi

Since run-with-flags-shell.sh has set -euo pipefail at the top, there's currently no need to add this — with pipefail, a failure on the right-hand side takes the script itself down. You'd make PIPESTATUS explicit when "there's a reason you can't turn on pipefail" or when "you want to record which of the commands failed in the log." If your design records channel-name-and-exit-code pairs in launchd job logs, writing PIPESTATUS[1] directly into the log makes later tracing easier.

Pattern 3: Rewrite the JS Dispatcher to Call Directly

The most fundamental fix. Replace shell: true + exec with spawn so no shell wrapper is involved.

// ❌ Before：shell: true のため $() 内の pipefail が死ぬ
const { execSync } = require('child_process');
const out = execSync(`echo '${payload}' | ${hookScript}`, { shell: true });

// ✅ After：spawn で直接呼ぶ。終了コードは close イベントの code で取れる
const { spawn } = require('child_process');

function runHook(hookScript, payload) {
  return new Promise((resolve, reject) => {
    const child = spawn(hookScript, [], { stdio: ['pipe', 'pipe', 'inherit'] });
    child.stdin.write(payload);
    child.stdin.end();

    child.on('close', (code) => {
      if (code !== 0) {
        reject(new Error(`hook exited with ${code}`));
      } else {
        resolve();
      }
    });
  });
}

Used without shell: true, spawn launches the command directly via execve(2). Since no shell wrapper is interposed, the set -euo pipefail inside run-with-flags-shell.sh stays in effect. The close event's code argument is either null (terminated by signal) or an integer (normal termination). When it's null, check the signal argument to see what happened.

Verify the Guard Lands With a Minimal `exit 1` Repro

You can't stop at "I fixed it." Verify with a minimal reproduction case that Claude's behavior actually gets blocked. Three steps.

Step 1: Create a dummy hook that always returns exit 1.

#!/usr/bin/env bash
# /tmp/test-hook.sh
exit 1

chmod +x /tmp/test-hook.sh

Step 2: Register that hook as a Stop hook in settings.json.

{
  "hooks": {
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "/tmp/test-hook.sh"
          }
        ]
      }
    ]
  }
}

Step 3: Open Claude Code and have it answer something.

The Stop hook runs at the moment Claude tries to finish its response. If /tmp/test-hook.sh returns exit 1, Claude emits a message saying the hook blocked it and loops again. If you see that, the wiring is alive.

If you're calling through a JS dispatcher, passing this test is a separate question from whether the actual hook script goes through spawn. To check, look at what Claude is launching with ps aux | grep hook, or trace execve-family system calls with strace / dtruss. On macOS you can check the system calls of child processes a process launches with dtruss -p <PID>.

Where I Got Stuck

From here I'll write in the order I actually got stuck. Three cases, each as a set of three: symptom, cause, fix. All of them are stories of getting stuck in the state of "I wrote the hook, it's running, the logs are there."

Case 1: 28 Hours of All-Green Logs, Zero Actual Replies

Symptom. On 2026-08-29, I started digging from a single remark: "Aren't DM replies on X (formerly Twitter) slow lately?" Checking the launchd job list, everything was exit 0. The log file had

[2026-08-29 06:15:32] inbox-agent done (exit 0)
[2026-08-29 08:15:17] inbox-agent done (exit 0)
[2026-08-29 10:15:41] inbox-agent done (exit 0)

lined up neatly. Green. But when I checked the actual reply count through X's own interface, it was zero for the entire day.

Cause. inbox-agent/run-medium.sh contained the following line.

node "$SCRIPT" "$@"
echo "[$(date '+%F %T')] $LANE done (exit $?)"

This. node "$SCRIPT" was dying with an error. $? holds node's exit code — right up until just before this line is evaluated. When expanding echo's arguments, bash evaluates left to right. $(date '+%F %T') spawns a subshell, returns a datetime string, and succeeds. At that instant, $? is rewritten to 0. Then $? is read. So no matter what happens, the log says (exit 0).

Fix.

node "$SCRIPT" "$@"
STATUS=$?
echo "[$(date '+%F %T')] $LANE done (exit $STATUS)"

Just add one line to save it into STATUS. After this fix, reproducing the same Chrome launch failure printed (exit 3). launchd received exit 3, and a notification arrived in the Discord alert channel.

What was frightening. All logs from the period that line was in place are invalid. "It was green last week and last month too" is not evidence. It only means "last week and last month, I kept running code that only prints green."

Case 2: The note Paid-Bonus ZIP Attachment Failure Went Unnoticed for 3 Weeks

Symptom. After fixing case 1, I thought "if I've come this far, other scripts might be stepping in the same trap," and scanned 328 shell scripts.

grep -rn 'exit \$?' --include='*.sh' ~/dev ~/.claude/scripts | grep '\$('

Three hits. inbox-agent/run-medium.sh (the main culprit this time), ~/.claude/scripts/dotfiles-snapshot.sh (exit code on commit failure), and note-autolike/scripts/retry-attach-kit.sh.

Checking retry-attach-kit.sh, it was a retry script for attaching the note paid-bonus ZIP to a post. When the attachment API returned 503, the script was written to return exit 1. But because the same trap was in its echo line, launchd recorded it as exit 0.

How long had people who bought the paid note been unable to download the bonus ZIP? Going back through the logs, everything is green, so you can't tell. The only option was to cross-reference the actual date the attachment script last succeeded against the attachment file's update date in note's admin screen. The result: it had been putting out green logs while in a failed state for at least 3 weeks.

Cause and fix. Same pattern.

# ❌ Before
python3 attach_kit.py "$POST_ID" "$ZIP_PATH"
echo "[$(date '+%F %T')] attach done (exit $?)"

# ✅ After
python3 attach_kit.py "$POST_ID" "$ZIP_PATH"
STATUS=$?
echo "[$(date '+%F %T')] attach done (exit $STATUS)"

After the fix, I made it raise an alert to Discord when STATUS ends at 1 or higher. Failures that affect people who bought a paid note aren't something I can afford not to notice until the next morning.

What I took away. If I hadn't immediately generalized the moment I found one instance, this job would still be failing silently today. The same code pattern always exists in multiple places. The correct procedure is not to stop at "one fix" but to scan the whole repository right then.

Case 3: YouTrust Was Failing in "Complete Silence" — A Library's `process.exit()`

Symptom. On that same August 29, I confirmed that on the X side "alerts appeared in Discord from the third run onward." But on the YouTrust side, not a single notification reached Discord. Six launches, six failures, and zero alerts.

Cause. Line 31 of outreach-multi/src/yt/api.mjs had process.exit(3).

// outreach-multi/src/yt/api.mjs（修正前）
async function fetchMessages(page) {
  const resp = await browser.fetch(YT_API_ENDPOINT);
  if (!resp.ok) {
    console.error(`[yt/api] fetch failed: ${resp.status}`);
    process.exit(3);  // ← ここ
  }
  return resp.json();
}

process.exit() is not an exception. Unlike throw, it doesn't walk back up the call stack. The caller's try/catch is never executed once.

The caller's code looked like this.

// エントリポイント（修正前）
try {
  const messages = await fetchMessages(page);
  await handleMessages(messages);
} catch (err) {
  // ここが走ると思っていた
  read_failures++;
  if (read_failures >= ALERT_THRESHOLD) {
    await discord.alert(`YouTrust failure: ${err.message}`);
  }
}

The instant process.exit(3) is called, the node process terminates immediately. The catch block is never reached. read_failures is never incremented. Nothing goes to Discord. launchd sees "the process terminated," but because that exit code went down a path where it was recorded nowhere, everything went silent.

Meanwhile, the reason the X side could fire alerts from the third run onward is that the X module was written with throw new Error(...). The catch block ran, read_failures incremented, crossed the threshold, and reached Discord. Same root cause of "Chrome launch failure," same architectural philosophy, but with the wiring differing by one line, one rang and the other was completely silent.

Fix.

// ✅ After：ライブラリ層は throw する。process.exit() しない
async function fetchMessages(page) {
  const resp = await browser.fetch(YT_API_ENDPOINT);
  if (!resp.ok) {
    throw new Error(`yt/api fetch failed: ${resp.status}`);
  }
  return resp.json();
}

There was an option to branch on an environment variable YT_API_THROW=1, but since there's no reason to write process.exit in the library layer in the first place, I simply rewrote it to throw.

The general rule I kept. The library layer must not hold the authority to terminate. Only the entry point may end the process. The instant a lower layer calls process.exit / sys.exit / os.Exit, all the observation, cleanup, and notification the upper layer prepared gets bypassed. Frameworks and external libraries sometimes do this too, so I've made it a habit to check grep -r 'process\.exit' node_modules/<packagename>/ when adding a new dependency.

The Common Pattern Across All 3

The symptom in every case was "the logs are green, the actual result is zero or failed." But the layer of the cause differs.

Case	Symptom	Layer of the cause	Fix
inbox-agent	Zero DM replies, all-green logs	The `echo` line in a shell script	Save first with `STATUS=$?`
note-autolike	ZIP attachment failure unnoticed for 3 weeks	Same as above	Same as above + added a Discord alert
YouTrust	6 silent failures out of 6	`process.exit` in a Node.js library layer	Rewritten to `throw`

The shell trap and the Node.js trap look different, but the root is the same: "written without providing a path for the failure signal to reach the caller." Whether a hook's guard actually lands can only be determined by measuring "does the caller stop when it actually returns exit 1?" — not by green logs.

Pitfalls

The first and middle sections traced 3 incidents, but there are more places where you can step in the same trap. Here I cover the pitfalls I found scanning 328 scripts, plus the patterns I keep getting caught by in Claude Code hook wiring.

(1) Any line where echo "... $?" has $(...) mixed in is out

No matter how short the line, if you write echo "[$(date)] done (exit $?)", that line is structurally incapable of printing anything but (exit 0). The instant date's subshell succeeds, $? is rewritten. The desire to "write it in one line" is right, but this particular combination simply cannot work. Your only options are to take STATUS=$? first, or to drive $? out of the echo.

(2) set -euo pipefail only affects the current shell context

Even with set -euo pipefail at the top of run-with-flags-shell.sh, if JS passes shell: true to child_process.exec() to call this script, a shell wrapper /bin/sh -c "..." is interposed. That /bin/sh session does not inherit pipefail. Calling the script alone with bash run-with-flags-shell.sh works correctly, but calling it from JS doesn't — "the script was written correctly" is the truth, and "the way JS calls it breaks it" is the cause.

(3) The || echo yes fallback swallows checker failures

Line 19 of run-with-flags-shell.sh reads like this.

ENABLED="$(node "${PLUGIN_ROOT}/scripts/hooks/check-hook-enabled.js" "$HOOK_ID" "$PROFILES_CSV" 2>/dev/null || echo yes)"

If check-hook-enabled.js breaks and dies, || echo yes fires and ENABLED becomes "yes". It's an intentional fail-safe design, but the state of "the hook-check script itself is broken, yet the hook proceeds as enabled" happens silently. Whether hooks are being judged correctly is predicated on this checker working properly.

(4) A missing script also passes through with exit 0

Look at lines 25–30 of the same script.

SCRIPT_PATH="${PLUGIN_ROOT}/${REL_SCRIPT_PATH}"
if [[ ! -f "$SCRIPT_PATH" ]]; then
  echo "[Hook] Script not found for ${HOOK_ID}: ${SCRIPT_PATH}" >&2
  printf '%s' "$INPUT"
  exit 0
fi

If the hook script's path is wrong and the file doesn't exist, it prints an error to stderr and ends with exit 0. Claude doesn't stop. When you've misconfigured something, the logs alone can't distinguish "the hook isn't wired in" from "the hook judged correctly and let it pass."

(5) The instant process.exit() is called in a library layer, every observation path is wiped out

Line 31 of outreach-multi/src/yt/api.mjs was this pattern. The caller's try/catch block never executed once, the read_failures counter was never incremented, and not a single notification reached Discord. In contrast, the X side was written with throw new Error(...), so it rang. Two modules written with the same architectural philosophy split into "rings / completely silent" over a one-line difference.

(6) PIPESTATUS is bash-specific — the name differs in zsh

PIPESTATUS is a bash-only array. In zsh, pipestatus (lowercase) is available as the equivalent array, but with a #!/bin/sh shebang there are shells where it's unusable. run-with-flags-shell.sh's shebang is #!/usr/bin/env bash, so no problem there, but if the hook script side is written in sh, referencing PIPESTATUS will come back empty.

(7) The exit code launchd receives and the script's exit code are different things

The exit code shown in a launchd job log is that of the job's outermost process. If there's a multi-layer call chain and the outermost layer ends with exit 0, launchd records exit 0 no matter how many exit 1s happened inside. "launchd shows exit 0 for everything" is not equal to "everything down to the end of the hook chain was exit 0."

(8) The trap of assuming execSync's default behavior means "an exception means failure"

execSync throws on a non-zero exit code, but when you're using a pipe with shell: true, the shell wrapper can return exit code 0. No exception occurred ≠ success; no exception occurred = the shell returned 0. The inner script's failure isn't transparent through it.

(9) A new npm package may call process.exit()

When an external library calls process.exit() internally, the entry point's try/catch gets bypassed. Without the habit of checking when you add a dependency, it surfaces as the symptom "alerts that used to ring stopped ringing after the addition."

(10) "The logs are printing" is not "the guard is landing"

On the day of the incident, the reason no notification appeared in Discord's #01_alerts was not "no failure occurred." It was "the failure code never reached the path that rings the alert." Logs printing = the script launched; the guard landing = the exit code propagated to the caller. Those are two different facts.

(11) grep can mechanically knock these out, but the || rc=$? form needs human eyes

The detection query grep -rn 'exit \$?' --include='*.sh' ~/dev ~/.claude/scripts | grep '\$(' is effective, but the form x="$(cmd)" || rc=$? is correct usage, so it must be excluded from the hits. Since an assignment statement's exit status becomes that of the command substitution, reading $? on the line after x="$(node ...)" is correct. Only the cases mixing $(...) and $? inside an echo line are the problem. You can't knock everything out automatically — one step of visually inspecting the hits remains.

(12) Even after switching to throw with YT_API_THROW, it's unproven until a failure actually occurs

The YouTrust fix I wrote about in the middle section — rewriting process.exit(3) to throw new Error(...) — has been confirmed statically with node --check and grep, but whether read_failures actually increments and shows up in Discord when Chrome really fails to launch will only be proven the next time a launch failure occurs. "The code is correct" and "the path went through in production" are two different facts.

Best Practices

Here are the lessons from 3 incidents and a 328-script scan, organized as guidance for the next time I build the same kind of environment.

1. Drill the rule of taking STATUS=$? first into your body

If you want to reference $? immediately after a command, always save it on one line with STATUS=$?. Physically forbid $(...) and $? from coexisting inside an echo. Make "writing a shell script = avoiding this combination" a reflex.

2. Write set -euo pipefail at the top of every shell script

When creating a new script, the first two lines are always

#!/usr/bin/env bash
set -euo pipefail

When touching an existing script, check the top first when you open the file. Read scripts that lack this as "ready and prepared to fail silently."

3. Use spawn when calling external scripts from JS

Calls that pass shell: true to child_process.exec() risk the shell wrapper swallowing the exit code. Using child_process.spawn() without shell: true launches directly via execve(2). The close event's code argument is the exit code. When it's null, check the signal argument for signal termination.

4. Don't write process.exit() / sys.exit() / os.Exit() in a library layer

Only the entry point may terminate the process. The instant a library calls process.exit, the catch blocks, notifications, counter increments, and cleanup the entry point prepared are all bypassed. The iron rule is: the library layer throws exceptions and passes them upward.

5. Check for process.exit when adding a new dependency

grep -r 'process\.exit' node_modules/<packagename>/

Make it a habit to run this query before adding. If the library calls process.exit internally, it bypasses the entry point's catch block. Checking at addition time is cheaper than finding out later via the symptom "alerts stopped ringing."

6. Measure whether "the guard lands" with a minimal exit 1 dummy

Once you've written a hook, always measure before trusting the wiring.

#!/usr/bin/env bash
# /tmp/test-hook.sh
exit 1

Register this dummy as a Stop hook in settings.json and have Claude answer something. If a message saying the hook blocked it appears, the wiring is alive. If it doesn't, the exit code is dead somewhere in the call chain. Take this one step before trusting green logs.

7. Generalize the instant you find one

The same code pattern always exists in multiple places. Because I immediately scanned everything the moment I found it in inbox-agent/run-medium.sh, I discovered that the paid-note bonus ZIP attachment failure in note-autolike/scripts/retry-attach-kit.sh had been going on for over 3 weeks. Had I stopped at one fix, buyers might still not be receiving their bonus.

The detection query can be used as-is.

grep -rn 'exit \$?' --include='*.sh' ~/dev ~/.claude/scripts | grep '\$('

Visually inspect the hits and exclude the x="$(cmd)" || rc=$? form; everything else needs fixing.

8. Don't treat "no alert fired" as evidence of health

On the X side, notifications came into Discord's #01_alerts from the third run onward. On the YouTrust side, not one arrived across 6 launch failures. "Discord was quiet" is not "there was no problem" — it includes the possibility that "the failure code never reached the path that rings the alert." When you add an alert, run the actual alert path once in something close to production before saying "monitoring is working."

9. Know the script-not-found path in run-with-flags-shell.sh

Lines 27–30 of the real code print an error to stderr and end with exit 0 when the script file doesn't exist. If you get the hook's configured path wrong, the error goes to stderr but Claude doesn't stop. When a hook feels like it "isn't working," check the stderr log first.

10. Know the || echo yes fallback in check-hook-enabled.js

The fallback on line 19 of run-with-flags-shell.sh is designed to proceed with the hook enabled even when the hook-check script is broken. That's an intentional fail-safe, but the state of "the checker itself is broken" proceeds silently. If a hook isn't being disabled as expected, check whether this checker is working properly.

11. If you use PIPESTATUS, align the shebang and the environment

PIPESTATUS is bash-specific. There are shells where it can't be used in a #!/bin/sh script. If you want to record each pipeline command's exit code individually, set the shebang to #!/usr/bin/env bash and make PIPESTATUS explicit.

printf '%s' "$INPUT" | "$SCRIPT_PATH" "$HOOK_PHASE"
PIPE_STATUS=("${PIPESTATUS[@]}")
# PIPE_STATUS[1] が $SCRIPT_PATH の終了コード

With pipefail, $SCRIPT_PATH's failure automatically propagates into the script's exit code, but if your design records "which command failed" in a log, make PIPESTATUS explicit.

12. Don't treat launchd's all-exit 0 as primary evidence

The exit code launchd records is that of the outermost process. In an environment with multi-layer wrappers, "launchd says exit 0" ≠ "everything inside was exit 0" either. Confirm actual success or failure with measured values from what the job produced — reply counts, presence of attached files, API responses.

13. Distinguish "the code is written correctly" from "the path went through in production"

Even if the fixed code is syntactically correct, the error-case path actually runs "the next time a failure happens in production." Reproduce the failure in a test environment, or deliberately cause a failure and confirm the alert arrives, before saying "fix complete." "The code is correct" is not "verified working."

Summary

The structure of the problem in one line: "The failures were happening. The exit codes just never arrived at the caller."

The single line echo "[$(date)] $LANE done (exit $?)" in inbox-agent/run-medium.sh had no syntax errors as a shell script, printed logs, and looked normal. But the (exit 0) that line was printing wasn't the result of the preceding node — it was the result of $(date) succeeding. The cause of 28 hours of zero outreach DM replies was that one line being "code that only prints green."

The process.exit(3) at outreach-multi/src/yt/api.mjs:31 is the same. It detected the error. But because it terminated the process at the place of detection, neither the counter nor the notification the entry point had prepared ever ran. From the same root cause of a Chrome launch failure, the X side rang Discord from the third run onward, and the YouTrust side was completely silent across all 6. The difference is a single line of wiring.

When measuring the "health" of an automation environment, the number of green log lines is not trustworthy. The only trustworthy thing is the measurement: "when it actually returns exit 1, does that signal reach the calling Claude Code and cause a block?" In an environment where 171 launchd jobs and multiple Claude Code sessions run simultaneously, a hook that hasn't had this measurement done is the same as a cliff with no guardrail.

It took several hours to scan 328 scripts, find 3 instances of the same trap, and commit fixes to 3 repositories. But the fact that "buyers of the paid note hadn't been able to download the bonus ZIP for over 3 weeks" only came to light because I scanned. Had I stopped at one, the green would still be lining up today.

An exit code doesn't arrive unless every layer of the propagation path is written correctly. The shell $? problem, the JS shell: true problem, the library-layer process.exit problem — all of them share the same root. Once you write a hook, verify with a minimal exit 1 dummy that it actually lands. That alone crushes most silent bugs like this one in advance.

I've written up the full picture of the system, the breakdown of ¥1.2M/month, and the 30-day procedure in a paid note.
📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

8 Runs a Day Became 3: Building a launchd plist Integrity Guard in 73 Lines

Lily — Mon, 31 Aug 2026 00:00:07 +0000

At 12:04 on August 23, 2026, something quietly rewrote my outreach DM schedule down to three runs a day — 2:31 AM, 10:31 AM, and 6:31 PM. For the several days it took me to notice, that cost me dozens of missed opportunities every single day. I still don't know what did it.

Why this mechanism works

The problem: you can't tell when automation has stopped

When you strip away the details, making money as a solo developer comes down to one equation: automation uptime = revenue. I was a college student making 100k yen a month, then 600k a month juggling multiple gigs, then back to zero when I was laid off for company reasons — and the reason I could build a Claude Code autonomous environment in six months and get to 1.2M yen in monthly revenue is that I had an outreach DM lane running while I slept.

Automated outreach DMs are made up of four launchd jobs. Instagram, Threads, follower prospecting, and YouTube each have their own independent plist and Python script. These control their start times via StartCalendarInterval. For example, com.lily.outreach-ig runs eight times at 8:20, 10:20, 12:20, 14:20, 16:20, 18:20, 20:20, and 22:20, and com.lily.outreach-th runs seven times every two hours from 9:38 to 21:38 — sends happen during daytime hours.

The problem is that a launchd plist is just an XML file, and anyone (or anything) can rewrite it. macOS does not notify the user when a plist changes. Even when launchctl list shows a job as "running," the schedule itself may be operating on post-rewrite values. That is exactly what happened on August 23: the job was alive. It was just running on a completely different setting — "three times a day, in the middle of the night."

What tipped me off was looking at the DM send-count log manually one evening. The noon and 4 PM slots kept showing zero, so I dug in and found that StartCalendarInterval inside com.lily.outreach-ig's plist had been reduced to just three entries: 2:31 / 10:31 / 18:31. At the same time, com.lily.outreach-th had been cut from seven runs to three. The mtime (last modified time) was 2026-08-23 12:04.

What I learned from this is that "running" and "running correctly" are two different things. What you need to monitor is not whether the process is alive, but the configuration values inside the plist themselves.

The value of designing for "record," not just "restore"

If all you want is a simple integrity check, you could implement it with cron or Watch Paths. But what I insisted on this time was a design that leaves a forensic log at the same moment it restores.

There are two reasons. The first is reproducibility. If the same rewrite happens repeatedly, a snapshot of which processes were running just before it gives me a way to narrow down the suspects. The second is the peace of mind of being able to trace why it changed after the fact. An automation environment is complex, and I can't rule out that I rewrote it myself by mistake from some script. With logs, I can separate "my own mistake" from "outside interference."

When outreach-schedule-guard.sh detects something, it does four things: record the rewrite along with the mtime, capture a list of recently running processes, copy the plist as a timestamped backup, and restore the correct values and make launchd reload them. Of these, the forensic part takes up close to half of the script's 50 lines. I could have taken the "it just needs to work" route and only done the restore, but I didn't want to give up on finding the cause.

The structural weak point in a launchd plist

Once you understand the structure of StartCalendarInterval, you see why reading it with Python's plistlib is the best approach.

launchd plists are stored in XML format (or in binary format via plutil -convert binary1). Taking com.shun.self-repair.plist as an example, it has an array structure like this:

<key>StartCalendarInterval</key>
<array>
  <dict>
    <key>Hour</key>
    <integer>9</integer>
    <key>Minute</key>
    <integer>20</integer>
  </dict>
  <dict>
    <key>Hour</key>
    <integer>13</integer>
    <key>Minute</key>
    <integer>30</integer>
  </dict>
  <dict>
    <key>Hour</key>
    <integer>19</integer>
    <key>Minute</key>
    <integer>30</integer>
  </dict>
</array>

The key point is that Hour and Minute are stored as integer types. Parsing XML with grep or awk in a shell script gives you no type guarantees, and it doesn't work at all against binary plists. If you read the plist with python3 -c "import plistlib", you can handle both text and binary formats through the same API, and you can reliably extract Hour as an int.

Rewrite detection is just a string comparison between this read value and the expected value from the SPECS array. The expected value is generated in the form "8:20,10:20,12:20,14:20,16:20,18:20,20:20,22:20", and if it doesn't match the value pulled out of the actual plist, an alert fires.

The overall flow

Architecture overview

定期実行（launchd job）
    │
    ▼
outreach-schedule-guard.sh
    │
    ├─ SPECS配列から期待スケジュールを生成
    │    com.lily.outreach-ig   → 8,10,12,14,16,18,20,22 時 :20
    │    com.lily.outreach-th   → 9,11,13,15,17,19,21    時 :38
    │    com.lily.followers-outreach → 8,10,12,14,16,18,20,22 時 :35
    │    com.lily.outreach-yt   → 10,12,14,16,18,20      時 :52
    │
    ├─ python3 plistlib で各 plist の StartCalendarInterval を実読み取り
    │
    ├─ [一致] → 処理なし・終了
    │
    └─ [不一致] ─────────────────────────────────────────┐
                                                          │
         ① forensicログ出力                              │
            - plist の mtime を記録                       │
            - 現在値 vs 期待値を並べて記録               │
            - ps で python/node/bash/launchctl/plutil      │
              の直近プロセスをキャプチャ（最大25件）     │
                                                          │
         ② バックアップ作成                              │
            $label.plist.bak-guard-YYYYmmdd-HHMMSS       │
                                                          │
         ③ python3 plistlib で正しい値に書き戻し         │
            d['StartCalendarInterval'] = [                │
              {'Hour': int(h), 'Minute': minute}          │
              for h in hours.split(',')                   │
            ]                                             │
                                                          │
         ④ plutil -lint で整合性検証                     │
            OK → launchctl bootout → sleep 1              │
                → launchctl bootstrap → 復元ログ         │
            NG → "🔴 復元後のplistが壊れている"ログ      │
                 → 手動対応に委ねる                       │

Core logic: the plist reading part

The script processes the four jobs in order with a shell for loop. The first thing each iteration does is read the plist through a python3 heredoc and print a CSV of Hour:Minute to standard output.

actual="$(python3 - "$plist" <<'PY'
import plistlib, sys
try:
    with open(sys.argv[1],'rb') as f: d = plistlib.load(f)
    rows = d.get('StartCalendarInterval') or []
    if isinstance(rows, dict): rows = [rows]
    print(','.join(f"{r.get('Hour')}:{r.get('Minute')}" for r in rows))
except Exception as e:
    print('ERR')
PY
)"

The reason for opening in binary mode with open(sys.argv[1], 'rb') is that plistlib.load() auto-detects whether the plist is XML or binary. rows = d.get('StartCalendarInterval') or [] guards against None when the entry doesn't exist. And the isinstance(rows, dict) check is defensive handling so things don't break when StartCalendarInterval is written as a single <dict> rather than an array (the case where only one time is registered).

The resulting actual becomes a string like "8:20,10:20,12:20,14:20,16:20,18:20,20:20,22:20".

Core logic: generating and comparing expected values

Each entry in the SPECS array is defined in a custom "label:minute:hours" format.

SPECS=(
  "com.lily.outreach-ig:20:8,10,12,14,16,18,20,22"
  "com.lily.outreach-th:38:9,11,13,15,17,19,21"
  "com.lily.followers-outreach:35:8,10,12,14,16,18,20,22"
  "com.lily.outreach-yt:52:10,12,14,16,18,20"
)

From there, label, minute, and hours are split apart with shell parameter expansion, and expected is assembled.

label="${spec%%:*}"; rest="${spec#*:}"
minute="${rest%%:*}"; hours="${rest#*:}"

Next, the hours are split into an array with IFS=',', and a CSV in "Hour:Minute" format is generated. For outreach-ig, expected becomes "8:20,10:20,12:20,14:20,16:20,18:20,20:20,22:20".

The single line [ "$actual" = "$expected" ] && continue is the heart of the check. If they match, that job is skipped and we move to the next. The moment there's a mismatch, the forensic recording and restore sequence kicks off.

Core logic: the restore part

Writing back is done with a python3 heredoc, same as reading.

python3 - "$plist" "$minute" "$hours" <<'PY'
import plistlib, sys
path, minute, hours = sys.argv[1], int(sys.argv[2]), sys.argv[3]
with open(path,'rb') as f: d = plistlib.load(f)
d['StartCalendarInterval'] = [{'Hour': int(h), 'Minute': minute} for h in hours.split(',')]
with open(path,'wb') as f: plistlib.dump(d, f)
PY

The reason minute is received via sys.argv[2] and cast with int() is that the value coming from the SPECS string is a str. plistlib.dump(d, f) writes in XML format by default (fmt=plistlib.FMT_XML), which has the side effect of converting an originally binary plist to XML. That said, launchd reads XML-format plists without any problem, so it doesn't matter in practice.

After writing back, the XML structure is validated with plutil -lint "$plist". Feeding a broken plist to launchd causes unpredictable behavior, so this gate can't be skipped. Only when validation passes do we re-register with launchd in the following order:

launchctl bootout "gui/$(id -u)/$label" 2>/dev/null
sleep 1
launchctl bootstrap "gui/$(id -u)" "$plist"

The reason for putting sleep 1 between bootout and bootstrap is to give launchd a grace period to update its internal state. If you omit it, there are cases where bootstrap returns an error. The failure of bootout is ignored with 2>/dev/null because even if it was already unloaded, there's no problem as long as bootstrap goes through.

Implementing the forensic log

Three kinds of information are recorded the instant a rewrite is detected.

log "$label: 書き換え検知 mtime=$(stat -f '%Sm' -t '%F %T' "$plist")"
log "$label:   現在   = $actual"
log "$label:   あるべき= $expected"
ps -Ao pid,lstart,comm | tail -n +2 | \
  grep -iE "python|node|bash|launchctl|plutil" | tail -25 | \
  while read -r l; do log "$label:   ps> $l"; done

stat -f '%Sm' -t '%F %T' is macOS-specific formatting that gets the plist's last modified time in YYYY-MM-DD HH:MM:SS format. This is the only physical evidence of when it was rewritten.

The ps filter targets python|node|bash|launchctl|plutil because the means of rewriting a plist are roughly limited to those commands. tail -25 caps it at 25 lines so the log doesn't balloon.

In the actual incident at 12:04 on August 23, this forensic log would have let me narrow down the suspect processes. Ironically, the guard script itself was built after the incident, so there's no ps snapshot from that time. The "I wish I'd built this earlier" regret is exactly what drove the design of this forensic recording mechanism.

Implementation details

`set -uo pipefail` — why I dropped `-e`

The top of the script is set -uo pipefail. There's no -e.

set -uo pipefail
export PATH="/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin"

I originally wrote it as -euo pipefail, since "exit immediately on error" is supposed to be best practice. But then a bug showed up where the script died abruptly in the forensic ps output section. Here's the cause:

ps -Ao pid,lstart,comm | tail -n +2 | while read -r p rest2; do echo "$p $rest2"; done \
  | grep -iE "python|node|bash|launchctl|plutil" | tail -25 \
  | while read -r l; do log "$label:   ps> $l"; done

grep -iE "python|node|bash|..." returns exit code 1 when there are zero matching lines. Under set -e, a non-zero exit code means immediate exit — so the bug was that the script died in the perfectly normal state where not a single suspicious process was running. The correct behavior for the forensic part is "output nothing if there's nothing," but -e turned that into "die if there's nothing."

By dropping -e and keeping only pipefail, I can still detect unintended silent failures (errors partway through a pipe) while letting an empty grep continue as-is.

Explicit PATH and the log function

export PATH="/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin"
LOG="$HOME/.claude/logs/outreach-schedule-guard.log"

launchd jobs do not inherit the user's shell environment. Even if PATH looks fine from Terminal, /opt/homebrew/bin may not exist in the shell launchd starts. Since python3 is installed via Homebrew, without an explicit PATH the job silently dies with python3: command not found right after launch. From this experience, I've made it a habit to always override PATH at the top of any shell script meant for launchd.

The log function is printf-based.

log() { printf '%s %s\n' "$(date '+%F %T')" "$*" >> "$LOG"; }

I originally wrote it as echo "$(date '+%F %T') $*" >> "$LOG". But echo's behavior changes depending on the shell implementation and whether the -e option is present. When the string being appended to a log message contains \n, the output differs depending on whether echo expands it. printf cleanly separates the format string from the values, so line breaks and whitespace in log output are identical in every environment.

Guarding against a missing plist

[ -f "$plist" ] || { log "$label: plist が無い"; continue; }

If a label defined in the SPECS array doesn't actually have a plist, python3's open() raises an error and the script stops. The design is to run an existence check, leave a log, then move on to the next job. Having "plist missing" in the log lets me later determine whether it was unloaded on the launchd side or the file itself was deleted.

Naming the backups

The current plist is always copied before restoring.

cp "$plist" "$plist.bak-guard-$(date +%Y%m%d-%H%M%S)"

That leaves a file like com.lily.outreach-ig.plist.bak-guard-20260823-120404. The reason for burning the timestamp into the filename is that when the same plist gets rewritten multiple times, I can tell which point in time each one is from with nothing but ls. Using .bak-guard-<datetime> instead of .bak as the extension is also deliberate — it gives the file a name that's unlikely to be recognized as a plist, so launchd doesn't load it by mistake.

For now, I haven't added automatic deletion of backups. If the same plist gets rewritten many times in a short period, the .bak files pile up — but that pile has value in itself as evidence that "the rewrites are recurring," so deletion stays manual.

What the self-repair job's plist teaches you

com.shun.self-repair.plist is the plist for the launchd job that periodically starts the outreach-schedule-guard script itself. Reading this plist directly, there are several values worth referencing as background-job design.

<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>10</integer>
<key>ProcessType</key>
<string>Background</string>
<key>RunAtLoad</key>
<false/>

LowPriorityIO: true lowers IO priority so it doesn't get in the way of other processes' disk access. The guard script only reads/writes plists and writes logs, so low IO priority is fine.

Nice: 10 lowers CPU priority by 10 steps. 0 is the default, and higher values mean lower priority. It would defeat the purpose if the monitoring job hogged CPU and slowed down the main outreach DM scripts, so I lower it explicitly.

ProcessType: Background means it's classified under the "Background" category in macOS Activity Monitor. Combined with LowPriorityIO, it becomes subject to the OS's battery and CPU optimizations.

RunAtLoad: false disables immediate execution right after loading. launchd has a feature that triggers a job once the moment it's loaded; with true, it runs the instant you launchctl bootstrap. The guard script only needs to run on schedule via StartCalendarInterval, so I've turned off the unnecessary immediate run.

StartCalendarInterval is three times a day: 9:20, 13:30, and 19:30. Outreach DMs run from 8 AM to 10 PM, yet the guard only runs three times, because rewrites don't happen continuously — the pattern is "everything gets rewritten at once and goes unnoticed for a while." Three times a day is enough to detect it, and since the check itself reads/writes plists, I want to minimize that.

Where I got stuck

Snag ①: the python heredoc broke from variable expansion

The first code I wrote used <<PY instead of <<'PY'.

# NG: クォートなしヒアドキュメント
actual="$(python3 - "$plist" <<PY
import plistlib, sys
...
PY
)"

An unquoted <<PY makes the shell expand variables inside the heredoc contents. Even if there isn't a single $() or $variable in the Python code, there are cases where, for example, the [1] in sys.argv[1] becomes a target for glob expansion in a zsh environment. In practice [1] wasn't expanded, but on another line the { and } inside f"{r.get('Hour')}:{r.get('Minute')}" were interpreted by the shell and produced a syntax error.

Wrapping it in single quotes as <<'PY' passes the heredoc contents to the Python interpreter as-is. Ever since, whenever I pass python/ruby/node code from Bash via a heredoc, I always use the <<'EOXX' form.

Snag ②: `StartCalendarInterval` returns a dict when there's a single entry

When you read a plist with plistlib, the value of StartCalendarInterval is normally a list (a Python array). But when the plist has a <dict> written directly without an <array> tag, plistlib returns a dict.

<!-- 1時刻だけ登録したplistの例 -->
<key>StartCalendarInterval</key>
<dict>
  <key>Hour</key><integer>10</integer>
  <key>Minute</key><integer>0</integer>
</dict>

At first I had rows = d.get('StartCalendarInterval') or [] and looped with for r in rows. Iterating a dict yields the string keys 'Hour' and 'Minute', so r.get('Hour') becomes a get on a string and returns None. The output became "None:None", the comparison naturally didn't match, and the restore process ran every single time — an endless loop.

# 修正後
rows = d.get('StartCalendarInterval') or []
if isinstance(rows, dict): rows = [rows]

The fix was the one line where the isinstance(rows, dict) check wraps a single dict in a list. Until I added it, restore logs kept appearing every time I verified behavior with a test plist that had only one time set — I initially thought it was a logic bug and spent hours searching in completely the wrong places.

Snag ③: not knowing `${expected:+,}` and getting a leading comma every time

Here's the code I first wrote for generating the expected CSV.

expected=""
for h in "${harr[@]}"; do expected="${expected},${h}:${minute}"; done

For com.lily.outreach-ig, once the loop ran, expected became ",8:20,10:20,12:20...". There's a comma at the front. Naturally it never matched actual ("8:20,10:20,..."). Mismatch every time meant restore every time — the guard script itself was rewriting the plist on every run.

Bash's ${var:+value} parameter expansion means "expand value if var is non-empty, empty string if it's empty."

for h in "${harr[@]}"; do expected="${expected}${expected:+,}${h}:${minute}"; done

With ${expected:+,}, the behavior becomes "insert a comma first only when there's already something there." On the first element expected is empty so no comma is inserted, and from the second onward a , goes in. The result is the correct CSV: "8:20,10:20,12:20,14:20,16:20,18:20,20:20,22:20".

I knew Bash parameter expansions like :- (default value) and := (assignment) well, but :+ (expand only when non-empty) was something I'd never been conscious of until I hit this bug.

Snag ④: `launchctl bootstrap` returned an error immediately

The first version of the restore process called bootstrap right after bootout.

launchctl bootout "gui/$(id -u)/$label" 2>/dev/null
launchctl bootstrap "gui/$(id -u)" "$plist"  # sleep なし

Run in that order, bootstrap failed with Load failed: 5: Input/output error. After receiving bootout, launchd completes the job's termination processing asynchronously and internally. Even when bootout returns 0, hitting bootstrap at a moment when launchd's internal state hasn't been cleared yet causes a race and an error.

launchctl bootout "gui/$(id -u)/$label" 2>/dev/null
sleep 1
if launchctl bootstrap "gui/$(id -u)" "$plist" 2>/dev/null; then
  log "$label: 復元して再読込した"
else
  log "$label: 🔴 bootstrap に失敗した（手動確認が要る）"
fi

Inserting sleep 1 resolved it. "Wait one second" isn't an elegant solution, but launchd's documentation says nothing about guarantees around asynchronous completion, and I confirmed empirically that one second is a sufficient safety margin, so I adopted it rather than spending more time investigating.

In the field, "it doesn't work for some reason" → "it worked after I added sleep 1" tends to be the kind of fix where you move on without the reason ever becoming clear. Here, leaving 🔴 bootstrap に失敗した（手動確認が要る） in the log builds a safety net so I'd notice immediately if it failed again.

Snag ⑤: why `rest2` was needed in `read -r p rest2` in the ps pipe

The forensic ps handling actually didn't work when I first tried to write it simply.

# 最初のナイーブな実装
ps -Ao pid,lstart,comm | grep -iE "python|node|bash|launchctl|plutil" | tail -25

This looked like it worked, but because bash is included in the grep, the bash process handling the pipe itself matched the grep and got mixed into the output. Also, since ps output column widths vary by environment, reading fields with a while loop using only read -r l pulled trailing whitespace into the variable and made the log messy.

The actual code is a two-stage pipe.

ps -Ao pid,lstart,comm | tail -n +2 | while read -r p rest2; do echo "$p $rest2"; done \
  | grep -iE "python|node|bash|launchctl|plutil" | tail -25 \
  | while read -r l; do log "$label:   ps> $l"; done

The rest2 in while read -r p rest2 looks unused, but it's important. With read -r p rest2, p gets the first field (PID) and rest2 gets all remaining fields. Re-emitting with echo "$p $rest2" afterwards produces a formatted line, excluding the header row (already removed with tail -n +2). If you receive whole lines with just read -r l, the extra spaces from ps's formatting are preserved; by decomposing into p rest2 and reassembling, runs of spaces are normalized to a single space.

As a result, the ps> lines in the forensic log come out in a readable form like 12345 Fri Aug 23 12:04:11 2026 python3. If this format had been in place when the August 23 incident happened, the very first thing I could have done was cross-check that pid, execution time, and command name.

In the next part, we'll look at the procedure for registering this guard script itself with launchd, along with real examples of the log output from when the guard actually fired (when I deliberately rewrote a plist to test it).

Gotchas

In addition to the five snags listed in part 2, here's a roundup of problems I actually ran into while building this. Bullet points, comprehensively.

① self-repair.plist's ProgramArguments didn't call the script directly

When I first read com.shun.self-repair.plist, ProgramArguments wasn't what I expected.

<array>
  <string>~/.claude/scripts/claude-quota-guard.py</string>
  <string>--job</string>
  <string>com.shun.self-repair</string>
  <string>--</string>
  <string>/bin/bash</string>
  <string>~/.claude/scripts/self-repair.sh</string>
</array>

Rather than calling /bin/bash self-repair.sh directly, it goes through claude-quota-guard.py. At first I didn't understand "why the extra layer," and I was testing by naively writing /bin/bash outreach-schedule-guard.sh directly into ProgramArguments. It's a design that anticipates the guard script itself having processing that calls the Claude API (including future extensions) and places it under the umbrella of quota management. I later established a rule that all guard-type jobs go through this wrapper.

② The job shows as "running" in launchctl list but the PID stays -

Right after launchctl bootstrap succeeded, I checked launchctl list | grep com.shun.self-repair and the PID was - with status 0. I panicked — "did the load fail?" — but this is normal behavior. With RunAtLoad: false and scheduled execution via StartCalendarInterval, launchd just registers the job and doesn't start it until the next scheduled time. A PID of - means "waiting." Only after one of 9:20, 13:30, or 19:30 has passed and you check launchctl list again does a PID value appear. The correct verification procedure is not to check liveness by PID, but to watch the log file with tail -f after the scheduled time.

③ $HOME comes out empty unless you specify HOME in EnvironmentVariables

Before I put an EnvironmentVariables block in the plist, I hit a bug where the path to $HOME/.claude/logs/ inside self-repair.sh became an empty string. launchd only partially inherits the GUI session's user environment variables, and there are cases where a shell script starts with HOME undefined.

<key>EnvironmentVariables</key>
<dict>
  <key>HOME</key>
  <string>/Users/実ユーザー名</string>
  <key>LANG</key>
  <string>en_US.UTF-8</string>
  <key>PATH</key>
  <string>/Users/実ユーザー名/.nvm/versions/node/v24.13.0/bin:/opt/homebrew/bin:...</string>
</dict>

Since ~ expansion doesn't work inside a plist, HOME has to be written as an absolute path. If you're calling scripts that use Node.js via nvm, PATH also needs to include ~/.nvm/versions/node/vXX.X.X/bin. Forget it and node: command not found piles up silently in the StandardErrorPath log.

④ The job won't start if the StandardOutPath / StandardErrorPath directory doesn't exist

I once wrote the following into a plist without creating the directory first.

<key>StandardErrorPath</key>
<string>~/.claude/logs/self-repair.launchd.err.log</string>
<key>StandardOutPath</key>
<string>~/.claude/logs/self-repair.launchd.log</string>

If you launchctl bootstrap while ~/.claude/logs/ doesn't exist, the job loads successfully (it even shows up in launchctl list), yet the script never runs at the scheduled time. Since the error log can't be written, nothing is left behind — just the result "it isn't running." The fix is to run mkdir -p first, and to build the mkdir -p "$(dirname "$LOG")" pattern from the top of outreach-schedule-guard.sh into the setup script that runs before plist registration too.

⑤ plistlib.dump() converts binary plists to XML

In the write-back processing, I call plistlib.dump(d, f) with default arguments. The default is fmt=plistlib.FMT_XML, so even if the original plist was in binary format (converted with plutil -convert binary1), it becomes XML format after write-back.

with open(path,'wb') as f: plistlib.dump(d, f)
# ↑ デフォルトはXML形式で出力。バイナリ維持したい場合は:
with open(path,'wb') as f: plistlib.dump(d, f, fmt=plistlib.FMT_XML)
# または
with open(path,'wb') as f: plistlib.dump(d, f, fmt=plistlib.FMT_BINARY)

launchd reads both XML and binary formats without issue, so there's no actual harm, but the format changes when you inspect the contents with plutil -p. I once got confused comparing against a backup and thinking "did something rewrite this?" If you want to preserve the original format, you need to detect whether it's binary at read time and branch the output format accordingly. This time I settled on "as long as launchd can read it, standardize on XML."

⑥ launchd mistakenly reads .bak backup files as jobs

If there's a file with the .plist extension in the LaunchAgents directory, macOS's launchd management features recognize it as a job candidate. In an early version where I gave backup files a .plist extension, I named them com.lily.outreach-ig.bak.plist instead of com.lily.outreach-ig.plist.bak, which caused confusion when launchd GUI tools (LaunchControl and the like) displayed them as jobs. The current naming convention $label.plist.bak-guard-YYYYmmdd-HHMMSS deliberately ensures the name doesn't end in .plist.

⑦ Including bash in the forensic ps filter makes it match itself

grep -iE "python|node|bash|launchctl|plutil"

Because this pattern includes bash, the bash process handling the pipe matches the grep itself. A line like /bin/bash /path/to/outreach-schedule-guard.sh always appears in the log. That's noise, but I've deliberately left it in — the very fact that "the guard script's own start time is recorded in ps" can serve as evidence. That said, when looking at a ps snapshot, it's nearly certain that the "bash" line is the guard script itself.

⑧ The gap between the guard's run frequency and the DM send frequency

com.shun.self-repair.plist's StartCalendarInterval is three times: 9:20, 13:30, 19:30. Meanwhile, for outreach DMs, com.lily.outreach-ig runs eight times every two hours from 8:20 to 22:20. The math works out to: "if the guard last ran at 19:30 and a rewrite happens at 20:00, the next detection takes 13 hours, until 9:20 the next morning."

I'm aware of this design flaw. As countermeasures, I'm considering either raising the guard's run frequency (say, hourly) or building a self-consistency check into the outreach DM scripts themselves at startup. For now I've kept it to three runs on the premise that "rewrites don't recur frequently," trading off against the side effects of reading/writing plists. Since the August 23 incident did real damage in the form of "went unnoticed for several days," this judgment may be revisited in the future.

⑨ The stat -f '%Sm' format is macOS-specific

The stat -f '%Sm' -t '%F %T' used to get the mtime in the forensic log is macOS stat command syntax. On GNU Linux, stat -c '%y' is the equivalent, but this script is macOS-only so it's not a problem. That said, if a colleague on another Mac asks "does this script work on Linux too?", I need to tell them the stat part won't. If you wanted to make it cross-platform, one option is to unify on os.path.getmtime() on the Python side.

Best practices

Here are the decision criteria I picked up through building and operating this.

1. Override PATH at the top of launchd shell scripts

export PATH="/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin"

Nearly every case of "works in the terminal but gives command not found from launchd" comes down to PATH. Node under nvm and Homebrew's python3 don't exist in the system PATH. This is always line one of my launchd script template.

2. Drop set -e and keep only pipefail

For monitoring and restore scripts, -e is all harm and no benefit. On top of the reason explained in part 2 (the script dies on an empty grep with exit code 1), a launchctl bootout failure (when it's already unloaded) is also fatal under -e. Keeping just pipefail still lets you detect silent errors partway through a pipe.

3. Read and write plists with python3 plistlib

XML parsing with shell grep/awk has no type guarantees and can't handle binary plists at all. python3 can auto-detect both XML and binary formats with the standard library's plistlib, and reliably extract Hour and Minute as integers. Since python3 -c "import plistlib" is complete in one line, there are zero external dependencies.

4. Always quote heredocs in the <<'EOF' form

When passing Python code from Bash via a heredoc, an unquoted <<EOF breaks things because the shell tries to variable-expand the Python code. Python code containing {, }, or $ is especially dangerous. Making <<'PY' with single quotes a habit from the start prevents bugs that are very hard to debug.

5. Always create a timestamped backup before restoring

cp "$plist" "$plist.bak-guard-$(date +%Y%m%d-%H%M%S)"

If the restore process malfunctions (for example, a mistake in the SPECS array causes correct settings to be judged as a "mismatch" forever), you can't get back to the original settings without a backup. The reason for burning in the timestamp is that when multiple rewrites occur, you can follow the chronology with nothing but ls -lt.

6. Validate plist structure with plutil -lint before handing it to launchd

If you feed launchd a plist whose structure broke during write-back, the job silently stops starting.

if plutil -lint "$plist" >/dev/null 2>&1; then
  # launchctl bootstrap へ進む
else
  log "🔴 復元後のplistが壊れている（戻していない）"
fi

With this gate in place, you eliminate the risk of polluting launchd with half-written XML from a plistlib bug or an interrupted write.

7. Put sleep 1 between bootout and bootstrap

launchd's bootout completes asynchronously. Even when bootout returns 0, hitting bootstrap at a moment when launchd's internal job-termination processing hasn't finished returns Load failed: 5: Input/output error. One second of waiting is empirically a sufficient safety margin. Further, catching bootstrap's success/failure in an if statement and logging it means you'd notice immediately if it failed again.

8. Always leave a forensic log (mtime + ps snapshot)

An implementation that only restores and calls it a day buries "what happened" in the dark. Recording the plist's mtime at the moment a rewrite is detected, plus a list of running processes, gives you clues for finding the cause. The regret from the August 23 incident — "with a guard script I could have identified the suspect process" — is the motivation for this design. As long as you have the logs, patterns emerge when the same rewrite recurs.

9. Write a printf-based log function

log() { printf '%s %s\n' "$(date '+%F %T')" "$*" >> "$LOG"; }

echo's handling of newlines varies with the shell implementation and the presence of the -e flag. When a string containing backslashes slips into a log message, echo may or may not expand \n into a newline depending on the environment. printf's separation of format string and values guarantees consistency.

10. Add a guard for a missing plist

[ -f "$plist" ] || { log "$label: plist が無い"; continue; }

If a plist for a label defined in the SPECS array has disappeared, python3's open() raises an exception and the whole script stops. Making it an existence check + log + continue to the next loop means that even if one of the four jobs is missing its plist, the other three still get checked normally.

11. Use RunAtLoad: false to prevent immediate execution right after registration

launchd has a feature that starts a job exactly once the moment it's loaded with launchctl bootstrap (equivalent to RunAtLoad: true). The guard script only needs to run on schedule, so RunAtLoad: false turns off the unnecessary immediate run. The restore process is idempotent, but reading/writing plists has a non-zero cost.

12. Minimize the monitoring job's resource impact with LowPriorityIO and Nice: 10

com.shun.self-repair.plist has the following configured.

<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>10</integer>
<key>ProcessType</key>
<string>Background</string>

It defeats the purpose if the monitoring job eats CPU and IO and degrades the performance of the outreach DM scripts it's monitoring. Nice: 10 lowers CPU priority by 10 steps (0 is the default), and LowPriorityIO lowers IO priority. Combined with ProcessType: Background, it becomes subject to the OS's battery and CPU optimizations.

13. Guard the StartCalendarInterval single-entry problem with an isinstance check

rows = d.get('StartCalendarInterval') or []
if isinstance(rows, dict): rows = [rows]

Without this one line, the restore process runs every time against a plist with only one time set (a <dict> written directly without an <array> tag). Whether plistlib returns a dict or a list depends on the XML structure, so you need to check defensively.

14. Solve the leading-comma CSV problem with ${var:+,}

for h in "${harr[@]}"; do expected="${expected}${expected:+,}${h}:${minute}"; done

${expected:+,} is a Bash parameter expansion meaning "insert a comma if expected is non-empty, do nothing if empty." Writing expected="${expected},${h}:${minute}" puts a comma at the front, producing an infinite restore loop where expected and actual values never match.

15. Always register the guard itself with launchd via bootout → sleep 1 → bootstrap

When updating an existing job (when you change ProgramArguments or StartCalendarInterval), running bootstrap without launchctl unload or bootout doesn't apply the changes. Bundling the following procedure into a setup script means never having to second-guess it.

launchctl bootout "gui/$(id -u)/com.shun.self-repair" 2>/dev/null
sleep 1
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/com.shun.self-repair.plist
launchctl list | grep self-repair

In that final launchctl list check, a PID of - with status 0 means it registered successfully.

Wrapping up

This mechanism, which began with the schedule rewrite at 12:04 on August 23, 2026, ended up being a step toward "an environment where automation uptime doesn't need human eyes to guarantee it." Whether the four outreach DM lanes are running on the correct schedule — 29 runs a day in total — is something outreach-schedule-guard.sh checks three times a day at 9:20, 13:30, and 19:30, and automatically restores if anything looks off, without me having to check.

The core implementation is 73 lines. Read with plistlib, compare against expected values, and if they differ, back up, restore, and re-register with launchd. Leave a forensic log along the way. That's it. Even launchd's seemingly complex internals can be controlled from a shell script, as long as you have python3's standard library to read and write the XML directly.

The reason I sustain 1.2M yen in monthly revenue as a solo developer is that I've stacked up "mechanisms that keep running autonomously during the hours no human is watching." Each individual mechanism is simple, but because a stoppage translates directly into lost revenue, I design them to repair themselves when they stop. The guard script in this article is one example of that philosophy.

What's the longest one of your automations has been quietly broken before you noticed?

I've written up the full picture of the system, the breakdown of the 1.2M yen, and the 30-day procedure in a paid note.

📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Broken Retries: Why 'Just Try Again' Kept 5 Automation Lanes Dead for 63 Minutes

Lily — Sun, 30 Aug 2026 11:00:05 +0000

Five automation lanes went down on the same morning. Every log said the same thing: "Failed, so I ran it again." Then again. Then again. Sixty-three minutes after the first failure, all I had was the exact same failure — three more times over.

Why this design works

When I first started writing scripts, I thought retries looked like this:

for attempt in range(3):
    try:
        result = do_something()
        break
    except Exception as e:
        time.sleep(2)

And that's correct — as far as it goes. A network blip, a server that got temporarily slow: that kind of failure really does get fixed by this. The problem is that it applies the assumption "the next attempt will succeed" uniformly to every failure.

As the number of running jobs grows, failures that break this assumption will inevitably show up.

In the SNS automation environment I operate, retry fixes came in from five separate lanes at once over two days, August 8–9, 2026. The symptoms all looked alike. The causes were completely different.

note-autolike produced zero rewrites for three days. Last success: 2026-08-06T09:03:20Z. There was a 58-character limit validation on article titles, but the generation prompt said nothing at all about a character limit, so the model returned a 63-character title every single time. Three attempts, three 63-character titles. A fourth was never tried — it went straight into the give-up path.

ai-portraits spun its wheels for 63 minutes on image generation. When a request got rejected by moderation, I had implemented a three-stage fallback that progressively softened the prompt and retried. Up to 21 minutes per concept; up to 63 minutes for three concepts. In practice, when the target content itself is what trips moderation, lowering the tier doesn't move the wall. Trying all three stages produced the same result as stage one.

browser-slot.sh sees 18–34 launch attempts per hour against 3 global slots. A single run takes 14–62 minutes, so the slots are always full. The pre-fix implementation printed SKIP: global limit reached and exited 0 the instant acquire failed. The numbers left in the log: 9 runs in the 9 o'clock hour, 8 in the 11 o'clock hour, and 12 in the 12 o'clock hour never even started. One account hit skip on both of its runs that day, ending with 0 likes and 0 follows.

All three had "just try again" implemented. None of them were fixed by it.

Because the kinds of failure were different. A 63-character title: throw the same prompt at the same model and you get the same 63 characters back. That's a deterministic failure. A moderation block, when the content is the cause, doesn't change no matter how many stages you try unless the conditions change. A slot shortage can only be solved by waiting for someone to free a slot.

In a retry implementation with no such classification, all three kinds of failure get sucked into the same "loop 3 times, then exit" handling. The result: fixable failures and unfixable failures are counted identically and come out the same exit.

There's a failure mode in the opposite direction too — cases logged as "failed" that were actually transient. My watch-arb script recorded only extract_failed when it couldn't fetch a product page. Look at the dashboard the next morning and it reads as "there are 16 shops we can't fetch." In reality it was just HTTP 429 coming back, and a prober that hit the same shops slowly, one at a time, fetched them correctly. Lining up the similar incidents I confirmed over the same period: the same 429 was recorded under seven different names in the logs.

Lane	Name in the log	Actual root cause
watch-arb product page	`extract_failed`	HTTP 429
watch-arb bootstrap	`FATAL`	undici's 10-second ConnectTimeout
social-autolike ig-3	`coverage 0/11 → all sources failed → exit 1`	429 from IG private API
social-autolike ig-sug	`following map fetch failed → run halted`	The same 429
outreach-multi enrich	all records `error` → all dropped at qualify → 0 messages total	`web_profile_info` returning 429 constantly
outreach-multi igJson	`unparsable json`	Response was 512,558 chars but sliced at 400,000

If it's recorded as extract_failed, you read it as "this shop's structure is unusual." In fact the problem was "I hit it too fast," and hitting it slowly the next day works. Because the name was wrong, I kept fixing in the wrong direction.

Cleaning this up, correct retry design converges on answering three questions.

(1) Will waiting fix it?
(2) When do you cut it off?
(3) What do you change before the next attempt?

"Failed, so try again" is an implementation that omits all three. On the morning those five lanes were down together, none of the three had an answer written down.

An automation environment doesn't keep running because I write code every day. It works because the environment self-corrects and keeps running while I'm asleep. From that premise, the requirements for retry design change. The requirement isn't "the next attempt will succeed after a failure" — it's "decide the next action per kind of failure." Between an implementation that has this design and one that doesn't, whether I have to be personally involved in incident recovery changes fundamentally.

The overall flow

Here are the three questions in one diagram.

A failure occurred
      │
      ▼
┌─────────────────────────────────────┐
│ (1) Will waiting fix it?             │
│                                     │
│  Slot shortage → wait for one to free│
│  Network drop → back off and retry   │
│  429 → wait, preferring Retry-After  │
│  Expired login → waiting won't fix it│
│  Validation error → waiting won't fix│
│  Moderation → depends on stage (below)│
└─────────────────────────────────────┘
      │ waiting won't fix it
      ▼
┌─────────────────────────────────────┐
│ (2) When do you cut it off?          │
│                                     │
│  Attempt cap or >3h since intake → stop│
│  Still attempts left → go to (3)     │
└─────────────────────────────────────┘
      │ still attempts left
      ▼
┌─────────────────────────────────────┐
│ (3) What do you change?              │
│                                     │
│  Same input → same result if deterministic│
│  Attach the error → recoverable failures recover│
│  Tier downgrade (vertical) → soften and retry│
│  Swap the task (horizontal) → change the task itself│
└─────────────────────────────────────┘

The three sections aren't processed in order — the combination that applies changes with the kind of failure. For a network drop, (1) alone is enough. A deterministic validation error skips (1) and enters at (3). A moderation block involves all of it: skipping (1), the threshold in (2), and the way you change things in (3).

(1) Wait — the skeleton of the implementation

Here's the core I added in the browser-slot.sh fix.

BROWSER_SLOT_WAIT_SEC="${BROWSER_SLOT_WAIT_SEC:-600}"

wait_for_slot() {
  local waited=0
  while ! try_acquire_slot; do
    local jitter=$(( RANDOM % 31 + 15 ))   # 15〜45秒ランダム
    sleep "$jitter"
    waited=$(( waited + jitter ))
    if [[ $waited -ge $BROWSER_SLOT_WAIT_SEC ]]; then
      log "RESULT waited=${waited}s SKIP: slot timeout"
      return 1
    fi
  done
  return 0
}

Make the retry interval a random 15–45 seconds. With a fixed interval, every job waiting for a slot piles in at the same instant (thundering herd). Adding random jitter offsets jobs from each other so they spread out naturally. I made the change only after confirming by measurement that a fixed interval serialized all the jobs and made congestion worse.

Don't hold the guard for the whole wait. Hold the lock only for the instant of the acquire check, and always release it afterward. Get this wrong and every waiting job holds a lock against every other, and they all stall.

Leave waited=600s on the RESULT line in the log. "Gave up after waiting 600 seconds" and "discarded immediately" are different events, and if you can't distinguish them you can't evaluate anything after the fact.

BROWSER_SLOT_WAIT_SEC=0 matches the pre-fix immediate-skip behavior exactly. More than 30 launchd jobs share this script, so I made it possible to switch over gradually while preserving backward compatibility.

On the other hand, there are cases that look transient and are in fact fixed by an immediate retry. I had a job that failed three days running with Cloudflare's Authentication error [code: 10000], which looked like "OAuth expired → a human has to intervene." But when I actually ran five probes in both a shell and a minimal launchd environment, all of them exited 0. Re-reading the logs from the failure days, 0.5 seconds after an access token reissue, a different endpoint returned 200 with that same token. It was a transient 401 right after token reissue, and the real root cause was that there was not a single retry. Failure rate: 3 days out of 9 (33%). Snap-judging "it's an auth error, so it's a human task" piles machine-fixable things onto a human queue.

(2) Give up — always place a terminator

Take ai-portraits' tier ladder. Moderation block errors have a moderation_stage field. Ones that stopped at output have a track record of getting through with a tier downgrade. Ones that started at input failed all 9 attempts over the same period.

for tier in [1, 2, 3]:
    result = generate(concept, tier=tier)

    if result.get("moderation_stage") == "input":
        # input段 → Tierを変えても壁は動かない、残りをbreak
        log(f"input-stage block, skip remaining tiers for this concept")
        break

    if result.get("moderation_blocked"):
        # output段 → Tier降格で通る可能性がある
        continue

    if result.get("success"):
        return result

Decide from the first-stage error body whether climbing down the ladder is worth anything. Just breaking on moderation_stage == "input" turns 63 minutes of spinning into 0. In the same spirit, gsheets_retry.py is designed to raise immediately on 401/403/404.

RETRYABLE_HTTP = {429, 500, 502, 503, 504}

def sheets_call_with_retry(fn, max_attempts=3):
    delay = 2
    for attempt in range(max_attempts):
        try:
            return fn()
        except HttpError as e:
            if e.resp.status not in RETRYABLE_HTTP:
                raise  # 401/403/404 は即 raise — リトライで隠さない
            if attempt == max_attempts - 1:
                raise
            time.sleep(delay)
            delay *= 3  # 2秒 → 6秒（合計最大8秒）
        except (socket.timeout, ConnectionError, TimeoutError):
            if attempt == max_attempts - 1:
                raise
            time.sleep(delay)
            delay *= 3

Decide "never exceed 8 seconds total" first. If you set exponential backoff by attempt count, it quietly eats the caller's slot time. Fixing an upper bound in seconds first and back-calculating the attempt count is the safe order.

If you implement "give up" using attempt count alone, you can't handle failures on the time axis. In kotonoha's job queue I had an incident where a job sat pending for 39 hours. Even when it broke off at the cap of 6 attempts, the requeue succeeded, so a failure was never raised, monitoring never fired, and the user's screen stayed on "generating."

The post-fix predicate looks like this.

function shouldGiveUpForQuota({ quotaHits, createdAtMs, resumeAtMs }) {
  const THREE_HOURS_MS = 3 * 60 * 60 * 1000;
  return (
    quotaHits >= 2 ||                                  // 2回目の上限
    resumeAtMs > createdAtMs + THREE_HOURS_MS          // 受付から3時間超
  );
}

Place the cutoff on both "attempt count" and "absolute time since intake." Putting the actual values from the incident day (intake 2026-08-19T00:36:57Z, resume 06:37:00Z) into the tests as a boundary case gives you a machine guarantee that the same incident can never get through again.

(3) Feed back — pass what you changed into the next attempt

Here's the note-autolike case. The reason rewrites were zero for three days was an implementation that just threw the identical prompt three times. Make the error machine-readable and attach correction instructions to attempts from the second onward.

// エラーに code / actual / limit を載せる
throw new ValidationError(`タイトルは1〜58字にしてください（実際: ${actual}字）`, {
  code: "title-too-long",
  actual,
  limit: 58,
});

// 2回目以降は修正指示を付けて投げる
function rewriteRetryInstruction(error) {
  if (error.code === "title-too-long") {
    return [
      `前回の出力は次の理由で不採用でした：`,
      `タイトルが${error.actual}字あり、上限の${error.limit}字を超えています。`,
      `同じ内容・同じ構成のまま、その点だけを満たす形へ直して、生JSONのみを再出力してください。`,
      `タイトルは絵文字込みで${error.limit}字以内（前回は${error.actual}字）に必ず収めてください。`,
    ].join("\n");
  }
}

async function askClaudeWithRetry(prompt, retries = 3) {
  let currentPrompt = prompt;
  for (let i = 0; i < retries; i++) {
    try {
      const result = await callClaude(currentPrompt);
      validateRewrite(result);
      return result;
    } catch (e) {
      if (e instanceof ValidationError) {
        // 決定論的な失敗 — スリープなし、修正指示を付けて即再試行
        currentPrompt = prompt + "\n\n" + rewriteRetryInstruction(e);
      } else {
        // プロセス系の失敗 — 30秒待つ
        await sleep(30_000);
      }
    }
  }
}

Retry deterministic failures immediately, with no sleep. Waiting 2 seconds on a validation error just throws 2 seconds away. If you mix "wait" and "fix and resubmit" into the same retry, you pay wait time even on fixable failures. Branch on the kind of failure and choose where the sleep goes.

The correction instructions you return to the LLM should be "human-language instructions plus the actual strings," not "machine identifiers." Passing "title-too-long" through as-is loses three times in a row. Passing "the title is 63 characters, exceeding the 58-character limit; keep it within 58 characters" recovers. This is a fact I had already demonstrated in a different script.

I'll stop the full picture of the three sections here. Next I'll break down how each section breaks and how to fix it, case by case, drawn from the five-lane simultaneous outage.

Implementation details

"I wrote the retry" isn't done — put fault injection in the same commit

The three sections designed above are, at the moment you write them, nothing but "code that should work." Intermittent failures have no reproduction method, so by default there's no way to confirm whether the retry actually traverses that path. My first four implementations ended there. I looked at git diff, felt like I'd confirmed it, and called it done.

What actually worked was bundling fault injection into the same commit as the implementation itself.

// watch-arb/collector/lib/api.mjs
let _failOnceRemaining = parseInt(process.env.WA_API_FAIL_ONCE ?? '0', 10);

async function req(path, opts = {}) {
  if (_failOnceRemaining > 0) {
    _failOnceRemaining--;
    throw new TypeError('fetch failed');  // 合成の障害、本番では絶対に発火しない
  }
  // ... 本来のfetch処理
}

The counter is module-scoped and shared across all req calls. Only for the first N calls after process start, it throws a synthetic TypeError before calling fetch.

The completion criteria are a set of two, and I always confirm both.

① Launch with WA_API_FAIL_ONCE=2 and see api retry attempt=1/3 and 2/3, then a full run to exit 0 with no FATAL
② With the env var unset, not a single retry log line appears

Having ② means you can demonstrate "the fault injection hasn't leaked into production" with the same command. Here's the log I actually got.

[INFO] api retry attempt=1/3 path=/api/collector/shops reason=fetch failed
[INFO] api retry attempt=2/3 path=/api/collector/shops reason=fetch failed
[INFO] 32件を ingest して exit 0

I rolled this pattern out from watch-arb to zaiko-radar, lily-line-funnel, and social-autolike — four in total. The rule changed to: "I wrote the retry" doesn't count as done unless it's "I watched it run with fault injection."

Confine the retry target to a single function

When you write a retry, the first thing to decide isn't "how many attempts" but "what gets retried."

// 判定を1関数に集約する
function isTransientStatus(status) {
  return status === 429 || (status >= 500 && status <= 504);
}

async function req(path, opts = {}) {
  for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
    try {
      const res = await fetchWithTimeout(path, opts);
      if (!res.ok) {
        if (!isTransientStatus(res.status)) throw new HttpError(res.status);
        if (attempt === MAX_ATTEMPTS) throw new HttpError(res.status);
        await sleep(backoff(attempt));
        continue;
      }
      return await res.json();
    } catch (e) {
      if (!isNetworkError(e)) throw e;  // ネットワーク例外以外は即throw
      if (attempt === MAX_ATTEMPTS) throw e;
      await sleep(backoff(attempt));
    }
  }
}

Throwing immediately on 401/403/404 isn't a shortcut — it's design that prevents a different kind of incident. Wrap a permission error in a retry and you throw the same request three times and get the same failure three times. The log records only "failed 3 times," and the real cause — "no permission" — gets buried.

This is the inverse of the "429 recorded under seven names" I documented in transient-failure-recorded-as-permanent.md. Same structure as a 429 masquerading as extract_failed: 401/403 masquerades as "sometimes slow."

The reason to pull isTransientStatus out into a single function is that when the judgment is scattered across multiple call sites, each site grows a subtly different definition. The constant set RETRYABLE_HTTP = {429, 500, 502, 503, 504} in gsheets_retry.py comes from the same idea: it confines the knowledge of "which statuses are retryable" to one place.

Adding retries changes how exceptions propagate

In the commit right after I implemented withRetry in kotonoha, three side-effect fixes went in. I thought I had "added a feature"; what I had actually done was "change how exceptions propagate."

Before the change, purge() continued on to the subsequent "clean local work directories older than 30 days" step even when a 5xx came back, because the implementation moved to the next line even when res.ok was false. Once withRetry was in and it threw the instant a 5xx appeared, the cleanup code became unreachable. The real damage was a quiet degradation of the kind you only notice if you manually clean periodically: "work directories accumulate 30 days' worth."

async function purge(jobId) {
  try {
    await uploadResult(jobId);
  } catch (e) {
    log(`purge upload failed: ${e.message}`);
    // ここで throw しない — 置き場の失敗は記録して続行する
  }
  await cleanWorkDir();  // 例外の有無にかかわらず必ず走らせる
}

The other one was setContentSafely, where the gap between the first and second retry was 0 seconds. Retrying immediately under the same conditions right after a failure caused by resource pressure gives the same result as long as the pressure remains. Earlier I wrote "retry deterministic failures immediately with no sleep" — this is the opposite side of that. For failures caused by resource pressure, a retry is meaningless unless you wait.

Attempt counts are set asymmetrically according to "what happens when it fails." report() (result reporting) leaves the user's screen stuck on "running" forever if it fails, so attempts=5; claim() (reservation acquisition) gets attempts=3. A uniform attempt count ignores the asymmetry of cost and damage. Timeouts are also split by the weight of the operation: putObject and getStream get 120 seconds, getText and listKeys get 30 seconds. A fetch with no timeout configured creates the worst un-retryable state of all: hanging forever.

You can't retry when you pass a stream

There's a constraint I wasn't conscious of until I implemented this. To make something retryable, the input has to be re-executable.

// NG: ストリームは一度消費したら再生できない
async function retryWithStream(stream) {
  for (let i = 0; i < 3; i++) {
    await fetch(url, { body: stream });  // 2回目はストリームが空
  }
}

// OK: Buffer か string に限定する
async function retryWithBuffer(buffer) {
  for (let i = 0; i < 3; i++) {
    await fetch(url, { body: buffer });  // 何度でも同じデータを送れる
  }
}

I restricted the body passed to putObject to Buffer or string. If you don't leave the type constraint in a comment, a caller later passes a stream and you get a bug where "the retry is working but it fails every time." Pass a stream into a retry and every attempt after the first sends an empty body. No error surfaces; it just fails every time.

Where I got stuck

A constant I placed for protection had become a 0%-success-rate wall

In outreach-multi's DM inbox parsing, an unparsable json error appeared every day. I suspected a broken selector or a changed JSON structure and kept fixing for three days.

The reality: the inbox-fetching code had a line reading text.slice(0, 400_000). It was an upper bound written to protect memory when a huge response came in. When I measured the actual size of the IG DM inbox, it was 512,558 characters. Since the limit is 400,000, it gets cut off every single time. Truncated JSON fails JSON.parse every single time. Success rate: 0%.

The reason I stared at the same failure for three days is that I had forgotten about the text.slice line and read it as "parsing is failing → the structure broke." Nor had the symptom started "today." It quietly started failing on the day the inbox size crossed 400,000 characters, and one day I noticed the running total was 0.

The fix was to complete JSON.parse inside page.evaluate and return only the first 300 characters for diagnostics when it fails.

The design rule I took from this: "if you place a limit, log the fact that the limit was hit." text.slice silently truncates and tells no one. Even when a protective constant has turned into a permanent failure the moment its value was exceeded, nothing is left in the log.

The quarantine had become an infinite loop

The unfollow processing in unfollow-core.js had a mechanism where three failures put an account into a 7-day hold (quarantine).

The problem is "what happens when it comes out of quarantine." A revived account goes back into the candidate list. Fail three more times, and it goes into another 7-day hold. This spins forever.

There were 2,605 candidates, and I puzzled over the numbers scanned: 8 / unfollowed: 2 for two weeks. Almost none of the candidates were being processed. When I investigated "why isn't this progressing," the answer was "the same accounts keep entering and leaving quarantine, and nothing moves forward."

The log said circuit-break. It's contradictory for circuit-break to be functioning while the execution count doesn't rise. I could have noticed much sooner, but I was reading it as "circuit-break is working = it's protected," so I never questioned it.

The fix was to physically delete from the ledger any account that reached a cumulative failure count (UNFOLLOW_GIVEUP_FAILS = 6, twice the quarantine failure threshold).

A "hold" only accumulates; it never drains. Unless you place a terminator that discards, the ledger keeps filling up with accounts that can't be processed. That said, before discarding, you first have to confirm "is my own processing working correctly?" There's an inverse incident where the session had merely expired and you discard all the healthy accounts. The principle I wrote on the kotonoha page — "check your own health before discarding" — applies here too.

Monitoring didn't fire for 39 hours

A kotonoha job sat stuck on "generating" for 39 hours. Monitoring never fired once. The job was never recorded as a failure.

The cause: "a retry that doesn't hold the attempt count in state can't have a cutoff placed on it."

When a job hits a quota limit, requeue puts it back to pending. The returned job carried no counter for "which attempt is this." A 7th-attempt job and a 1st-attempt job are indistinguishable from the implementation's point of view. Since requeue succeeds, no failure is raised. It never hits a monitoring threshold. The user's screen keeps displaying "generating" indefinitely.

The fix came in two stages. First, "hold the attempt count in state." Then, "place the cutoff on both attempt count and absolute time since intake." When I put the shouldGiveUpForQuota function shown earlier through tests with boundary cases, I included the actual values from the incident day (intake 2026-08-19T00:36:57Z, scheduled resume 06:37:00Z) as one case. That gives a machine guarantee that the same incident can never get through again.

With a cutoff on attempt count alone, you let through "only the 1st attempt, but resume is 12 hours away." With time alone, you let through "resume is soon, but the limit hits back to back." This incident got caught on the first attempt, on the time-condition side. It's a case where the value of doubling up was measurable in real data.

"The worker waits" and "give up on that job" are separate decisions. A quota is a failure that waiting fixes, so the worker waits. But that isn't grounds for making a user wait more than three hours.

I snap-judged "auth error" and left it in a human queue for three days

com.lily.line-pdca stopped three days running with Cloudflare's Authentication error [code: 10000]. Since it was an auth error, I judged "OAuth expired, a manual wrangler login is needed." I also suspected a missing environment variable.

Classifying it as "a case requiring human GUI operation" before verifying was the mistake.

When I actually ran wrangler whoami, kv key list, and deploy --dry-run in both a shell and a minimal launchd environment, all exited 0. Even the output byte counts were identical (1,516 bytes). Re-reading the logs from the failure days, KV keys returned 401 0.5 seconds after an access token was reissued. With that same token, /user 0.5 seconds later returned 200. It was a transient 401 right after token reissue, and the real root cause was that there was not a single retry.

The failure rate was 3 days out of 9 (about 33%). It doesn't happen every day, but it does happen once every three days. That frequency was itself part of why it "looked like a human task."

The criterion that changed my judgment was actually hitting the endpoint to confirm "would the same procedure pass if I ran it again right now?" Don't classify something as "unfixable, period" based only on the kind of error (401, Authentication error). Same pattern as the "429 recorded under seven names" documented in transient-failure-recorded-as-permanent.md — this time the name "auth error" made a transient failure look like a human task.

Since this incident, the rule changed to: when an auth error appears, run one probe before deciding. The premise behind "don't pile machine-fixable things onto a human work queue" is "I hit it with a machine and checked." Snap-judging without checking is no different from declaring "it won't be fixed" without ever trying.

Pitfalls

Here are the structural patterns behind the individual cases above. Only the "name of the place I got stuck" differs; the root is the same pattern.

Compressing a failure into "a single name" when recording it

watch-arb's product pages were recorded as extract_failed, the bootstrap run as FATAL, and the DM inbox as unparsable json. All three had HTTP 429 as the actual root cause. Organizing this phenomenon, recorded in transient-failure-recorded-as-permanent.md, the same 429 was left under 7 different names within the same period. When the HTTP status disappears at the naming step, you can no longer judge whether it's "waiting will fix it" or "the structure needs fixing." The fix is one line. Before recording, classify: 429/503 → rate_limited / 401/403 → auth_error / everything else → extract_failed. This three-way branch is the precondition sitting in front of the wait/give-up/feed-back branch.

Throwing away the error body

Because ai-portraits' generate.py discarded codex exec's stdout/stderr, the reason no image was produced was recorded only as missing. If you can see neither moderation_blocked nor safety_violations: ["sexual"], you can't judge whether a tier downgrade will recover it or whether you should cut it off immediately at the input stage. In an implementation that doesn't retain errors, the three sections of "wait / give up / feed back" can't exist in the first place, so error retention is a design precondition, not a feature addition.

Trying every tier without checking moderation_stage

Concepts that stopped with moderation_stage: input were tried through all stages from T1 to T3 and lost all 9 attempts. Ones that stopped with moderation_stage: output have a track record of passing with a tier downgrade (partial 01_feet_up_window at T2, 03_legs_on_bed_topdown at T3). A single word in T1's moderation_stage field determines whether climbing down the ladder is worth anything, yet trying every stage without reading the field wastes up to 21 minutes per concept — up to 63 minutes for three. Judge "is it worth climbing down based on the first-stage error body" before "how many safe-side fallback stages are there."

Wiring a fallback without measuring the destination's capacity

Before adding a fallback in social-autolike's follow-source-followers, I first measured that the UI modal's following side grows to 408 entries in 63 seconds while the followers side stops at 12. That's why I adopted only the side with capacity as the fallback. Wire it without measuring and all the traffic flows to a destination with no capacity, producing "I added a fallback but the success rate didn't change." outreach-multi's pk resolution likewise builds in the known wall of IG cutting off around 70 entries, limiting to 30 per run.

Counting a permanent failure against the "one more try" budget

In outreach-multi's DM sending, when no メッセージ button (DM-disabled setting) came back, it rode the exhausted after 3 attempts path and consumed 3 attempts' worth of send budget per recipient. The other party's account settings don't change no matter how much we retry. Read the reason field in sent.jsonl, put recipients containing no メッセージ button / profile gone / このページは存在しません into a permanentlyFailed Set, and place the shouldSkip() check ahead of exhausted. This shares a root with the "inverse incident of recording a transient failure as permanent" mentioned earlier: the design of failure classification is needed before the design of retry counts.

Continuing to hit at the same rate after a 429

social-autolike's ig-3 fired 17 sources within a few seconds at 0.3–0.5 second intervals and produced zero results over 14 hours. If you don't change the interval after receiving a 429, you pile up the same requests during the limit and extend your own throttling. The fix recorded in retry-and-giveup-design.md includes a latch that cuts off that process's API calls after 3 consecutive 429s, plus a design that drops concurrency to 1 for any shop that has seen a 429 even once. Slowing down the rate itself comes as a set with waiting before retrying — not instead of it.

Retries stacked in two layers

watch-arb's ingestShop had its own 3-attempt loop, and req() had a 3-attempt loop as well. Worst case 3×3 = 9 attempts, with "retry attempt 3/3" appearing twice in the log, making it unreadable where the limit is and which attempt you're on. Even with fault injection in place, you can no longer tell "which layer is running." The fix was to delete ingestShop's own loop and consolidate onto the single layer in req(). The caller just throws immediately.

Counting fallback successes toward the cutoff counter

The latch that "cuts off API calls after 3 consecutive 429s" was also counting cases where the UI fallback succeeded. Even with an escape route, it stopped at 3 items. The cutoff counter should increment only "when every path failed." Cases that succeeded via the fallback shouldn't be logged against the main failure count.

Judging "couldn't fetch (abnormal)" and "had no time to fetch (a normal cutoff)" by the same zero count

When follow-source-followers uses up its time budget on source fetching alone and ends with zero candidates, it falls into the existing "all sources failed → halt to prevent mis-operation" path. "Couldn't fetch" and "just ran out of time" are separate events. Judge them by the same zero count and you get false alarms every day and lose the ability to tell real failures apart. Prepare a separate exit called time-budget to distinguish them.

Hardcoding the give-up predicate inside the catch block

When the cutoff conditions are written inside a catch, you can't put them into boundary-case tests. Pull them into a pure function and you can add one case with the actual values from the incident day, giving a machine guarantee that the same condition never gets through again. As long as the logic lives inside the catch, the test passes in a state where "the case just happened not to be exercised."

Error messages carrying no machine-readable information

Throw a ValidationError with only a string and you can't extract "what was wrong" from code at retry time. The reason throwing the same prompt from the second attempt onward gives the same result is that "feed back" isn't implemented; put on a structure like code: "title-too-long", actual: 63, limit: 58 and you can assemble correction instructions dynamically. When returning to the LLM, pass a human-language instruction and the measured values, not the identifier ("title-too-long"). Passing the identifier through as-is loses three times in a row.

Best practices

Here are the rules distilled from the five-lane simultaneous outage. There are a lot of them, but every one is something I added because I actually got stuck.

① Classify failures by HTTP status and exception type before recording them

429/503 → rate_limited / 401/403 → auth_error / everything else → processing error. Just placing this three-way branch first changes the meaning of the "list of failing shops" shown on the dashboard. After adding this classification, watch-arb also split the wording of shops.last_error into "couldn't fetch due to rate limiting" and "can't extract the structure."

② When you write a retry, put fault injection in the same commit

let _failOnceRemaining = parseInt(process.env.WA_API_FAIL_ONCE ?? '0', 10);
// 起動後の最初のN回だけ合成のエラーを投げる

The completion criteria are a set of two. With =2, retry attempt=1/3 and 2/3 appear and the run completes at exit 0; with it unset, not a single retry log line appears. Without ②, you can't confirm with the same command that "the fault injection hasn't leaked into production."

③ Place the cutoff on both "attempt count" and "absolute time since intake"

Attempt count alone lets through "only the 1st attempt, but resume is 12 hours away." Absolute time alone lets through "resume is soon, but the limit hits back to back." The kotonoha incident got caught on the first attempt, on the time-condition side (intake 2026-08-19T00:36:57Z, resume 06:37:00Z, outside intake+3 hours). A case where the value of doubling up was measurable in real data.

④ Decide the total wait ceiling in seconds first, then back-calculate the attempt count

For gsheets_retry.py, I fixed "never exceed 8 seconds total" in the request text first, then back-calculated 2s → 6s (3 attempts). If you fix exponential backoff by attempt count first, the caller's slot time gets quietly eroded.

⑤ Throw immediately on 401/403/404 — don't hide them in a retry

if e.resp.status not in RETRYABLE_HTTP:
    raise  # 権限エラーは即 raise

Wrap a permission error in a retry and only "failed 3 times" remains, burying the real cause of "no permission." Same structure as the "429 masquerading as extract_failed" recorded in transient-failure-recorded-as-permanent.md — this time a 401 masquerades as "sometimes slow."

⑥ Use moderation_stage to judge whether climbing down the ladder is worth it before trying tiers

if result.get("moderation_stage") == "input":
    log("input段の拒否。残りTierをbreak")
    break

Adding one line turns 63 minutes of spinning into 0. Ones that stopped at output are worth running the ladder for. Ones that stopped at input get discarded concept and all, and you move on.

⑦ Consolidate retry layers into one place

If the caller has its own loop and the library also retries, you get up to N×M attempts. When you find duplication, delete the outer loop and consolidate onto the single layer in req().

⑧ Measure the destination's capacity before wiring it up

Before adding a fallback, confirm by measurement "how many entries this destination returns" and "where it jams." Routing all traffic to a destination with no capacity won't change the success rate.

⑨ Record permanent failures in a persistent list once and exclude them

Failures rooted in the other party's settings, like no メッセージ button / profile gone, should be excluded before they consume retry attempts. Place the shouldSkip() check ahead of exhausted.

⑩ Actually hit it to check "would it pass if I ran it again right now?" before classifying it as a human task

I snap-judged Cloudflare's Authentication error as "OAuth expired" and left it in a human queue for three days. Running probes in both a shell and a minimal launchd environment, all exited 0. Don't classify something as "unfixable, period" from the name of the error alone. Check, then judge.

⑪ Log the fact that a limit constant was hit

text.slice(0, 400_000) just silently truncated. Even when a protective constant becomes the trigger for a permanent failure, nothing is left in the log. Complete JSON.parse internally and return the first 300 characters for diagnostics only when it fails, and the fact that "it got cut off" survives in the record.

⑫ After introducing retries, check how the range of exception propagation changed

Adding withRetry can make required downstream processing unreachable where the behavior changed from "swallow" to "throw." A retry isn't a feature addition; it's a change to how exceptions propagate. Where there's something downstream that "must happen even on failure," wrap it individually.

⑬ For failures caused by resource pressure, wait before retrying

With setContentSafely's retry interval at zero seconds, it fails again under the same conditions right after failing under high load. Immediate retry with no sleep is correct for deterministic failures, but on the flip side, a retry is meaningless unless you wait for resources. Choose where the sleep goes based on the kind of failure.

⑭ Pull the give-up predicate into a pure function and put the incident day's actual values into the tests

Extract it into shouldGiveUpForQuota({quotaHits, createdAtMs, resumeAtMs}) and add intake 2026-08-19T00:36:57Z / resume 06:37:00Z as one case, and the same incident can never get through again. A predicate hardcoded inside a catch block can't have its boundary cases tested.

⑮ Leave the fact that you waited, gave up, or cut off on a single log line

waited=600s SKIP: slot timeout and SKIP: global limit reached (immediate discard) are different events. If both look like the same "SKIP," you can't distinguish afterward whether "slot contention increased" or "it was discarding immediately all along." Emit the wait time you invested as a number in the log.

Summary

When you write "failed, so try again," that code has no answers to three questions. Will waiting fix it? When do you cut it off? What do you change before the next attempt? An implementation missing these three burns resources in front of a deterministic failure and stalls.

What I noticed when I lined them up on the morning five lanes went down together: whether I have to be personally involved in incident recovery is almost entirely determined by whether this design is present. With it, the script decides its own next action per kind of failure. Without it, the same failure just gets recorded three times, and the next morning it's stuck in the same place.

An automation environment runs every day not because I write code every day. It works because failures are classified correctly and the machine recovers on its own. From that premise, retry design isn't "a feature you add later" — it belongs in the first implementation.

"I wrote the retry" counts as done when it's "I watched it run with fault injection."

What's the failure in your own automation that's been recorded under the wrong name the longest?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

How My Own Script Killed Instagram Sessions Every Night: pkill and a Shared Chrome Profile

Lily — Sun, 30 Aug 2026 05:00:06 +0000

I built a fleet of 171 automation jobs to grow my business — and one of them was quietly murdering another one's login session every single night for days before I figured out why.

Here's the arc: I made ¥100k/month as a university student, stacked side jobs up to ¥600k/month, got laid off and went back to zero, then spent six months building an autonomous Claude Code environment that now does ¥1.2M/month in revenue. The Instagram account underpinning that revenue was being killed daily by a script I wrote myself.

Why This Setup Matters

When you're building up revenue as a solo developer, the first wall you hit isn't "not enough hands" — it's "I have no idea what's happening while I sleep."

In my current environment, 171 jobs launch automatically from launchd every day. Posting, liking, following, unfollowing, and DMs across X / Instagram / Threads / TikTok are all automated, and a Bash script called sns-output-watchdog.sh monitors whether each one actually produced output that day, based on artifact logs.

But before I built that monitoring, I have a track record of not noticing for two weeks. The header comment of sns-output-watchdog.sh says this:

# content-watchdog.sh は article/note/maker/series/ameba だけを見ており、
# X/IG/TikTok/Threads の投稿・返信は誰も監視していなかった。その結果
# IG投稿は7/29から、TikTok投稿は7/28から止まったまま2週間気づかれなかった。

If IG posting stops, new followers stop coming in, the flow into my official LINE account stops, and revenue growth eventually flattens. A single failed slot doesn't mean "I missed one post today" — it means "the entire funnel that was supposed to start there was dead."

Even after I set up monitoring, the next problem showed up: exit 0 and "something actually happened" are different things. In measurements on 2026-08-08, the IG like lane ig-1 finished with exit 0 and zero likes on 8 out of 12 daily runs. The watchdog at the time only counted "how many times the marker 終了 ( appeared," so it judged both circuit-break and need-login as "healthy."

# 🔴 従来は '終了 (' の出現回数だけを数えていたため、circuit-break でも need-login でも
#    「健全」と判定していた。2026-08-08 実測で ig-1 は12run中8回、tt-1/tt-2 は6割が
#    いいね0件のまま exit 0 で終わっており、2週間誰も気づけなかった。

Learning from that, I added count_dead_runs() to detect runs with zero output, and sum_likes_today() to total the actual like count. But that still didn't solve the structural problem: the side doing the breaking and the side reporting the breakage lived in different repositories. That's the 2026-08-24 incident. If you run multiple automations on the same Mac, you're at risk of falling into exactly the same hole.

The Overall Picture

A Shared Resource Across Repositories

To lay out the problem, here's the structure as a diagram.

scent-media リポジトリ                social-autolike リポジトリ
──────────────────────────            ──────────────────────────────────
scripts/ensure_chrome.sh              config/accounts.json
  └ CDP :9223 へ接続試行                └ "reuseProfile": ".profiles/chrome-ig"
    応答なし → pkill 実行
         │                            th-scent ジョブ（Playwright）
         │                              └ .profiles/chrome-ig を掴んで起動
         │                                ※ CDP 9223 は一切開かない
         ↓                                         ↓
         └──────────────┬───────────────────────────┘
                        ↓
             .profiles/chrome-ig  ← 両者が同じプロファイルを参照
             Cookie SQLite 強制破壊
             instagram.com の sessionid → 0行

ensure_chrome.sh in scent-media checks, before an IG post, whether Chrome is in a CDP-controllable state. The check is an attempted connection to port 9223. If there's no response, it decides "Chrome is dead" and force-kills it with this command:

pkill -f "user-data-dir=$PROFILE"

$PROFILE is .profiles/chrome-ig. The problem is that the th-scent job in social-autolike shared that same profile via a reuseProfile setting. th-scent launches Chrome through Playwright, but it never opens CDP port 9223. From ensure_chrome.sh's point of view, "the Chrome that th-scent is using" always looks like "9223 is closed = a dead process."

A Collision at the Same Time Every Day

This collision wasn't random — the launchd schedule made it happen deterministically at fixed times every day.

Time	Job	Duration
5:18 / 13:18 / 21:18	th-scent autolike	up to 40 min
6:56 / 14:56 / 22:56	th-scent unfollow	up to 40 min
7:20	scent-media daily-generate	—
19:00 / 21:00 / 23:00	scent-media daily-post (IG carousel post)	—

At 22:56, th-scent unfollow grabs .profiles/chrome-ig; at 23:00, daily-post calls ensure_chrome.sh. 9223 doesn't respond, so pkill runs. This repeated every night. The 19:00 and 21:00 slots overlap with th-scent autolike in the same way.

Destroying the Cookie SQLite

pkill -f "user-data-dir=..." sends SIGTERM and then SIGKILL. Chrome can't complete its shutdown routine and loses the chance to write the profile's Cookie SQLite back in a consistent state.

On next launch, Chrome recreates an empty Cookie DB. Here are the measured values:

The cookies table shrank to 3 rows total across all hosts, and instagram.com had 0 cookies. The IG sessionid was gone.

On 2026-08-24, all three slots — 19:00, 21:00, and 23:00 — posted nothing.

Why the Watchdog Misdiagnosed It as "Logged Out"

The IG post check in sns-output-watchdog.sh looks like this:

check "ig-autopost" \
  "$(count_today "$CL_LOGS/sns-ig-autopost.retry.log" "$TODAY_JST" 'OK posted')" \
  1 "IGカルーセル投稿" \
  "$CL_LOGS/sns-ig-autopost.retry.log" "$TODAY_JST"

If OK posted doesn't appear even once in the day's log, the lane goes into FAILED and 🚨 SNS当日未出力: ig-autopost fires to Discord. That alert was firing. But the text of the alert says "posting failed," not "session destroyed by pkill."

The session checker correctly reports "there is no IG sessionid." need-login shows up in the log. A human reads that and concludes "the login expired → let's log in again." The next day, at the same time, pkill runs again and the cookies vanish. This loop kept repeating.

watchdog: 🚨 ig-autopost 未出力
    ↓
session-liveness: sessionid が無い（正しい報告）
    ↓
人間: GUI 再ログインを実行
    ↓
翌日 22:56: th-scent がプロファイルを掴む
    ↓
翌日 23:00: ensure_chrome.sh → pkill → Cookie 消滅
    ↓
watchdog: 🚨 ig-autopost 未出力（同じ報告が出る）

Because the side doing the breaking (ensure_chrome.sh / scent-media) and the side reporting the breakage (retry.log / the session checker) live in different repositories, reading only one of them will never connect the dots. The reuseProfile setting written in social-autolike's config/accounts.json appears nowhere in scent-media's code.

The Skeleton of the Fix

The fix comes down to one thing: eliminate pkill entirely and replace it with waiting and skipping.

# 修正後の ensure_chrome.sh（概要）
if pgrep -f "user-data-dir=$PROFILE" > /dev/null; then
  # 他プロセスが掴んでいる → 10秒間隔・最大420秒ポーリング
  for i in $(seq 1 42); do
    sleep 10
    pgrep -f "user-data-dir=$PROFILE" > /dev/null || break
  done
  if pgrep -f "user-data-dir=$PROFILE" > /dev/null; then
    echo "他ジョブが使用中(pid=$(pgrep -f "user-data-dir=$PROFILE"))。Cookie破壊を避けるため起動を見送る" >&2
    exit 2   # 「今スロット見送り」の専用コード
  fi
fi
# ここまで来たら誰も掴んでいない → SingletonLock等の掃除 → 起動処理へ

The caller, daily_post.sh, receives exit 2 in a separate branch and treats it as "skip this slot, retry at the next one," exiting with exit 0. For verification I ran two bash -n syntax checks, confirmed grep -c pkill returned 0, and confirmed the exit 2 path really exists at line 57, then bundled it into commit 15307d7.

Why I didn't fold exit 2 into "error," what the "3 slots per day" premise behind the skip design means, and how I reworked the watchdog so it won't have the same structural blind spot again — I break all of that down in the next part.

Implementation Details

Why You Must Never Mix `count_today()` and `has_today()`

The first thing you agonize over when writing a watchdog is distinguishing "zero count = failure" from "it just didn't run." In sns-output-watchdog.sh I handle this with two independent functions.

# 当日分のログ行だけに成功マーカーがあるか数える。
count_today() {
  local file="$1" daymark="$2" marker="$3"
  [ -f "$file" ] || { echo 0; return; }
  /usr/bin/awk -v day="$daymark" -v mark="$marker" '
    index($0, day) { seen = 1 }
    seen && index($0, mark) { n++ }
    END { print n + 0 }
  ' "$file"
}

# そのログに「当日を示す行」自体があるか。
has_today() {
  local file="$1" daymark="$2"
  [ -f "$file" ] || return 1
  /usr/bin/grep -qF "$daymark" "$file"
}

count_today() uses awk's seen flag so that "only lines after the day marker appears" are considered. A simple grep -c marker would mix in success lines from previous days. The logs are designed not to rotate, so if this one-day offset breaks, you get "today judged healthy based on yesterday's post count."

has_today() matters because of the branching inside check().

check() {
  local lane="$1" count="$2" min="$3" note="$4" file="${5:-}" daymark="${6:-}"
  # ...
  if [ "$count" -ge "$min" ]; then
    log "ok lane=$lane count=$count"
    return
  fi
  if [ -n "$file" ] && ! has_today "$file" "$daymark"; then
    UNKNOWN="${UNKNOWN:+$UNKNOWN,}$lane"
    log "UNKNOWN lane=$lane 当日行なし"
    return
  fi
  FAILED="${FAILED:+$FAILED,}$lane"
}

When the count falls below min, if has_today() returns false the lane goes into UNKNOWN rather than FAILED. If you conflate the two, a lane that only runs three times a week will emit UNHEALTHY every day — Monday, Wednesday, and Friday included. The moment alerts stop being trusted, your monitoring is finished.

`count_dead_runs()` — Catching Runs That Finished but Produced Nothing

count_dead_runs() {
  local file="$1" day="$2"
  [ -f "$file" ] || { echo ""; return; }
  /usr/bin/grep -a "^$day" "$file" 2>/dev/null \
    | /usr/bin/grep -E '終了 \((need-login|circuit-break|error|rate-limit)\)' \
    | /usr/bin/grep -c 'いいね:0' || true
}

There's a reason this is a two-stage grep. The first regex, 終了 $(need-login|circuit-break|error|rate-limit)$, narrows to "lines with a harmful termination reason," and the second, grep -c 'いいね:0', narrows to "and zero output." If you counted a run that terminated early on rate-limit but still managed a few likes as a "dead run," you'd get an alert every time a mild nighttime rate limit kicks in. The point is to AND the two conditions to isolate "genuinely accomplished nothing."

This function is used in the outcome-based monitoring loop.

for lane in x-1 ig-1 ig-2 ig-3 ig-sug th-1 th-2 tt-1 tt-2 tt-3; do
  lf="$SA_LOGS/$lane.log"
  [ -f "$lf" ] || continue
  likes=$(sum_likes_today "$lf" "$TODAY_UTC")
  dead=$(count_dead_runs "$lf" "$TODAY_UTC")
  if [ -n "$likes" ] && [ "$likes" = "0" ]; then
    FAILED="${FAILED:+$FAILED,}$lane-likes0"
    log "UNHEALTHY lane=$lane 本日のいいね合計=0 (成果ゼロrun=${dead:-?}回)"
  elif [ -n "$dead" ] && [ "${dead:-0}" -ge 3 ]; then
    FAILED="${FAILED:+$FAILED,}$lane-dead${dead}"
    log "UNHEALTHY lane=$lane 成果ゼロrunが${dead}回 (いいね合計=${likes})"
  else
    log "OK lane=$lane いいね合計=${likes:-?} 成果ゼロrun=${dead:-0}回"
  fi
done

The branch differs between likes being an empty string (the file itself doesn't exist) and being "0". In the empty case it simply skips to the next lane rather than reporting UNKNOWN. That subtle distinction is what keeps "a lane not yet promoted to monitored status" from being confused with "a monitored lane that isn't running."

The Three-Stage Pipeline of `sum_likes_today()`

sum_likes_today() {
  local file="$1" day="$2"
  [ -f "$file" ] || { echo ""; return; }
  /usr/bin/grep -a "^$day" "$file" 2>/dev/null \
    | /usr/bin/grep -oE 'いいね:[0-9]+' \
    | /usr/bin/grep -oE '[0-9]+' \
    | /usr/bin/awk '{s+=$1} END {print s+0}'
}

Log lines start with the 2026-08-24 ... format, so stage one narrows to today's lines with "^$day". Stage two extracts marker-prefixed numbers with grep -oE 'いいね:[0-9]+', and stage three strips down to the bare digits and sums them with awk. The reason for the intermediate grep is that log lines can contain multiple markers, like いいね:0 フォロー:3 — pulling with just [0-9]+ would also pick up the 3 from フォロー:3.

The フォロー total uses the same structure in a separate function, sum_follows_today(). Likes and follows are independent outcomes, and there are cases where likes are zero but follows are working, so they must always be judged separately.

ensure_chrome.sh — From Killing pkill to a Skip Design

Here's the core of the fix in real code. Before the fix, it tried to solve everything in one line.

# 修正前：CDP 9223 が応答しない → 無条件で殺す
pkill -f "user-data-dir=$PROFILE"

After the fix, it's three stages.

# 修正後
if pgrep -f "user-data-dir=$PROFILE" > /dev/null; then
  # 他プロセスが掴んでいる → 10秒間隔・最大420秒ポーリング
  for i in $(seq 1 42); do
    sleep 10
    pgrep -f "user-data-dir=$PROFILE" > /dev/null || break
  done
  if pgrep -f "user-data-dir=$PROFILE" > /dev/null; then
    echo "他ジョブが使用中(pid=$(pgrep -f "user-data-dir=$PROFILE"))。Cookie破壊を避けるため起動を見送る" >&2
    exit 2
  fi
fi
# ここまで来たら誰も掴んでいない → SingletonLock 等の掃除 → 起動処理へ

The reason I didn't fold exit 2 into "error" (exit 1) rests on a design fact. IG carousel posting has three slots a day: 19:00, 21:00, and 23:00. Skipping one slot still leaves the next slot to post the same content. With exit 1, on the other hand, Discord alerts would keep firing every time the th-scent collision window comes around, and genuinely abnormal alerts would drown in the noise. "Skipped" is a third state that is neither failure nor success, and representing it with a dedicated exit code lets the caller, daily_post.sh, receive it in an independent branch.

It also matters that the SingletonLock cleanup now happens only when nobody is confirmed to be holding the profile. Before the fix, it unconditionally ran rm -f SingletonLock SingletonSocket SingletonCookie right after pkill. If you delete lock files while someone is using the profile, the process using it crashes and leaves the profile in a half-broken state.

Where I Got Stuck

"I Was Watching err.log" — Two Weeks of Daily False UNKNOWNs

My first watchdog implementation pointed the IG post check at the wrong file.

# 🔴 誤り：err.log は 2026-08-09 で更新が止まっている
check "ig-autopost" \
  "$(count_today "$CL_LOGS/sns-ig-autopost.err.log" "$TODAY_JST" 'OK posted')" ...

In reality, post success/failure logs go to retry.log, and err.log had stopped updating after 2026-08-09. has_today() kept returning false, and every morning ⚠️ SNS監視が判定不能: ig-autopost（当日行がログに無い） arrived in Discord.

Judged by symptoms alone, it looks like "the ig-autopost log is broken." The log wasn't actually broken — I just had the filename wrong. It took me two days to find the cause. I only noticed after checking the file's mtime with ls -la. The comment is still in the code:

# 成果は err.log ではなく retry.log に出る。err.log は 2026-08-09 で更新が止まっており、
# ここを見ている限り毎日 UNKNOWN 誤報になる(実際は当日3本投稿できていた)。
check "ig-autopost" \
  "$(count_today "$CL_LOGS/sns-ig-autopost.retry.log" "$TODAY_JST" 'OK posted')" ...

A monitoring script that reads log files can't notice "the file went stale = monitoring is dead" unless there's a mechanism that periodically checks the mtime of the files it reads. Receiving false alarms for two weeks is the same state as receiving no alerts at all.

Only TikTok Has a Different Date Format

TikTok's log inherited the format launchd emits, so its date format differs from the other lanes.

# X・IG・Threads: TODAY_UTC = "2026-08-24"
# TikTok だけ: TODAY_HUMAN = "Mon Aug 24" （%e で日を空白詰め → "Mon Aug  8"）
TODAY_HUMAN="$(date '+%a %b %e')"

%e pads the day with a space, so August 8th becomes Aug 8 (two spaces). Grepping TikTok's log with "^$TODAY_UTC" never matched the date format at all, so it always returned 0.

tt_ok=$(count_today "$CL_LOGS/tiktok-bokuwalily.log" "$TODAY_HUMAN" 'run end (exit 0)')
tt_maybe=$(count_today "$CL_LOGS/tiktok-bokuwalily.log" "$TODAY_HUMAN" 'run end (exit 2)')
check "tt-autopost" "$((tt_ok + tt_maybe))" 1 \
  "TikTok投稿(exit0=$tt_ok exit2=$tt_maybe)" "$CL_LOGS/tiktok-bokuwalily.log" "$TODAY_HUMAN"

The fix is just splitting out a separate variable that uses TODAY_HUMAN, but the reason I got stuck finding the cause is that "since every other lane was working fine, the TikTok problem looked like a TikTok-side defect." Format inconsistencies inside a script are invisible unless you line them up against the working lanes and compare.

"Zero Follow Successes, 20 Undecidable" — The friendship API Throttle

count_undecidable_today(), which I added on 2026-08-22, was born from this lesson.

count_undecidable_today() {
  local file="$1" day="$2"
  [ -f "$file" ] || { echo "0 0"; return; }
  /usr/bin/awk -v day="$day" '
    index($0, day) && index($0, "follow成功") { success++ }
    index($0, day) && index($0, "follow判定不能") { undecidable++ }
    END { print success + 0, undecidable + 0 }
  ' "$file"
}

The friendship API is Instagram's private API, used to check follow relationships. When it gets throttled, "is this person following me back?" becomes undecidable, and the job records follow判定不能 and moves on. Likes are working fine, so the sum_likes_today() check passes. From the outside, "ig-1 is healthy again today."

The actual symptom was 2255 uncollected entries piled up in ff-ig-2. Since unfollows can't happen, the backlog grows, you approach the follow limit, and one day follows suddenly stop too. I added the combination of follow成功=0 AND follow判定不能>=20 as a dedicated detection condition.

for lane in ff-ig-1 ff-ig-2 ff-ig-3; do
  # ...
  if [ "$follow_success" -eq 0 ] && [ "$undecidable" -ge 20 ]; then
    FAILED="${FAILED:+$FAILED,}$lane-blocked"
    log "UNHEALTHY lane=$lane follow成功=0 判定不能=$undecidable -> friendship APIが絞られている疑い"
  fi
done

"source-follow: All Sources Failed" — Four Times in a Row, Reported to Nobody

The source-followers lane is a subprocess that collects follower lists to find follow targets. Until 2026-08-22 it wasn't included in the monitoring loop at all.

# 2026-08-22 追加前は、このレーンの成果は一度も監視されていなかった。
# 実害: ig-sug が 23:14/02:13/11:16/14:16 と4回連続で
# 「coverage 0/17 -> 全ソース取得失敗 -> exit 1」になり日次20件で止まっていたのに、
# 誰にも報告されなかった（同時刻に ig-3 も14時間ゼロ成果）。
# 実体は private API の 429。

ig-sug emitted 全ソース取得失敗 in four slots: 11 PM, 2 AM, 11 AM, and 2 PM. It was falling over with exit 1 at zero coverage against 17 target accounts, but because sum_likes_today() on the main lane's ig-sug.log was working normally on its own, the watchdog summary said OK.

The monitoring I added watches source-followers-*.log in an independent loop.

for lane in ig-1 ig-2 ig-3 ig-nagi ig-sug th-1 th-2 th-3 th-nagi x-1 x-2 x-nagi x-reina; do
  sf="$SA_LOGS/source-followers-$lane.log"
  [ -f "$sf" ] || continue
  sf_dead=$(count_today "$sf" "$TODAY_UTC" '全ソース取得失敗')
  if [ "$sf_dead" -ge 2 ]; then
    FAILED="${FAILED:+$FAILED,}$lane-srcfollow-blocked"
    log "UNHEALTHY lane=$lane-srcfollow 全ソース取得失敗=${sf_dead}回 -> APIスロットリング疑い"
  fi
done

I use sf_dead >= 2 as the threshold because a one-off failure (a temporary 429) self-recovers on the next run. Two consecutive failures let you conclude "the throttling is ongoing."

Deleting My Neighbor's Cookies Daily via pkill — A Structure Where the Symptom Can Only Look Like "Logged Out"

This was the nastiest failure of all. The symptoms were clear. No OK posted at all in IG's sns-ig-autopost.retry.log. 🚨 SNS当日未出力: ig-autopost arriving in Discord from the watchdog. The session checker correctly reporting sessionid がない（need-login）.

I re-logged in via the GUI every time. It would be back the next morning. It would be gone again the next night. This cycle went on for several days, and I was starting to form the hypothesis that "maybe Instagram shortened its session lifetime."

In reality, ensure_chrome.sh was pkill-ing the profile used by social-autolike's th-scent job every night. th-scent launches Chrome with Playwright but never opens CDP port 9223. From ensure_chrome.sh's point of view, "9223 is closed = a dead process." pkill sends SIGTERM → SIGKILL, Chrome can't complete its shutdown routine, and it can't write the Cookie SQLite back in a consistent state. On the next launch, Chrome recreates an empty Cookie DB.

I found the cause when I lined up the log timestamps side by side.

22:56  th-scent unfollow start (.profiles/chrome-ig を掴む)
23:00  daily-post → ensure_chrome.sh → CDP 9223 応答なし
23:00  pkill -f user-data-dir=.profiles/chrome-ig
23:00  Cookie SQLite 破壊 → 空の DB 作り直し
23:00  daily-post: need-login → 投稿ゼロ

Because "the side doing the breaking (ensure_chrome.sh / scent-media)" and "the side reporting the breakage (retry.log / the session checker)" live in different repositories, reading only one of them will never connect the dots. The setting "reuseProfile": ".profiles/chrome-ig" in social-autolike's config/accounts.json appears nowhere in scent-media's code.

You must not write a presence check for a shared resource as "does the interface I expect respond?" — this incident was the first time I learned that lesson. 9223 being closed doesn't mean "dead"; it may mean "not launched in my particular style." Identify the owner with interface-independent means (pgrep / lock files), and allow kill only against processes you started yourself — that principle is what led to the pkill removal in commit 15307d7.

Pitfalls

Here are the traps I hit in an environment running multiple repositories on the same Mac. On top of the "wrong err.log filename," "TikTok date format difference," "friendship API throttling," "source-follow monitoring gap," and "Cookie destruction via pkill" detailed in the previous part, the same structural hole showed up in other forms too.

Unfollow monitoring didn't exist at all. Unfollow lanes like ig-1.unfollow.log / ig-2.unfollow.log / tt-1.unfollow.log weren't in the monitoring loop at all until 2026-08-17. The damage shows up in measured numbers: ig-1 had 712 unfollows uncollected for over 72 hours, ig-2 piled up to 2255, and tt-1 was unable to unfollow 11 times in a row — while the watchdog returned result=healthy every day. When unfollows jam up, you approach the follow limit, and a few days later follows stop too. Because the direct symptom (follows stopped) is one step removed from the root cause (unfollows jammed first), digging out the cause takes time.
"Zero like attempts" is a different failure from "zero like results." th-2 was launching daily with likeBudget=400, but the log contained not a single line with the string live like. It was only attempting follows and ending on circuit-break. From sum_likes_today()'s point of view, this looks like one verdict: "total likes = 0." But "attempted and got blocked every time" and "never attempted at all" have completely different causes. The former is an account restriction; the latter is a selector mismatch or an action-flow bug. The detection loop that makes this distinction possible now lives at lines 244–259 of sns-output-watchdog.sh:

# 「起動した」記録があるのに「いいね試行」が0回 = セレクタ不一致/アクション制限の疑い
for lane in x-1 x-4 x-5 ig-1 ig-2 ig-3 ig-sug th-1 th-2 tt-1 tt-2 tt-3; do
  lf="$SA_LOGS/$lane.log"
  [ -f "$lf" ] || continue
  started=$(count_today "$lf" "$TODAY_UTC" '開始 [')
  attempts=$(count_today "$lf" "$TODAY_UTC" 'live like')
  if [ "$started" -ge 1 ] && [ "$attempts" -eq 0 ]; then
    FAILED="${FAILED:+$FAILED,}$lane-noattempt"
    log "UNHEALTHY lane=$lane いいね試行が0回(セレクタ不一致/アクション制限の疑い) 起動=${started}回"
  fi
done

Mixing FAILED and UNKNOWN destroys trust in your alerts. If you run a lane that only launches three times a week through check() every day, the days it doesn't launch have no log lines for that day, so it's treated as zero count and piled into FAILED daily. When "🚨 SNS当日未出力: lane-X" arrives in Discord four days a week, a genuine outage alert one day gets skimmed past as "there it goes again." The moment alerts stop being trusted, your monitoring infrastructure is finished. After adding the branch that checks "does a log line for today exist?" with has_today() and routes lanes with no lines to UNKNOWN, my false-positive rate dropped by what felt like 90%+.
Mixing UTC and JST creates a bug that only breaks at certain times of day. The gap between TODAY_JST and TODAY_UTC is 9 hours. If a job that runs between midnight and 9 AM Japan time aggregates with TODAY_UTC, the string 2026-08-24 in the log matches the previous UTC date, and output completed overnight gets counted as "yesterday's success." From today's monitoring perspective it looks like zero, and a FAILED alert fires. Since it passes without issue when run during the day, it looks like "occasional false alarms" and the cause takes longer to find. I make this mixture explicit at the top of sns-output-watchdog.sh:

TODAY_JST="$(date '+%Y-%m-%d')"
TODAY_UTC="$(date -u '+%Y-%m-%d')"
TODAY_HUMAN="$(date '+%a %b %e')"  # launchdが出す形式。日は空白詰めなので %e

Using the three variables appropriately and matching each log to the one it actually emits resolved it, but every time I add a new lane it needs re-checking.

Discord notifications can silently fail to send. The notify() function does [ -f "$DISCORD" ] || return 0, so if discord_tool.py doesn't exist, it exits successfully without notifying. You can end up in a state where the watchdog log correctly records result=unhealthy lanes=ig-1-likes0 but nothing arrives in Discord. I have a track record of the path being wrong on day one of deployment and notifications silently not going out. To verify that a monitoring script is actually raising alerts, you have to check on the receiving end (Discord's last-received timestamp). Looking only at the watchdog log, you'll never notice.
Nobody notices when a referenced log file goes stale. has_today() checks "is there a line for today?" but doesn't guarantee "has this file been updated recently?" Right after a job stops, has_today() also returns false and the lane becomes UNKNOWN — but the state where the file exists and has today's lines, i.e. "it ran once today, but every run after that has silently failed," is undetectable. Without a mechanism to periodically check the mtime of referenced logs, your monitoring falls into "continuing to judge today on stale evidence."
Shared configuration across multiple repositories is "written only in the config file." Even though social-autolike/config/accounts.json says "reuseProfile": ".profiles/chrome-ig", the code in scent-media/scripts/ensure_chrome.sh contains no mention whatsoever that this profile is also used by another repo. In day-to-day work you look at git log / git diff separately for each, so collisions keep happening with no visible point of contact. I wrote this lesson into the learning note learning/shared-resource-kill-corrupts-the-neighbor.md: "Write it in the code of the side being shared, not the side doing the sharing" — that's the only way to minimize discovery cost.
Among completed runs, "runs that genuinely accomplished nothing" need to be counted separately from zero output. If you count a run that terminated early on rate-limit but still managed 2 likes as a "dead run," you'll get an alert every night from mild nighttime rate limiting. That's exactly why count_dead_runs() narrows to "a harmful termination reason (need-login / circuit-break / error / rate-limit) AND いいね:0." Changing that AND to an OR alone would send the false-positive rate through the roof.

Best Practices

Here are the design principles I actually hit and fixed, in reproducible form.

1. Check the presence of a shared resource in an interface-independent way

"Does CDP port 9223 respond?" is not a check for "is Chrome running" — it's a check for "is a Chrome that was launched in my particular style running." A Chrome on the same profile launched by someone else exists without opening the port. Check process presence on a PID basis, like pgrep -f "user-data-dir=$PROFILE", and don't depend on whether an interface responds.

2. Allow kill only against processes you started yourself

pkill -f <pattern> takes down every process matching the pattern. It doesn't ask who started them. If there's even a 1% chance someone else started it, fall back to "wait and skip" instead of kill. The cost of killing the wrong process isn't "this run failed" — it's "every slot until recovery, plus human GUI work."

3. Return unacquirable resources with "bounded polling → a dedicated exit code"

# 10秒間隔・最大420秒ポーリング
for i in $(seq 1 42); do
  sleep 10
  pgrep -f "user-data-dir=$PROFILE" > /dev/null || break
done
if pgrep -f "user-data-dir=$PROFILE" > /dev/null; then
  echo "他ジョブが使用中。Cookie破壊を避けるため起動を見送る" >&2
  exit 2   # 「今スロットは見送り」
fi

If you fold it into exit 1 (error), Discord alerts keep firing on every slot collision and real outages get buried. Only when there's a design fact like "with 3 slots a day, one skip ≈ zero loss" can you treat exit 2 as "skipped = normal." Assign a dedicated code to states not worth alerting on, and have the caller handle it in an independent branch.

4. Monitor by outcome logs, not exit codes

exit 0 only means "it didn't crash." In measurements on 2026-08-08, ig-1 finished 8 of 12 runs — and tt-1 / tt-2 60% of theirs — with exit 0 and zero likes. During the period when monitoring only looked at exit codes, this state persisted for two weeks and nobody noticed. Judge whether output happened by aggregating actual like counts, follow counts, and post counts from the logs with awk.

5. Always branch between "zero count" and "no lines for today"

check() {
  # ...
  if [ "$count" -ge "$min" ]; then return; fi        # 健全
  if ! has_today "$file" "$daymark"; then
    UNKNOWN="${UNKNOWN:+$UNKNOWN,}$lane"; return      # 判定不能
  fi
  FAILED="${FAILED:+$FAILED,}$lane"                   # 異常
}

When a lane that only runs three times a week emits FAILED four days in a row, the alert becomes "there it goes again." Don't bundle "today was a non-run day" and "it ran today but produced zero" into the same alert.

6. Split zero-output, zero-attempt, and zero-completion into three tiers

Monitoring precision improves in three stages.

Stage	What's aggregated	Failure it can detect
Stage 1	Number of completed runs	Whether the job is crashing
Stage 2	Total actual like count	Running but producing nothing
Stage 3	Number of like attempts	Not even attempting

Until I added stage 3, th-2's selector mismatch had been invisible for over two weeks.

7. Keep an independent monitoring loop per type of outcome

Posting, liking, following, unfollowing, and source collection are each different processes with different failure modes. Trying to watch them all in one loop mixes different "definitions of output" and produces misjudgments. sns-output-watchdog.sh currently has six independent loops: post checks, outcome-based like verification, follow verification, unfollow verification, source-follow verification, and zero-attempt detection. It looks redundant, but without that separation I'd never have found "2255 unfollows piled up in ig-2."

8. Periodically inventory whether all lanes are centrally managed

When you add a new lane, it's easy to forget to add it to the monitoring loop. source-followers-*.log wasn't included in monitoring until 2026-08-22, so ig-sug failing all-source collection four times in a row was reported to nobody. Make it routine to cross-check, once a month, the list of running jobs against the list of monitored lanes and confirm the number of unmonitored lanes is zero.

9. Align date formats, and when there's an exception, make it explicit in the variable name

If you're using three variants — date '+%Y-%m-%d', date -u '+%Y-%m-%d', and date '+%a %b %e' — confirm first which format each log emits, then map the variables accordingly. Timestamps emitted by launchd can be in %e (space-padded) format. Aug 8 and Aug 8 do not match under grep. Putting the meaning "this one's format differs from the others" into the variable name, as with TODAY_HUMAN, makes it easier for a later reader to notice.

10. Annotate cross-repository sharing in the code of the side being referenced

"Writing it in the config file of the referencing side" alone is invisible to whoever reads the code of the referenced side. If the implementer reading ensure_chrome.sh knew that "this profile is also used by social-autolike's th-scent via reuseProfile," they'd stop before writing pkill. Document "who uses it and for what" in a comment on the shared resource, placed inside the code of the side being referenced.

11. Design on the assumption that force-killing a process corrupts persistent state

A process that receives SIGKILL can't run its shutdown handlers. In Chrome's case, it exits without writing the Cookie SQLite back in a consistent state and recreates an empty DB on next launch. As a measured value, instagram.com cookies were confirmed to have shrunk to 0 rows. When force-killing a process that holds SQLite, caches, or session information, either prepare a recovery procedure in advance on the assumption that persistent state will be corrupted, or switch to a design that doesn't force-kill in the first place.

12. Verify that your notification infrastructure works from the receiving end

Even when the watchdog log records result=unhealthy, there are cases where nothing arrives in Discord. The notify() function silently return 0s if the path to discord_tool.py doesn't exist. The script only leaves a record of "attempted to notify." Looking at the watchdog log alone is not enough to confirm monitoring is functioning. You need to separately check the last-received timestamp in Discord, or periodically send a "test notification" and confirm connectivity from the receiving end.

13. Skipping is neither failure nor success — express the third state with a dedicated exit code

There are situations where the binary exit 0 = success / exit 1 = failure isn't enough. "The conditions weren't right for this slot, so I skipped it" is not a success, but it's not a failure either. By assigning exit 2 to "skip this slot" and having the caller keep an independent branch that treats it as "retry at the next slot, exit 0," you can convey state accurately without adding alert noise. The premise that makes this design work is the design fact that there are multiple slots per day. A skip design doesn't fit a job that only has one slot a day.

Summary

Compressed into one sentence, this incident is: "my own automation was destroying the persistent state of my own other automation, every day."

The breaking side (ensure_chrome.sh / scent-media) and the reporting side (sns-ig-autopost.retry.log / the session checker) lived in different repositories, and each was working correctly. ensure_chrome.sh was correctly detecting "a Chrome process where CDP 9223 doesn't respond." The session checker was correctly reporting "there is no IG sessionid." To a human, it can only look like "IG got logged out."

This is the nastiest structure a failure can have. "The report is correct, but the cause is somewhere else" — with this shape, no amount of chasing symptoms in detail will get you to the cause. Only when I lined up the outcome logs from sns-output-watchdog.sh against the launchd start times side by side, and confirmed that the 22:56 th-scent unfollow start and the 23:00 ensure_chrome.sh execution were 4 minutes apart, did the two repositories connect. The general form I recorded in the learning note learning/shared-resource-kill-corrupts-the-neighbor.md is now the criterion for my design decisions going forward.

The core of the fix is deleting one line of pkill. Commit 15307d7, which removed pkill -f "user-data-dir=$PROFILE" and replaced it with 10-second-interval polling up to 420 seconds plus exit 2 (skip), doesn't even add 10 lines of code. But without the context that "another repository is using the same profile," you'd never reach the decision to delete that one line.

sns-output-watchdog.sh is now a script of over 270 lines. X posts, likes, replies / IG posts, likes, follows, unfollows / two Threads accounts / TikTok / 13 source-followers lanes — each has a different definition of output and is judged daily by an independent aggregation function. The first version only looked at "exit code 0 = healthy." From there, each time a function was added — count_today() / has_today() / count_dead_runs() / sum_likes_today() / count_undecidable_today() — another invisible "silent failure" surfaced.

If you treat an SNS foundation as something you merely "build," then from the moment it breaks you're in a state of "stopped while appearing to run." The starting point was IG posting being stopped since 7/29 and not noticing for two weeks. What today's ¥1.2M/month foundation rests on isn't a mechanism for checking whether 171 jobs are running — it's one for checking daily whether those 171 jobs are actually producing output.

Automation is two processes as a set: "making it run" and "making it observable." If either is missing, you can't tell for yourself whether this month is going well or failing. And if you can't tell, you can't decide your next move.

If you're running multiple automations on one machine: do you actually know which of them share a resource?

I've written up the full picture of the system, the ¥1.2M/month breakdown, and the 30-day process in a paid note.

📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Make Codex Prove It: A Three-File Design That Leaves Evidence on Disk

Lily — Sun, 30 Aug 2026 00:00:46 +0000

An AI agent telling you "done" is not evidence. When I started delegating work to Codex, I took those reports at face value — until I checked the code and found the change missing, the wrong file edited, or no commit at all. So I stopped trusting language and started making the shell write the facts to disk.

Why this design works

When you hand a task to Codex, it comes back with "Completed." At first that satisfied me. But when I actually checked the code, the critical change wasn't there, or a different file had been touched, or git commit had never run. The output "I did it" and the fact "it was actually done" are two different things.

This is true of Claude Code too. Whether tool results were read correctly, whether errors were swallowed — even with code I wrote myself, running a self-audit right after declaring completion turns up something every single time. Delegating implementation to an AI amplifies that problem by one more notch.

The fix is simple: make it write state to a file, not to language.

Even if the AI says "completed," it isn't complete unless State: completed exists in the status file. If the handoff file doesn't contain the real output of git status --short, you don't know what changed. If the four sections you specified in the task file (Summary, Files Changed, Validation, Remaining Risks) aren't there, you can't verify it.

Files don't lie. An AI under pressure will insist "I did it," but the output of cat status-file can't be forged. Pushing state management down into the filesystem is what makes it possible for a human to cross-check it in a shell. That's the essence of this design.

The other important piece is separation of concerns. orchestrate-codex-worker.sh takes three arguments up front.

bash scripts/orchestrate-codex-worker.sh <task-file> <handoff-file> <status-file>

Each of these three files has a clear role.

task-file: The work order for Codex. It contains only "what to do."
handoff-file: The handoff note after Codex finishes. Written on success or failure alike, on the assumption that the next person in line (the next Claude Code session, or me) will read it.
status-file: Machine-readable progress state. It takes only three values: State: running → State: completed / State: failed.

What happens without this separation? When instructions and execution results live in the same place, "is this an instruction or post-execution output?" becomes ambiguous. When you run large numbers of tasks in parallel, that ambiguity is fatal. Multiple Codex workers can run in the same directory without interfering as long as each has its own independent task/handoff/status files.

On top of that, the script starts with set -euo pipefail. That's a declaration that "the entire script stops the moment any command fails." Without it, Bash ignores errors and moves to the next line. With set -euo pipefail, the behavior becomes: if git rev-parse fails, stop; if mkdir fails, stop. A design that doesn't swallow errors matters especially in a script built on the premise of AI delegation.

The overall flow

Following the script's behavior in order looks like this.

呼び出し元（Claude Code / cronジョブ等）
    │
    ├─ task-file を渡す（作業指示）
    ├─ handoff-file のパスを渡す（引き継ぎ先）
    └─ status-file のパスを渡す（状態管理先）
         │
         ▼
orchestrate-codex-worker.sh
    │
    ├─ [起動直後] write_status "running"
    │        └→ status-file: State: running / Branch / Worktree / timestamp
    │
    ├─ task-file の読み込み確認
    │   ├─ [失敗] write_status "failed" + handoff-fileにエラー書き出し → exit 1
    │   └─ [成功] 処理継続
    │
    ├─ mktemp で prompt_file / output_file を作成
    │   └─ trap cleanup EXIT（終了時に自動削除）
    │
    ├─ prompt_file を組み立て（Codexへの指示 + task-fileの内容）
    │
    ├─ codex exec -p yolo -m gpt-5.4 -C $(pwd) -o output_file < prompt_file
    │   │
    │   ├─ [成功]
    │   │     handoff-file に書き出し:
    │   │       - Completed: timestamp
    │   │       - Branch: git rev-parse --abbrev-ref HEAD
    │   │       - Worktree: pwd
    │   │       - output_file の内容（Summary/Files Changed/Validation/Remaining Risks）
    │   │       - git status --short
    │   │     write_status "completed"
    │   │
    │   └─ [失敗]
    │         handoff-file に書き出し:
    │           - Failed: timestamp / Branch / Worktree
    │           - "The Codex worker exited with a non-zero status."
    │         write_status "failed" → exit 1
    │
    └─ 完了

Looking at the write_status function — the core of the actual code — shows what's being recorded.

write_status() {
  local state="$1"
  local details="$2"

  cat > "$status_file" <<EOF
# Status

- State: $state
- Updated: $(timestamp)
- Branch: $(git rev-parse --abbrev-ref HEAD)
- Worktree: `$(pwd)`

$details
EOF
}

git rev-parse --abbrev-ref HEAD returns the branch name at that moment. $(pwd) is the absolute path of the worktree. The timestamp comes out as UTC ISO 8601 via date -u +"%Y-%m-%dT%H:%M:%SZ".

In other words, the status file tells you line by line when, on which branch, in which worktree, and in what state. When running in parallel across multiple worktrees, just looking at the Worktree field in the status file identifies which is which.

The prompt handed to Codex is also assembled directly inside the script.

cat > "$prompt_file" <<EOF
You are one worker in an ECC tmux/worktree swarm.

Rules:
- Work only in the current git worktree.
- Do not touch sibling worktrees or the parent repo checkout.
- Complete the task from the task file below.
- Do not spawn subagents or external agents for this task.
- Report progress and final results in stdout only.
- Do not write handoff or status files yourself; the launcher manages those artifacts.
- If you change code or docs, keep the scope narrow and defensible.
- In your final response, include exactly these sections:
  1. Summary
  2. Files Changed
  3. Validation
  4. Remaining Risks

Task file: $task_file

$(cat "$task_file")
EOF

Two things stand out.

"Do not write handoff or status files yourself" — an explicit prohibition. Who writes the handoff/status files is a key design fork. If you let Codex write them, Codex may output something that merely looks right. Writing them from the script side means you get the actual output of git status --short, the actual branch name from git rev-parse --abbrev-ref HEAD, and the actual time from timestamp. Those can't be falsified.

Forcing four sections. Requiring "exactly these sections" in Codex's output fixes the positions that downstream processing and review will reference. Open the handoff file, read the "Files Changed" section, and you have the list of changed files; read the "Validation" section and you have the verification commands Codex actually ran and their results. With free-form output, when the next session loads the handoff file you no longer know where to look.

The codex invocation itself is one line.

codex exec -p yolo -m gpt-5.4 --color never -C "$(pwd)" -o "$output_file" - < "$prompt_file"

-p yolo means no confirmation prompts, -m gpt-5.4 specifies the model, -C "$(pwd)" sets the working directory, -o "$output_file" sets the output destination, and - < "$prompt_file" tells it to read the prompt from stdin. --color never keeps ANSI escape sequences from contaminating the handoff file, so no junk characters get in the way when you grep or parse it later.

Whether this call succeeds (exit 0) or fails (non-zero) decides the branch that follows. Because the script has set -euo pipefail, a failing codex command doesn't exit outright — the if codex exec ...; then ... else ... fi structure routes it into the error branch. Even on failure, the handoff file and status file are always written. Never producing a state of "no file = it was never even run" is what the later cross-check verification requires.

Implementation details

What `trap cleanup EXIT` protects

There's a mechanism I didn't cover in the first half that you should read first.

prompt_file="$(mktemp)"
output_file="$(mktemp)"
cleanup() {
  rm -f "$prompt_file" "$output_file"
}
trap cleanup EXIT

mktemp creates a temp file like /tmp/tmp.XXXXXX. trap cleanup EXIT declares "when the script exits — whether exit 0 or exit 1 — run the cleanup function."

Why is this needed? codex exec reads prompt_file and writes to output_file, but Codex sometimes exits non-zero. When Codex fails in a set -euo pipefail environment, the script enters the else block and ends with exit 1. Without trap, /tmp/tmp.XXXXXX would linger. Once is fine, but run 30 workers in parallel and /tmp bloats. With trap cleanup EXIT, the temp files disappear no matter which path exits.

One more point: note that prompt_file and output_file are global variables. At definition time, the cleanup function doesn't know the contents of $prompt_file / $output_file. It reads the variable values at exit, when the function runs. That's exactly why the order is: assign the variables right after mktemp, then set the trap. Reverse the order and cleanup tries to delete empty paths and errors out.

Why the task-file existence check comes before "running"

Reading the script, the task-file existence check comes before write_status "running".

mkdir -p "$(dirname "$handoff_file")" "$(dirname "$status_file")"

if [[ ! -r "$task_file" ]]; then
  write_status "failed" "- Error: task file is missing or unreadable (\`$task_file\`)"
  {
    echo "# Handoff"
    echo
    echo "- Failed: $(timestamp)"
    echo "- Branch: \`$(git rev-parse --abbrev-ref HEAD)\`"
    echo "- Worktree: \`$(pwd)\`"
    echo
    echo "Task file is missing or unreadable: \`$task_file\`"
  } > "$handoff_file"
  exit 1
fi

write_status "running" "- Task file: \`$task_file\`"

write_status "running" means "submission to Codex has begun." If the task file can't be read, the premise for submitting to Codex has collapsed, so it isn't entitled to be "running." It writes "failed" directly and exits 1.

The reason mkdir -p "$(dirname "$handoff_file")" "$(dirname "$status_file")" comes first is the same. When the task file can't be read and you try to write failed, if the directories for the handoff file or status file don't exist, that write itself fails. Make sure the parent directories exist before any write to the handoff/status files. That's why mkdir -p dirname ... sits at the top.

What the handoff file holds on success

When Codex's exit code is 0, the following gets written to the handoff file.

if codex exec -p yolo -m gpt-5.4 --color never -C "$(pwd)" -o "$output_file" - < "$prompt_file"; then
  {
    echo "# Handoff"
    echo
    echo "- Completed: $(timestamp)"
    echo "- Branch: \`$(git rev-parse --abbrev-ref HEAD)\`"
    echo "- Worktree: \`$(pwd)\`"
    echo
    cat "$output_file"
    echo
    echo "## Git Status"
    echo
    git status --short
  } > "$handoff_file"
  write_status "completed" "- Handoff file: \`$handoff_file\`"

cat "$output_file" pulls in Codex's entire output. Immediately after, echo "## Git Status" and git status --short follow.

That git status --short is the linchpin of verification. Suppose Codex wrote "Files Changed: src/api/index.ts, tests/api.test.ts" in its Summary — if git status --short shows nothing, that means git doesn't recognize any change to those files.

The cross-check commands I actually use are these.

# handoff-fileの"## Git Status"以降を確認
grep -A 20 "## Git Status" /path/to/handoff-file

# 実際のgit diffと比較
git diff --stat HEAD

If the handoff file's ## Git Status and git diff --stat agree, what Codex said matches git's actual state. If they don't, it's one of two things: "Codex thought it made changes but actually didn't," or "it went all the way through commit, so nothing showed in git status --short (= clean)." For the latter, I check the Branch in the status file and trace it with git log.

The reason for passing --color never to Codex lies here. If ANSI escape sequences (control characters like \e[32m or \033[0m) get into output_file, they're transcribed verbatim into the handoff file. When you run grep -A 20 "## Git Status" handoff-file, invisible control characters break the pattern match. --color never is the instruction "don't include ANSI codes in output," and without it, mechanical post-processing gets contaminated.

Don't assume "nothing changed anyway" on failure

The handoff file on failure is minimal.

else
  {
    echo "# Handoff"
    echo
    echo "- Failed: $(timestamp)"
    echo "- Branch: \`$(git rev-parse --abbrev-ref HEAD)\`"
    echo "- Worktree: \`$(pwd)\`"
    echo
    echo "The Codex worker exited with a non-zero status."
  } > "$handoff_file"
  write_status "failed" "- Handoff file: \`$handoff_file\`"
  exit 1
fi

Just the single line The Codex worker exited with a non-zero status. Codex's output is partially written to output_file, but cat "$output_file" is not executed here. Why?

The output_file on failure is incomplete output. The four sections may not all be there. An API error may have hit partway through, leaving everything from Summary onward unwritten. Mixing incomplete output into the handoff file means that when the next session reads it, you can't tell "is this completed output or partial output?" Record failure as failure, with zero content — that's the design intent.

To investigate what was happening after a failure, you need to capture the stderr of the running codex exec separately, not output_file. This script doesn't go that far, so on failure I enable Codex-side logging with an environment variable like CODEX_DEBUG=1 and check separately.

Where I got stuck

Before I added `set -euo pipefail`

When I first wrote this script, I didn't have set -euo pipefail at the top. Bash's default behavior is to ignore errors and move to the next line.

What happened? The Codex invocation failed, but the script didn't go into the next if ... then branch (back then it was a direct call, not an if statement) and write_status "completed" ran. The status file said State: completed. The handoff file had Completed: 2026-05-14T08:23:11Z. But git diff --stat showed nothing.

The symptom is "the status file says completed but the code hasn't changed." At first I thought "did Codex commit everything?" and looked at git log — nothing there either.

Pinning down the cause took 30 minutes. I ran codex exec manually and checked the exit code: it was 1. echo $? returned 1. But inside the script, that 1 was ignored and it moved to the next line.

The fix was just adding set -euo pipefail at the top and wrapping the Codex call in if ... then ... else ... fi. A two-line change. But before I noticed, I'd believed a "falsely completed" status file and stacked further work on top of it, which meant redoing all of it later.

When you write a script on the premise of delegating work to an AI, set -euo pipefail is not an option, it's a requirement. Bash's error-ignoring is tolerable when a human is debugging a script by hand, but with AI delegation, "silently continuing after a failure" becomes fatal.

Before I added `--color never`

When I tried to grep the handoff file, I got output like this.

## Git Status

?? src/^[[0mapi^[[0m/^[[32mindex.ts^[[0m

^[[0m is the ANSI reset code and ^[[32m specifies green. Dumping Codex's terminal output straight to a file lets ANSI escape sequences in.

In that state, running grep "index.ts" handoff-file doesn't match, because the pattern is index.ts but in the file it's split up as index^[[0m.ts. It's readable to the eye, but it falls apart when you try to process it with a script.

At first I tried stripping the ANSI codes in post-processing with sed 's/\x1b\[[0-9;]*m//g'. That works, but a sed pattern covering every terminal escape sequence is complex, and it breaks if Codex starts using different escapes in the future.

The fundamental fix is passing --color never to codex exec. Tell Codex not to emit ANSI codes in the first place. Controlling it at the entrance is simpler and more robust than post-processing with sed.

When Codex wrote the status file itself

There was a period when I put "when you finish the work, write to the status file" in the prompt. The idea was, "if Codex itself can record completion, the script doesn't need if/else."

What actually happened: Codex wrote State: completed. But the content wasn't real.

# Status

- State: completed
- Updated: 2026-05-20T14:33:00Z
- Branch: main
- Worktree: `/path/to/project`

Branch says main. But the actual worktree was on the feature/api-refactor branch. That's not git rev-parse --abbrev-ref HEAD — it's a string Codex guessed as "probably main." The timestamp wasn't the actual completion time either; it was a plausible-looking time Codex inferred from its training data.

Worse was the case where the Codex invocation failed partway through. Codex sometimes tries to write the status file "just in case" right before exiting with an error. The result was a state where codex exec exited non-zero, yet the status file said State: completed.

The current prompt has an explicit prohibition.

- Do not write handoff or status files yourself; the launcher manages those artifacts.

The reason for that one line is that "if you let Codex write it, the values become guesses instead of measurements." $(git rev-parse --abbrev-ref HEAD) is actually executed by the shell. The main Codex writes is guessed by the model. That difference decisively changes verification accuracy. Files that record state should be written by the shell.

When I passed the task file as a relative path

I once passed the task file as a relative path like ./tasks/refactor-api.md when calling the script. At that point the behavior of mkdir -p "$(dirname "$handoff_file")" went wrong.

Because the handoff file was also passed as a relative path like ./handoffs/refactor-api-handoff.md, when the script changed the working directory with cd (the version at the time did cd into the worktree), the path dirname computed ended up somewhere other than intended.

The symptom was a Permission denied or No such file or directory error meaning "can't write to the handoff file." When I debugged it and echoed the path dirname returned, it was /handoffs instead of /worktree/subdir/handoffs.

The fix is to convert to absolute paths on the caller side before passing them.

bash scripts/orchestrate-codex-worker.sh \
  "$(realpath ./tasks/refactor-api.md)" \
  "$(realpath -m ./handoffs/refactor-api-handoff.md)" \
  "$(realpath -m ./status/refactor-api.status.md)"

realpath returns the absolute path of an existing file. realpath -m computes and returns an absolute path even if the file doesn't exist (-m = --no-require-file). The handoff and status files are created by the script, so they don't exist at call time. Using realpath -m lets you fix the absolute path of a nonexistent file in advance.

Inside the script, $(dirname "$handoff_file") then always computes the dirname of an absolute path, so it's safe to call the script from any directory.

The loop that detects false completion

Based on these failures, the verification flow I use now is this.

# 1. status-fileでState確認
grep "State:" /path/to/status-file

# 2. 実際のgit diffと突き合わせ
git diff --stat HEAD

# 3. handoff-fileのGit Statusセクションと比較
grep -A 10 "## Git Status" /path/to/handoff-file

Only when all three agree can I confirm "what Codex said was actually done."

Status file says State: completed → git diff --stat shows changes → the handoff file's ## Git Status lists the same files. That's the three-piece set.

Conversely, the patterns where it breaks down are fixed.

State: completed but git diff --stat is empty → Codex said "completed" without making changes
git diff --stat shows changes but they're not in the handoff file's ## Git Status → there's a bug in the script's write order (I did this once)
State: failed but git diff --stat shows changes → Codex made partial changes and exited 1 (the dangerous pattern)

The last pattern needs the most caution. When Codex crashes after partial changes, git diff --stat shows changes, but whether those changes are "partway toward a correct change" or "a broken state" isn't clear until you read the code. In that case I read the full diff with git diff HEAD, then decide whether to fully revert or continue. Often I shelve it with git stash before running the next Codex worker.

When the handoff file contains only The Codex worker exited with a non-zero status., the cause of Codex's error only exists in stderr. This script currently doesn't capture stderr, so the only option is to reproduce it. When I submit the same task next time, I add one line to the task file: "the previous run exited non-zero; identify the cause of the error before starting." I count on the root cause of the error being written in the "Remaining Risks" of the four sections.

Since I started using this "leave state in files" design, the state of "I don't know what changed" after delegating to Codex has nearly disappeared. cat the status file, grep the handoff file, look at git diff --stat — with those three commands I can verify any worker's completion state in under 30 seconds. Completion that's "only said" doesn't get engraved into the filesystem.

Pitfalls

The task file's instructions were "plausible-sounding Japanese"

I once handed over a task file saying "please improve the API's error handling." Codex came back in the four-section format. State: completed was there too. The ## Git Status section existed. But the Git Status field was empty.

Reading the handoff file's Validation section, it said "after investigation, there are no problems with the current implementation." Because the instruction was vague, "a judgment that no improvement is needed" got processed as "task complete." Doing nothing while adhering to the four-section format functions as a loophole in the design.

After I standardized the task file into the following format, this pattern nearly vanished.

対象: ~/dev/myapp/src/api/client.ts
やること: fetchUser関数のcatch節でエラーをconsole.errorに出力し、呼び出し元へrethrowする
完了条件: catch節にconsole.error + throw eが入っていること
検証: grep -n "console.error" ~/dev/myapp/src/api/client.ts && grep -n "throw e" ~/dev/myapp/src/api/client.ts

When the completion criteria and verification command are written as a pair, Codex actually runs that command in the Validation section and pastes the stdout. The escape hatch of "investigated, no problems found" is closed off.

Running in parallel without separating git worktrees

orchestrate-codex-worker.sh passes the current worktree to Codex via -C "$(pwd)". What happens if you run two workers simultaneously in the same worktree?

While Codex A is rewriting src/api/client.ts, Codex B reads the same file and makes a different change. B finishes first and lands a git commit. A, unaware of the state after B's commit, stages the file it wrote out and commits on top. The result is a commit where A overwrote lines it never intended to.

git diff --stat had A's changes and B's changes mixed together, and I could no longer trace which worker wrote which lines. Looking at git log --oneline, the ordering between commits was a mess too.

When running parallel workers, always separate worktrees with git worktree add.

git worktree add ~/dev/myapp-worker-a feature/api-fix-a
git worktree add ~/dev/myapp-worker-b feature/cache-fix-b

If you move into each worktree's directory before launching the script, the status file's Worktree field splits into ~/dev/myapp-worker-a and ~/dev/myapp-worker-b, and it's obvious at a glance which worker's result you're looking at.

output_file was empty and nothing went into the handoff file

There were times when codex exec exited 0, yet there was nothing before ## Git Status in the handoff file. Because cat "$output_file" transcribed an empty file as is, Codex's four-section output was missing entirely.

The cause was a timeout on the Codex API side. When processing drags on and the API cuts the session, codex exec can exit 0 (behavior varies by Codex version). The output_file is created but is 0 bytes.

I added two countermeasures.

Wrap codex exec from the outside as timeout 600 codex exec .... It gets killed at 10 minutes and returns exit 124, so in a set -euo pipefail environment it enters the else block and failed is recorded. Dangling workers disappear.
Right after writing the handoff file, check its size with wc -c "$handoff_file" and, if under 200 bytes, log "the output may have been empty." The judgment is made by a human after looking at the status file.

Codex kept saying "Remaining Risks: None"

As I accumulated Codex's four-section outputs, I noticed None lining up in the Remaining Risks section. Yet multiple cases came up where a later worker reported "there was a bug in the code the previous worker implemented." Adhering to the format while skewing the content positive — that's a common Codex tendency.

After adding the following line to the task file, concrete risks started appearing.

- Remaining Risksには必ず1件以上の懸念事項を書くこと。「None」は禁止。

You start getting things like "the type definitions are provisional," "test coverage is low," "behavior against production data is unverified." By keeping Remaining Risks at one or more items at all times, you close off one form of false completion — completion by concealing problems.

A space snuck into the handoff file's absolute path

During a period when I had a folder with a Japanese name directly under my macOS home directory, I passed a handoff file path containing that path. "$(dirname "$handoff_file")" is wrapped in double quotes so it's safe against spaces, but because the calling script omitted quotes, dirname received a path split at the space.

The symptom is No such file or directory from mkdir -p. The handoff file's directory can't be created, so the task fails before it even starts.

I solved it with both thorough quoting on the caller side and realpath conversion. When passing paths containing spaces to a shell script, always wrap them in double quotes, including on the right-hand side of variable assignments.

Failure causes don't survive because stderr isn't captured

The handoff file on failure has only the single line The Codex worker exited with a non-zero status. Codex's error messages go to stderr, but this script doesn't redirect stderr, so they vanish the moment they hit the terminal.

The only option is to reproduce and identify the cause. I append "the previous run exited non-zero; identify the cause of the error before starting" to the next task file and hand it to Codex. I resubmit hoping the root cause of the error gets written in the "Remaining Risks" of the four sections.

Fundamentally fixing this requires a change that routes stderr to a separate file with codex exec ... 2>"$error_file" and runs cat "$error_file" into the handoff file on failure. It needs to be understood as an unaddressed area of the current script.

Best practices

1. Convert all three arguments with realpath before passing them

Convert the task file with realpath (errors if the file doesn't exist) and the handoff/status files with realpath -m (computes an absolute path even if they don't exist). Fixing relative paths into absolute paths at call time means the dirname inside the script always computes the correct path, no matter which directory it's called from.

TASK="$(realpath ./tasks/my-task.md)"
HANDOFF="$(realpath -m ./handoffs/my-task-handoff.md)"
STATUS="$(realpath -m ./status/my-task.status.md)"
bash scripts/orchestrate-codex-worker.sh "$TASK" "$HANDOFF" "$STATUS"

2. Write task files all the way through "completion criteria + verification command"

A task file that ends with "please improve X" leaves the interpretation to Codex. Write the completion criteria and an executable verification command as a pair, and Codex will actually run that command in the validation section and paste the result. The escape hatch of "investigated, no problems" gets structurally closed off.

3. Wrap it from the outside with timeout 600

codex exec has no CLI-level timeout argument. Putting timeout 600 codex exec ... around it kills it at 10 minutes and returns exit 124. In a set -euo pipefail environment it's recorded as failed. It's the minimum defense against creating workers that hang for a long time.

4. Always separate parallel workers with git worktree add

Don't run multiple workers in the same worktree. Separate worktrees with git worktree add <path> <branch> and launch the script from each worktree. Because the status file's Worktree field is recorded per worktree, you can trace afterward "which worker wrote what to which branch."

5. After a failure, shelve with git stash before resubmitting

When the state is State: failed and git diff --stat shows changes, Codex made partial changes and failed. Submitting the next worker as is means overwriting on top of the previous half-broken changes. Shelve it with git stash -u, then append to the task file "the previous run exited non-zero. The previous changes have been shelved. Start by identifying the cause," and resubmit.

6. Ban "None" in Remaining Risks

Put Remaining Risksには必ず1件以上の懸念事項を書くこと。Noneは禁止。 in the task file. A handoff file with consecutive Nones is a sign that Codex is omitting risk descriptions. With concerns continually written down, later workers and reviewers inherit the points that deserve attention.

7. Put the date/time and task ID in the handoff file's filename

Naming like handoff-20260824-api-fix.md lets you trace multiple worker results chronologically. You can check the latest completion with ls -lt handoffs/, and combining it with grep -rl "State: failed" status/ gets you a list of failed tasks. Just making filenames meaningful makes post-processing scripts far easier to write.

8. Wire status file polling to a Slack notification

while true; do
  if grep -q "State: completed\|State: failed" "$STATUS_FILE"; then
    state=$(grep "State:" "$STATUS_FILE" | awk '{print $2}')
    curl -s -X POST "$SLACK_WEBHOOK" \
      -H "Content-Type: application/json" \
      -d "{\"text\": \"Worker ${state}: $(basename $STATUS_FILE)\"}"
    break
  fi
  sleep 15
done

Run Codex workers on a late-night cron and receive completion or failure over Slack. When you wake up, just reading the handoff file tells you the results of the overnight batch. With this setup I've actually received "3 tasks completed while I slept, 1 failed" two to three times a week.

9. Separate, at design time, what Codex writes from what the shell records

Codex writes: Summary / Files Changed / Validation / Remaining Risks (the task's content)
The shell writes: State / timestamp / Branch / Worktree / git status --short (measured values at execution time)

This separation is condensed into the prompt's single line, Do not write handoff or status files yourself; the launcher manages those artifacts. Break the boundary and Codex's guesses ("probably the main branch") get mixed with the shell's measurements (the result of git rev-parse --abbrev-ref HEAD), and cross-check verification falls apart.

10. Manage prompt changes as git diffs of the script itself

The cat > "$prompt_file" <<EOF ... EOF inside the script is embedded in the script body. Every time you change a prompt rule, a diff shows up in git diff scripts/orchestrate-codex-worker.sh. That's actually an advantage: git log scripts/orchestrate-codex-worker.sh lets you trace the prompt's change history. Write "why I added the rule banning None in Remaining Risks" in the commit message, and months later when you read the script, the intent is clear.

11. Prepare to add stderr capture later

The current script doesn't capture stderr. The cause of a failure vanishes the instant it hits the terminal. It's solved just by routing stderr to a separate file with codex exec ... 2>"$error_file" and adding a line that runs cat "$error_file" into the handoff file on failure. Making this change after the script has stabilized dramatically shortens failure-investigation time.

12. Templatize task files instead of making them disposable

If you repeat the same type of work (e.g. "add validation to a new API endpoint"), templatize the task file. Create tasks/templates/add-validation.md and swap in just the target filename with sed before the call. Because the instructions to Codex stay consistent, the handoff file's structure stays stable too, and grep and post-processing scripts are less likely to break.

Summary

The reasons delegation to Codex falls apart are almost always fixed. You believe the output "completed" and skip the cross-check that follows — that's it. What orchestrate-codex-worker.sh does in 108 lines is a design that structurally refuses to allow that omission.

Don't hide errors, with set -euo pipefail. Separate the three roles of task/handoff/status files to divide the concerns. Branch name, worktree path, git status — values that need to be measured are written by the shell, not by Codex. Force four sections to demand a format from Codex's output. Prevent ANSI code contamination at the entrance with --color never. Reclaim temp files no matter which path exits, with trap cleanup EXIT.

Each of these design decisions is unremarkable on its own. Combined, they form a loop where you can confirm "whether what Codex said was actually done" in 30 seconds.

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Editing the plist Changed Nothing: launchd Kept Running My Old 30s Timeout Until 31 Lines of Bash Fixed It

Lily — Sat, 29 Aug 2026 11:00:07 +0000

I'm Lily. I made ¥100k/month in college, pushed it to ¥600k/month juggling multiple side gigs, lost all of it when I was laid off for reasons that had nothing to do with me, spent six months rebuilding an autonomous Claude Code environment, and now run ¥1.2M/month in revenue.

Here's the concrete before/after this article is about: I changed a timeout from 30 to 45 in a plist, and launchd kept running the job at 30 — for as long as the job stayed loaded. The fix is a 31-line shell script that compares the two values and re-syncs only when the job is idle.

Why this mechanism matters

launchd "freezes" environment variables at load time

Say you're running automated social-media likes across several lanes. You want to stretch one lane's timeout from 30 seconds to 45. You open ~/Library/LaunchAgents/com.lily.autolike.lane1.plist in an editor, change AUTOLIKE_TIMEOUT_SEC under EnvironmentVariables from 30 to 45, save, and close the window thinking "that should take effect on the next run."

The next morning, the logs show timeout errors at exactly the same rate as yesterday.

This comes from a fundamental property of how launchd works. macOS launchd reads the plist at the moment it bootstraps a job, and freezes the environment variables into that job's definition. Rewriting the plist file afterwards changes nothing about the environment variables of a job already loaded in memory. A silent divergence opens up between the plist on the filesystem and the execution definition launchd is holding.

You can confirm that divergence by running launchctl print gui/$(id -u)/com.lily.autolike.lane1. If the output still contains a line reading AUTOLIKE_TIMEOUT_SEC => 30, launchd is still operating on the old value. Even though the plist file says 45, the job keeps running with 30.

The fix is simple: unload the job with launchctl bootout and load it again with launchctl bootstrap. But that's exactly where the real problem starts.

Killing a running job destroys that run's results

Think about how much work the auto-like script does in a single run: authenticating to the target account, fetching the feed, liking items one by one, writing out the log when it's done — all of that in order, as one "run." If you fire launchctl bootout in the middle of it, the process is force-killed and every like that run performed is lost. The comment at the top of the script spells this out:

# 走行中のジョブを bootout するとその run のいいねが丸ごと消えるため、必ず待つ。

("If you bootout a job while it's running, that run's likes are wiped out entirely — so always wait.")

In an environment where multiple lanes run 24 hours a day, this accumulates. On an automation platform backing ¥1.2M/month in revenue, "I killed one job" is not a joke. Especially late at night when several lanes are running at once, a script that issues bootout without checking anything is a landmine that shreds your own output.

But "change the setting, then manually confirm the job is idle before reloading" isn't an autonomous environment either. Check whether it's stopped, and quietly re-sync only the lanes that are stopped — handing that judgment to the script is the design philosophy behind autolike-plist-reconcile.sh.

Building an "environment," not doing "work"

Rebuilding from zero income taught me something. There are two kinds of people making money on the side: people who do the work, and people who build an environment that keeps doing the work.

The first kind logs in every day and moves their hands; when their hands stop, the income stops. The second kind sets up the environment once and owns a system that keeps running whether they're there or not. Spending six months building an autonomous Claude Code environment came from exactly that idea. Keep multiple automation jobs resident under launchd, and when you change a setting the environment catches up on its own — having a mechanism where the config files and the running state stay in agreement without human intervention is the core of a ¥1.2M/month autonomous platform.

autolike-plist-reconcile.sh is a small, 31-line script, but the "build an environment" mindset is condensed into it. Instead of reloading by hand, the script periodically detects the divergence and repairs it automatically, aiming only at safe moments. That is an environment where "a human has to step in every time a setting changes" no longer holds.

The overall flow

Big picture

The script processes com.lily.autolike.*.plist under ~/Library/LaunchAgents/ one file at a time. It skips files with .bak-* and .disabled* suffixes, and for the remaining files it compares the configured value in the plist against the running value in launchd.

~/Library/LaunchAgents/com.lily.autolike.*.plist
  (excluding .bak-* / .disabled*)
          │
          │ ① read the configured value from the plist with PlistBuddy
          ▼
      want=45  ← the target value written in the plist
          │
          │ ② read the value inside launchd with launchctl print
          ▼
      have=30  ← the value launchd is actually holding
          │
          ├─ want == have ──────────── skip (no change)
          │
          └─ want != have
                │
                │ ③ check the PID with launchctl list
                │
                ├─ pid present (running)
                │     └── log a "deferred" line → wait until the next run
                │
                └─ pid absent (stopped)
                      ├── unload with launchctl bootout
                      ├── reload with launchctl bootstrap
                      └── record "reloaded 30 -> 45" in the log

① Read the target value from the plist (want)

PB=/usr/libexec/PlistBuddy
want=$($PB -c "Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC" "$P" 2>/dev/null) || continue

PlistBuddy is the standard macOS command for manipulating plists. With -c "Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC" you can reference a nested key directly. 2>/dev/null throws away errors, and || continue skips plists that don't have that key. This want is "the value that, according to the plist file, ought to be in effect."

② Read the value launchd actually holds (have)

D="gui/$(id -u)"
have=$(launchctl print "$D/$L" 2>/dev/null | sed -n 's/.*AUTOLIKE_TIMEOUT_SEC => \([0-9]*\).*/\1/p' | head -1)
[ -n "$have" ] || continue

launchctl print gui/501/com.lily.autolike.lane1 (501 being the UID) dumps that job's complete information in text form. From that, sed looks for the AUTOLIKE_TIMEOUT_SEC => 30 pattern and extracts only the numeric part (30). head -1 prevents multiple matches.

If have comes back empty (the job itself isn't loaded), || continue skips it. Attempting bootout against a job that isn't loaded makes launchctl return an error. This have is "the value that currently exists in launchd's memory and is actually running." The key point is that it reads launchd's internal state directly rather than the filesystem — the runtime reality you could never discover by reading the plist alone becomes visible for the first time here.

③ Comparing values and checking the PID

[ "$want" = "$have" ] && continue
pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then
  echo "[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)" >>"$LOG"
  continue
fi

If want == have, no change is needed, so it skips. If there is a difference, it next checks that job's current PID with launchctl list. The output of launchctl list has three columns — PID, LastStatus, Label — and awk pulls the PID from the row whose label matches. Unless the PID is - (launchd's conventional notation for "stopped"), the job is considered running and "defer" is chosen.

The log records lines in this form:

[2026-08-23 08:30:00] com.lily.autolike.lane1 実行中(pid=12345) のため見送り (30 -> 45)

("deferred because it's running (pid=12345)")

So the intent — "I wanted to change this, but I'm waiting for the right timing" — is still legible the next time you check. Since this script is registered with launchd as a periodic job, even a deferred lane gets checked again at the next launch, and re-syncs automatically as soon as there's an opening.

④ Re-syncing with bootout → bootstrap

launchctl bootout "$D/$L" 2>/dev/null
if launchctl bootstrap "$D" "$P" 2>/dev/null; then
  echo "[$(date '+%F %T')] $L reloaded $have -> $want" >>"$LOG"; changed=$((changed+1))
else
  echo "[$(date '+%F %T')] $L RELOAD FAILED" >>"$LOG"
fi

Only when the job is confirmed stopped does it run bootout → bootstrap. bootout unloads it once, and bootstrap re-reads the plist and loads it again, which pulls the new value of AUTOLIKE_TIMEOUT_SEC into launchd's memory.

A successful log line looks like this:

[2026-08-23 08:30:05] com.lily.autolike.lane1 reloaded 30 -> 45

Putting the before and after values side by side makes "what changed" obvious at a glance. On failure, RELOAD FAILED is left behind and the next run retries. At the end of the script, it records a summary of how many re-syncs happened in that single run.

[ "$changed" -gt 0 ] && echo "[$(date '+%F %T')] done changed=$changed" >>"$LOG"

When changed=0 (all lanes already in agreement), it writes nothing. With this design, the log file only accumulates records "when something changed," which suppresses day-to-day noise. In an autonomous environment where you read across dozens of log files to understand state, noise-free logs translate directly into debugging speed.

Making PATH explicit is handled by a single line at the top of the script.

PATH=/usr/bin:/bin:/usr/sbin:/sbin; export PATH

A script running as a launchd job starts in a minimal execution environment that differs from a normal terminal session. Without an explicit PATH, path resolution for PlistBuddy and launchctl fails, and you sink into the classic swamp of "it works on my machine but not when launchd starts it." Together with set -uo pipefail, this structure blocks unintended behavior from the outset.

Implementation details

The three ways `set -uo pipefail` keeps things from breaking

The second line of the script is this:

set -uo pipefail

Without that one line, the script can fall into a state where it "appears to work but does nothing." -u makes references to undefined variables an error. Suppose you typo'd the variable name used to fetch $have somewhere. Without -u, the typo'd variable expands to an empty string, and $have in [ "$want" = "$have" ] becomes empty. "want is 45, have is an empty string" doesn't match, so the comparison is false and it proceeds to the PID check. Naturally that label isn't in launchctl list's output, so no PID is retrieved, [ -n "$pid" ] is false — meaning it misjudges the job as "stopped" and runs bootout → bootstrap. Sometimes nothing breaks and the re-sync just happens, but the possibility always remains that "something being empty" because of a typo'd variable throws off some other piece of logic. -u forces early discovery of this class of bug by making the shell stop with an error.

-o pipefail controls the exit code of the pipeline as a whole. Without -o pipefail, a pipeline's exit code is that of the last command. That is, even if the output of launchctl print "$D/$L" 2>/dev/null | sed -n '...' | head -1 is empty, $? is 0 as long as head -1 exits normally (exit code 0). Since this script doesn't use set -e, that isn't an immediate problem — but because the presence or absence of pipefail can change behavior when you modify this script in the future, it's safer to state it explicitly from the start.

Writing plain -uo without the -o has the same effect, but writing set -uo pipefail explicitly conveys the intent that "pipefail was configured deliberately."

Exclusion design with glob + case

for P in "$HOME"/Library/LaunchAgents/com.lily.autolike.*.plist; do
  case "$P" in *.bak-*|*.disabled*) continue ;; esac

After narrowing target files with a glob pattern, there's a second stage that excludes with case. The reason for using case rather than find options or negated glob syntax is clear. Shell negated globs (extglob forms like !(*.bak*)) are poorly portable, and bash and zsh differ in the options needed to enable them. Scripts launched from launchd run under /bin/bash, so unless you explicitly enable extglob with set, you can't use it. Also, when exclusion patterns grow, case only needs another |, which keeps it readable.

The .bak- suffix becomes a problem because text editors sometimes leave automatically created backup files inside the LaunchAgents directory. A file like com.lily.autolike.lane1.plist.bak-20260823 doesn't match *.plist, but depending on the editor, some create the backup under a different name next to the .plist rather than in .plist.bak form. .disabled is a convention for renaming a job when you want to disable it manually; including this exclusion lets you express "this file isn't in use right now" without resorting to launchctl unload.

Exactly how the sed extraction works

have=$(launchctl print "$D/$L" 2>/dev/null \
  | sed -n 's/.*AUTOLIKE_TIMEOUT_SEC => \([0-9]*\).*/\1/p' \
  | head -1)

In the output of launchctl print gui/501/com.lily.autolike.lane1, the part carrying environment variables looks like this (501 being the actual UID):

environment = {
    AUTOLIKE_TIMEOUT_SEC => 30
    HOME => /Users/...
}

The sed expression s/.*AUTOLIKE_TIMEOUT_SEC => $[0-9]*$.*/\1/p targets the whole line, swallows everything before and after the key name with .*, captures only the numeric part into a capture group with $[0-9]*$, and prints it as \1. The combination of the -n flag and p means "print only matching lines," so nothing is printed if the environment variable doesn't exist.

Narrowing to the first line with head -1 prevents multiple matches in case a job name or another setting happens to contain a string resembling AUTOLIKE_TIMEOUT_SEC =>. It can't really happen in practice, but defensively taking a single line eliminates the situation where "multiple lines come back, $have becomes multi-line, and every subsequent comparison fails."

PID extraction with awk, and what "-" means

pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then

The output of launchctl list has three columns.

PID     Status  Label
12345   0       com.lily.autolike.lane1
-       0       com.lily.autolike.lane2

The first column is the PID: a number when the job is currently running, and - when it's stopped (waiting for the next StartInterval). awk -v l="$L" '$3==l{print $1}' prints the first column of the row whose third column matches the label.

Testing with only [ -n "$pid" ] is not enough. -n tests "not an empty string," so the string - is non-empty and evaluates to true. In other words, you get a state where "a stopped job with PID - is misjudged as running and deferred forever." The additional [ "$pid" != "-" ] check prevents that misjudgment.

The idempotence carried by `2>/dev/null` on bootout

launchctl bootout "$D/$L" 2>/dev/null
if launchctl bootstrap "$D" "$P" 2>/dev/null; then

The bootout line has neither || exit 1 nor || continue. The design throws away errors with 2>/dev/null, ignores the exit code, and moves on to bootstrap.

There's a reason for this. bootout may be called when the target job is already unloaded (if another script booted it out first, or right after the job crashed). In that case launchctl bootout returns an error, but the goal is "getting the job into an unloaded state," and if it's already unloaded, that's equivalent to success. Stopping on the error means the following bootstrap never runs and the whole re-sync fails. Discarding errors with 2>/dev/null and unconditionally proceeding to bootstrap secures idempotence.

bootstrap, on the other hand, has its success checked with if. A bootstrap failure — a syntax error in the plist file, insufficient permissions — means the abnormal state "the re-sync failed," so the design leaves RELOAD FAILED in the log and retries on the next run.

The design philosophy of the changed counter and the logs

changed=0
# ... ループ内で changed=$((changed+1)) ...
[ "$changed" -gt 0 ] && echo "[$(date '+%F %T')] done changed=$changed" >>"$LOG"

When changed is 0, no summary line is printed. That's intentional. This script is registered as a periodic launchd job. All lanes matching on want == have is the normal state, and if every normal run left one line in the log, you'd have hundreds of lines within a few days. Designing it so "a record is left only when something is wrong" means that the moment you open the log file, only "the timestamp and content of the changes" enters your eyes.

"Deferred," meanwhile, is logged every time the loop comes around.

echo "[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)" >>"$LOG"

That's deliberate. When you're debugging "why hasn't my change taken effect," several consecutive deferral lines immediately tell you "there's a running job and it's waiting for an opening." If a reloaded line eventually appears, you can confirm it completed normally; if RELOAD FAILED arrives, you learn about the anomaly. Because log density translates directly into debugging speed, the design is neither "always verbose" nor "always silent" but "record only changes and waits."

Where I got stuck

Misjudging jobs with PID "-" as "running" and deferring forever

When I first wrote the script, the PID check looked like this:

pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ]; then
  echo "実行中のため見送り"
  continue
fi

The judgment is "if the PID isn't empty, it's running." I registered this and ran it for a day, and the log was nothing but an endless stream of "deferred."

[2026-08-10 01:00:00] com.lily.autolike.lane2 実行中(pid=-) のため見送り (30 -> 45)
[2026-08-10 02:00:00] com.lily.autolike.lane2 実行中(pid=-) のため見送り (30 -> 45)
[2026-08-10 03:00:00] com.lily.autolike.lane2 実行中(pid=-) のため見送り (30 -> 45)

Seeing the output pid=- is what tipped me off. For a stopped job, launchctl list doesn't leave the PID column empty — it puts in the single character -. Since - isn't an empty string, [ -n "$pid" ] is always true, and no matter how long you wait it keeps being judged "running."

The fix is a single added line.

if [ -n "$pid" ] && [ "$pid" != "-" ]; then

Now it judges "running" only when a PID exists and it isn't the - that indicates stopped. This is a pit you're guaranteed to fall into if you don't know launchd's conventional notation, and man launchctl only mentions it in passing. I only noticed after looking at actual launchctl list output with my own eyes.

PlistBuddy not found because PATH wasn't set, so everything got skipped

When I ran the script manually from the terminal, it worked perfectly. Register the same script with launchd for periodic execution, and the log file was never updated at all. Not only was the changed counter not moving, not even deferral lines were coming out.

Because of set -uo pipefail, it had to be stopping on an error. Adding exec >> /tmp/debug.log 2>&1 at the top of the script to capture debug output for the experiment produced this error:

/bin/bash: /usr/libexec/PlistBuddy: No such file or directory

PlistBuddy uses the absolute path /usr/libexec/PlistBuddy, yet it wasn't found — the cause was that despite line 14 specifying the absolute path with PB=/usr/libexec/PlistBuddy, the expansion of $PB was coming out empty for some reason. The -u option turned the empty-variable reference into an error, and the for loop terminated in a way closer to exit than continue.

Digging in, I found leftover traces before PB=/usr/libexec/PlistBuddy of an attempt to use PlistBuddy through a different variable (commented out, but actually causing a separate problem because of -u). The variable reference had been broken during cleanup — that's the precise cause.

But the essential problem I discovered in parallel was PATH. A shell script launched from launchd inherits none of the PATH set up by a normal terminal session (things like /opt/homebrew/bin added in .zshrc or .zprofile). Without explicitly setting PATH=/usr/bin:/bin:/usr/sbin:/sbin, the commands you use inside the script end up in the "works locally, doesn't work under launchd" state.

Since PlistBuddy lives in /usr/libexec/, the absolute-path form PB=/usr/libexec/PlistBuddy is fine, but launchctl, sed, awk, and date are all resolved via PATH. /usr/bin:/bin:/usr/sbin:/sbin is standard in launchd's minimal environment, but in some environments it can be even narrower. The explicit PATH on line 2 is written as "insurance against ever stepping on this problem twice."

PATH=/usr/bin:/bin:/usr/sbin:/sbin; export PATH

`launchctl print` output came back empty and every lane was skipped

One morning I hit a situation where have was empty for every lane, so the script skipped everything. Nothing was in the log, so the state was "exiting normally but skipping."

Running launchctl print gui/501/com.lily.autolike.lane1 directly in the terminal returned an error.

Could not find service "com.lily.autolike.lane1" in domain for port

For a job not loaded into launchd, launchctl print returns an error rather than an empty string. Since 2>/dev/null discards the error, have becomes empty and [ -n "$have" ] || continue skips it.

Tracing the cause back, I found that during some other configuration work the night before I had mistakenly run launchctl bootout against several lanes and reached morning having forgotten the bootstrap. When the jobs themselves don't exist, autolike-plist-reconcile.sh can do nothing. A feature for "raising an alert when it detects an unloaded job" is outside this script's scope.

That's a limit of the design, not a bug. This script's responsibility is "fixing divergence in the environment variables of loaded jobs," and recovery when a job doesn't exist is handled by a different mechanism. Clearly separating what the script does from what it doesn't cuts off the temptation for one routine to bloat and solve several problems in one piece of code.

Processing backup files flooded the error log

During a period when I was editing plists directly in a text editor, the editor was creating automatic backups inside ~/Library/LaunchAgents/, in a form like com.lily.autolike.lane1.plist~ (~ being the backup suffix some editors use). That file doesn't match the glob *.plist, so it was harmless — but a file left in the form com.lily.autolike.lane1.plist.bak-20260810 did match the glob.

Because the backup file also starts with com.lily.autolike. and ends with .plist, the script treats it as a processing target. Read the backup's want with PlistBuddy, read the job's have with launchctl print, and — if they naturally agree — skip. That flow is fine, but when the backup file's contents held an older value, the backup's want differed from the running job's have and an unnecessary bootout → bootstrap fired.

The exclusion pattern case "$P" in *.bak-*|*.disabled*) continue ;; esac was added from that experience. It's tuned to the editor's automatic backup naming convention with the .bak- pattern (with the hyphen, because plain .bak risks colliding with other cases). In an environment that has a backup strategy, crushing this kind of side effect in advance is directly tied to the stability of an autonomous environment.

Pitfalls

Here's a complete rundown of the traps I actually stepped on while wiring a launchd × plist re-sync script into an autonomous environment. On top of the four detailed above (misjudging PID -, unset PATH, empty launchctl print, backup files sneaking in), I've organized the patterns that keep catching you during the scaling phase.

Runtime environment and startup

launchd's PATH is only /usr/bin:/bin:/usr/sbin:/sbin. It works in the terminal but gives command not found when launched by launchd. Nothing starts until you write PATH=/usr/bin:/bin:/usr/sbin:/sbin; export PATH on the first line.
PlistBuddy cannot be resolved via PATH. Keep it in a variable as the absolute path /usr/libexec/PlistBuddy (PB=/usr/libexec/PlistBuddy). Trying to add it to PATH is pointless because /usr/libexec is outside the default PATH.
Forgetting to give the script exec permission. launchd executes the path specified in ProgramArguments, but if you forget chmod +x it dies instantly with Permission denied. Nothing is left in the log either, so diagnosis is slow.
Mixing plist Program and ProgramArguments[0]. If you specify Program without knowing that ProgramArguments[0] is then treated as argv[0], and write both, your own script — not /bin/bash — gets passed directly and is treated as a syntax error.

Label / filename mismatches

The plist's Label value doesn't match the filename. The awk -v l="$L" used to filter launchctl list takes its value from $L=$(basename "$P" .plist), so if the filename and Label don't match, no PID is retrieved, it always misjudges "stopped," and an unnecessary bootout fires.
Leaving test plists that the glob pattern catches. If you drop in something named com.lily.autolike.test.plist for verification, it's processed just like production. Unless you follow the convention of excluding it with a .disabled suffix, a test configuration can overwrite the production environment.
Uppercase letters or spaces sneaking into the Label. The third column of launchctl list prints the Label itself, so uppercase notation that differs from the filename makes the awk match fail. Standardize the naming convention so everything after com.lily.autolike. is all lowercase, dot-separated.

launchctl output format

The output format of launchctl print shifts subtly across macOS versions. With the macOS 14 → 15 upgrade, the indentation of environment = { changed and the sed pattern stopped matching. Because sed -n 's/.*AUTOLIKE_TIMEOUT_SEC => $[0-9]*$.*/\1/p' is designed so the .* swallows both sides, it tolerates most indentation changes — but if the number of spaces around the key name changes, the sed pattern needs revisiting.
launchctl list can mix in labels from other processes across domains. In an environment where multiple users are logged in, another user's jobs can appear in launchctl list. Starting label names with com.lily. makes real damage unlikely, but the premise is using exact matching ($3==l) so awk doesn't react to unexpected labels.
Getting the domain+label format passed to launchctl print wrong. gui/$(id -u)/com.lily.autolike.lane1 is correct, but writing user/$(id -u)/… points at a different domain. Making D="gui/$(id -u)" a variable managed in one place erases the risk of a typo propagating across the whole script.

Environment variables

Writing the monitored key name AUTOLIKE_TIMEOUT_SEC in multiple places in the script. It appears twice: the PlistBuddy command that fetches want (line 14) and the sed pattern that fetches have (line 15). To prevent the accident of fixing only one when you change the key name later, it's safer to put it in a variable and consolidate it in one place.
sed's $[0-9]*$ fails to match when a non-integer value is used. If AUTOLIKE_TIMEOUT_SEC is written with a unit, like 45s, [0-9]* comes out empty. The value types diverge between PlistBuddy's want and sed's have, they're always judged different as separate strings, and bootout fires every single time. Manage the value as a pure integer.
The EnvironmentVariables key doesn't exist in the plist. The || continue in $PB -c "Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC" "$P" 2>/dev/null || continue handles this case, but if you remove 2>/dev/null while debugging, a "Does Not Exist" error appears and reveals the cause. Since it's designed to stay silent, note that nothing is left in the log even when everything gets skipped.

Logging and diagnostics

The log directory doesn't exist. If the directory of LOG="$HOME/.claude/logs/autolike-plist-reconcile.log" (~/.claude/logs/) hasn't been created, echo … >>"$LOG" errors and the script dies from the write error rather than from set -uo pipefail's -u. Run mkdir -p ~/.claude/logs before registering the plist.
Mistaking a continuous changed=0 for "normal." It's normal if all lanes agree on want == have, but the case where have is empty (jobs not loaded) and everything is skipped also produces changed=0 with no log output. Periodically confirm through a separate channel — launchctl list | grep com.lily.autolike — that all lanes are loaded.
Leaving hourly deferral lines alone as "no problem." If a job never finishes for a long time, the plist change is never reflected, permanently. Unless you have a mechanism that alerts when deferrals continue for N hours, at least build the habit of eyeballing the log weekly.

Boundaries of the PID check

A PID is retrieved, but that process is already a zombie. macOS launchd reaps crashed processes quickly so it's rare in practice, but there's a window where a PID remains in launchctl list while the actual process no longer exists. In that case bootout succeeds but no trace of "stopping something that was running" is left in the log, which makes later debugging harder.
In an environment with only long-StartInterval jobs, it feels like "an opening never comes." A job with StartInterval 3600 (one hour) runs only once an hour, so the PID is - only for a short window. As long as the reconcile script itself runs periodically at a short interval (say, 5 minutes), it catches the next opportunity. Conversely, if that job's execution time is comparable to its StartInterval, it's effectively always running and an opportunity to re-sync never arrives.

Best practices

Here are the rules I organized from real operation, the ones where I thought "I should have done this from the start." I've written out the design philosophy packed into 31 lines in a reproducible form.

① Pin PATH on the first line

PATH=/usr/bin:/bin:/usr/sbin:/sbin; export PATH

The biggest trap is verifying in the terminal and assuming you're done. Execution via launchd carries only a minimal PATH environment and inherits nothing from .zshrc or .zprofile. Pinning PATH on the leading line structurally eliminates the "works locally, doesn't work under launchd" problem.

② Always write set -uo pipefail

set -uo pipefail

-u turns undefined variables into immediate errors, and pipefail propagates mid-pipeline failures into the exit code. Without this one line, a typo'd variable name or a failed intermediate command passes through "as if nothing happened," and processing continues in a wrong state. Write it without exception as the safety device of a shell script.

③ Put PlistBuddy in a variable as an absolute path

PB=/usr/libexec/PlistBuddy

Since it can't be resolved via PATH, keep it in a variable at the top of the script. Even when used in several places, unifying on a single variable reference means a one-line fix if the path ever changes.

④ Put the launchd domain specification in a variable

D="gui/$(id -u)"

Writing the gui/UID form inline every time raises typo risk. Consolidating it into one variable means launchctl print "$D/$L", launchctl bootout "$D/$L", and launchctl bootstrap "$D" "$P" all follow from a single change.

⑤ Manage exclusion patterns with case

case "$P" in *.bak-*|*.disabled*) continue ;; esac

Rather than relying on extglob or negated globs, excluding with case is the most portable design. When exclusion patterns grow, you just append with |. Matching the editor's backup naming convention with .bak- (with the hyphen) prevents mistaken matches on plain .bak.

⑥ Make "skip on failure" explicit when fetching want and have

want=$($PB -c "Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC" "$P" 2>/dev/null) || continue
have=$(launchctl print "$D/$L" 2>/dev/null | sed -n '...' | head -1)
[ -n "$have" ] || continue

A failed want fetch (a plist without the key) skips immediately via || continue. A failed have fetch (job not loaded) skips via the empty-string check. The reason the two failure patterns are handled by different techniques is that PlistBuddy returns a non-zero exit code on failure, whereas the launchctl print pipe can exit 0 as a whole.

⑦ Explicitly judge a PID of - as "stopped"

if [ -n "$pid" ] && [ "$pid" != "-" ]; then

launchctl list puts the string -, not a numeric PID, for stopped jobs. The -n test alone misjudges - as "has a PID" and defers forever. Always add [ "$pid" != "-" ]. Since it isn't spelled out in man launchctl, this is a pit you can't notice until you look at real output with your own eyes.

⑧ Make bootout idempotent; check success only for bootstrap

launchctl bootout "$D/$L" 2>/dev/null
if launchctl bootstrap "$D" "$P" 2>/dev/null; then
  …; changed=$((changed+1))
else
  echo "… RELOAD FAILED" >>"$LOG"
fi

bootout gets called even when the job is already unloaded. Stopping on the error means the following bootstrap never runs and the whole re-sync fails. Discard errors with 2>/dev/null and proceed unconditionally to bootstrap to secure idempotence. Log only bootstrap's success or failure and let the next run retry.

⑨ Log only changes and waits

# 見送り → 毎回記録する（デバッグのため）
echo "[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)" >>"$LOG"

# 変更なし → 記録しない
# 変更あり → 記録する
[ "$changed" -gt 0 ] && echo "[$(date '+%F %T')] done changed=$changed" >>"$LOG"

(The comments read: "deferred → record every time (for debugging)", "no change → don't record", "changed → record".)

Make fully normal runs silent. Keep deferrals, because they're needed to debug "I changed it but it isn't taking effect." reloaded and RELOAD FAILED are state changes, so always keep them. These three tiers let you judge "did anything happen" in one line the moment you open the log.

⑩ Narrow the script's responsibility to one thing

All this script does is "fix divergence in the environment variables of loaded jobs." It has no features for "load the job automatically if it isn't loaded," "send a Slack notification after repeated failures," or "compress and rotate logs." Narrowing responsibility to one thing makes it easier to test and easier to combine with other scripts. If you need alerts, split them into a separate script.

⑪ Manage the monitored key name in one place

The key name AUTOLIKE_TIMEOUT_SEC appears in two places: the PlistBuddy command and the sed pattern. In case you add or change monitored targets in the future, defining it as a variable near the top improves maintainability.

KEY=AUTOLIKE_TIMEOUT_SEC
want=$($PB -c "Print :EnvironmentVariables:$KEY" "$P" 2>/dev/null) || continue
have=$(launchctl print "$D/$L" 2>/dev/null | sed -n "s/.*$KEY => \\([0-9]*\\).*/\\1/p" | head -1)

The current 31-line script is self-contained as a single-file, single-key tool so it isn't variabilized, but if you monitor multiple environment variables you expand it into a loop.

⑫ Derive the periodic interval from the reflection delay you can tolerate

The StartInterval you use when registering the reconcile script itself with launchd becomes "the maximum wait between rewriting the plist and the environment variable actually taking effect." If you expect reflection within 5 minutes, register with StartInterval 300. However, because there are cases where a running job is deferred, the actual reflection delay is bounded by "the reconcile interval + the job's maximum execution time."

⑬ Don't point the log file at the same place as launchd's StandardOutPath

launchd's StandardOutPath and StandardErrorPath are stdout/stderr redirection targets specified in the plist. Manage the reconcile script's own application log (the ~/.claude/logs/autolike-plist-reconcile.log pointed at by the LOG variable) separately from those. If both point at the same file, launchd's system messages and the application log get mixed together and analysis becomes difficult.

⑭ Test-run with bash -x

When verifying behavior before registering with launchd, run it with bash -x so commands are traced.

bash -x ~/.claude/scripts/autolike-plist-reconcile.sh

Because each line's actual expanded values are printed with a + prefix, you can visually confirm whether variables expand as expected and whether the output of launchctl print is being passed properly to sed. Verifying behavior locally before registering with the plist heads off silent bugs (the state where nothing is logged and nothing happens).

⑮ End with an explicit exit 0

exit 0

Putting exit 0 at the end of the script guarantees a normal exit code back to launchd. Combined with settings like KeepAlive, it prevents cases where a non-zero exit steers the job into an unexpected restart loop. If the last command is of the form [ "$changed" -gt 0 ] && …, the whole && returns false (exit code 1) when the condition is false, so the explicit exit 0 is necessary.

Summary

In one sentence, what autolike-plist-reconcile.sh does is: "periodically compare the values in launchd's memory against the plist on the filesystem, and quietly re-sync only when the job is stopped."

Concretely, four steps:

Read the plist's target value (want) with PlistBuddy
Read launchd's running value (have) with launchctl print
If there's a difference, check the PID from launchctl list
If the PID is - (stopped), re-sync with bootout → bootstrap; if the PID is a number (running), defer and just leave a log line

This "defer while running, wait for an opening" design lets you apply configuration changes without interrupting running automation jobs. You don't have to time the reload by hand — register the reconcile script itself with launchd and it's handled automatically at the next stopping point.

The point is the cleanliness of the design. All the 31-line script takes on is "fixing divergence in loaded jobs" — it doesn't handle job startup, alerts, or log rotation. One script, one responsibility is the basis of an autonomous environment. In an environment where multiple jobs run 24 hours a day, "build something that doesn't break" has a dramatically lower long-term maintenance cost than "notice it when it breaks."

A ¥1.2M/month autonomous platform is built out of an accumulation of these "unglamorous but reliably functional 31 lines." Not a single flashy feature. Understand launchd's specification precisely, crush the failure patterns first, design the logs carefully — repeating that is what creates an environment that keeps running without a human present.

One question for you: how long is your own tolerable reflection delay between "I edited a config file" and "the running job is actually using it" — and do you have anything closing that gap automatically today?

I've written up the full picture of the system, the breakdown of the ¥1.2M/month, and a 30-day walkthrough in a paid note.
📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

8 Pitfalls I Hit Auto-Repairing 'Plugin directory does not exist' — and the 3-Layer Guard That Stopped a 22 8 Wipeout

Lily — Sat, 29 Aug 2026 05:16:51 +0000

Liquid syntax error: Unknown tag 'endraw'

A launchd Job That Fires Once on Deprecation Day, Then Deletes Itself — and the 4 Pitfalls I Hit

Lily — Sat, 29 Aug 2026 00:00:04 +0000

This is a follow-up to my earlier post, "Automating a config migration with a one-shot launchd job." This time the trigger is an external event with a known end-of-life date (Fable 5 shutting down on 2026-07-07), and the question is how to design a launchd job you can set up today, have it fire only on that day, and have it remove itself once it's done.

Some deprecations come with a date stamped on them, and hand-editing a config on that exact day is the kind of chore you forget. But I also didn't want a script waking up every morning to rewrite the same JSON for no reason. What I landed on was a three-part set: a date gate, a backed-up jq rewrite, and a self-unload.

The problem: on the day I learn about a deprecation, I want to plant a job that runs only on the deprecation day

Right now ~/.claude/settings.json says this:

{
  "model": "claude-fable-5[1m]",
  ...
}

The moment I found out Fable 5 ends on 2026-07-07, putting a calendar reminder to hand-edit that "model" value felt too flimsy — I'd forget it. On the other hand, a daemon that checks the date on every launch is overkill. What I wanted was a job I could set once and stop thinking about, that fires when the day arrives and disappears afterward.

launchd can fire at specified times via StartCalendarInterval. But there's no way to express "exactly once at 9:00 on 7/7" — you only get recurrence or fixed date components. The standard macOS launchd move is to specify multiple slots and absorb the duplication with idempotency.

The implementation: the three-part set

Here's ~/.claude/scripts/model-transition-0707.sh in full (comments trimmed).

#!/bin/bash
set -uo pipefail
SETTINGS="$HOME/.claude/settings.json"
LOG="$HOME/.claude/logs/model-transition.log"
PLIST="$HOME/Library/LaunchAgents/com.shun.model-transition-0707.plist"

log() { echo "[$(date '+%F %T')] $*" >> "$LOG"; }

# ① 日付ゲート
if [ "$(date +%Y%m%d)" -lt 20260707 ]; then
  log "skip: before 2026-07-07"; exit 0
fi

# ② バックアップ付き jq 書き換え
current=$(jq -r '.model // empty' "$SETTINGS")
if echo "$current" | grep -qi 'fable'; then
  cp "$SETTINGS" "$SETTINGS.bak-model-transition"
  jq '.model = "opus"' "$SETTINGS" > "$SETTINGS.tmp" \
    && jq . "$SETTINGS.tmp" > /dev/null \
    && mv "$SETTINGS.tmp" "$SETTINGS"
  log "switched model: $current -> opus"
  /usr/bin/osascript -e \
    'display notification "Fable 5終了に伴いデフォルトモデルをOpusへ切替えました" with title "Claude model transition"' \
    >/dev/null 2>&1 || true
else
  log "no-op: model is already '$current'"
fi

# ③ 自己 unload
launchctl unload "$PLIST" 2>/dev/null || true
log "done (job unloaded)"

Let's walk through the three parts in order.

① A date gate to block early firings

if [ "$(date +%Y%m%d)" -lt 20260707 ]; then
  log "skip: before 2026-07-07"; exit 0
fi

date +%Y%m%d produces a numeric string you can compare as an integer. 20260706 < 20260707 → skip. That's all there is to it.

Why does this matter? Because the plist starts firing the instant you launchctl load it today. If the 6:50 AM slot comes around right after registration, that firing needs to be a no-op. Without the date gate, you'd get a misfire on the very day you plant the job: it would try to rewrite the model even though the value isn't fable yet.

Note
Numeric comparison with date +%Y%m%d works as-is under macOS's /bin/bash. -lt is an arithmetic comparison, so as long as the strings are the same length, lexicographic and integer ordering give the same result.

② A `jq` rewrite with a backup

cp "$SETTINGS" "$SETTINGS.bak-model-transition"
jq '.model = "opus"' "$SETTINGS" > "$SETTINGS.tmp" \
  && jq . "$SETTINGS.tmp" > /dev/null \
  && mv "$SETTINGS.tmp" "$SETTINGS"

This breaks into three steps.

Step	Purpose
`cp ... .bak-model-transition`	Keep the original as it was before the rewrite
`jq '.model = "opus"' > .tmp`	Write out to a temp file
`jq . .tmp > /dev/null`	Verify the generated JSON isn't corrupt
`mv .tmp settings.json`	Replace the original only after verification passes

If you write jq ... settings.json > settings.json directly, the original file is truncated to empty the moment the shell opens the redirect target. Going through a temp file is the basic pattern for avoiding redirect destruction. It also matters that the && chaining means mv never runs if verification fails.

The reason I test with grep -qi 'fable' — case-insensitive — is to cover "claude-fable-5[1m]" as well as any future variant spelling. Here's the value actually sitting in settings.json:

"model": "claude-fable-5[1m]"

After the rewrite it's just "opus" (an alias, not a model ID — this follows the "don't hardcode model IDs in scripts" policy from my CLAUDE.md).

③ Deleting itself after success

launchctl unload "$PLIST" 2>/dev/null || true
log "done (job unloaded)"

launchctl unload <plist> detaches that job from the daemon. The plist file itself stays on disk, so you can re-register it with launchctl load if you need to.

The 2>/dev/null || true is there so an already-unloaded state doesn't abort with an error. Combined with the idempotent design described below, it guarantees the script is safe no matter how many times it's called.

Warning
launchctl unload detaches the job immediately, even while it's running. That's exactly why the call sits at the end of the script — if you unload before finishing the rewrite, you cut yourself off mid-operation.

The plist: why three slots a day is fine

<key>StartCalendarInterval</key><array>
  <dict><key>Hour</key><integer>6</integer><key>Minute</key><integer>50</integer></dict>
  <dict><key>Hour</key><integer>12</integer><key>Minute</key><integer>50</integer></dict>
  <dict><key>Hour</key><integer>20</integer><key>Minute</key><integer>50</integer></dict>
</array>

Three slots: 6:50, 12:50, and 20:50. Why not just one? Because launchd skips slots that fall while the Mac is asleep. If I sleep through the morning slot, the midday or evening one can still pick it up.

The firing flow on 7/7 looks like this:

6:50  → 日付ゲート通過 → fable 検出 → opus に書き換え → unload → ジョブ消滅
12:50 → ジョブが存在しないので発火しない（unload済み）
20:50 → 同上

On 7/6 and earlier, each slot just leaves:

skip: before 2026-07-07

in the log and exits 0 immediately. No rewrite at all.

Drawn out, it looks like this:

7/5         7/6         7/7
6:50  skip  6:50  skip  6:50  書換+unload ←ここで終了
12:50 skip  12:50 skip  12:50 (消滅)
20:50 skip  20:50 skip  20:50 (消滅)

Idempotency is what makes "configure multiple slots and reject early firings with the date gate" work.

Pitfalls I hit

I'd written the date +%Y%m%d comparison with a string < → inside bash's [[ ]], that's lexicographic ordering, so I switched to -lt. With consistent 8-digit zero padding there's no actual harm, but use the arithmetic comparison that states the intent clearly.
I'd put the temp file in /tmp/ → when mv crosses filesystems, the rename can fail. Putting it in the same directory ($HOME/.claude/) guarantees the same fs.
I'd passed the label as the argument to launchctl unload → you have to pass the plist's full path, not the label (com.shun.model-transition-0707), or you get "No such process."
I'd only set StandardErrorPath → the script's log() writes to its own log file, but StandardErrorPath is still needed as the destination for output when the script itself dies on a syntax error.

Summary

A date gate, [ "$(date +%Y%m%d)" -lt YYYYMMDD ], turns every firing between setup day and the target date into a skip
A backed-up jq rewrite — "cp → jq > tmp → jq verify → mv" is the minimum safe four-step configuration
launchctl unload $PLIST after success detaches the job. The plist remains, so re-registering is possible
Multiple plist slots are sleep insurance. Idempotency is what makes over-specifying them safe

For anything with a fixed deprecation date, the best move is to plant it the day you find out and then forget about it. It's more reliable than a calendar entry, and easier to cancel than cron.

Next time I might write about how to read the logs this job leaves behind to confirm the migration succeeded — or, if it failed, the recovery procedure from the backup.

What deprecation date do you currently have sitting in a calendar reminder instead of in a script?

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

3 Token Burn From an Infinite Retry Loop: How I Made Claude Refill My Article Queue When It Runs Dry

Lily — Fri, 28 Aug 2026 11:00:07 +0000

Back when I was a college student earning ¥100,000 a month, one finished article meant my week was over. Today launchd stocks one every morning at 8:00, and the series that underpins ¥1.2M in monthly revenue hasn't gone dark once. The difference isn't talent or discipline — it's that I built, exactly one time, an environment that refills its own topic queue.

Why this setup works

When you try to mass-produce articles on your own, you almost always hit the same wall: when the topics run out, everything stops. You can block time on the calendar for topic brainstorming, but if you're not in the mood that day, you skip it. Any design where a human is the bottleneck will jam somewhere, guaranteed.

The strategy I took was to hand the work over to the environment. Not the work of writing articles — instead I assembled, one time only, a "detect that topics ran out and refill them" mechanism plus a "convert topics into articles" mechanism, then left them alone. The only action a human performs is the last one: checking the finished article and publishing it to note or Zenn.

The article-daily-stock.sh script introduced in this article is the core of that. There are three key points in the design.

Key point 1: Decouple stocking from deployment

As the comment at the top of the script says, the design is such that "even if the Zenn deploy (deploy-next) jams, this part does not stop" (lines 1–11 of the actual code). Success or failure of generation is judged solely by "was a stock file written to ~/content/article?" The job that publishes articles (zenn-daily) runs in a separate process, and the two are independent of each other. A structure where one job failing doesn't cause a chain stop is what supports stability over long-term operation.

Key point 2: Use the queue as a buffer

Topics are stacked as an array of objects in a JSON file at ~/zenn-articles/.topic-queue.json. The current queue has 16 topics waiting. Every morning at 8:00 it pulls the first one off, converts it into an article, and when done moves that topic to done-queue. As long as the queue has entries left, the 8 AM job can jump straight into the real work.

The problem is the moment the queue bottoms out. Traditionally that would end with "zero articles today." But I wanted to avoid that. Because once you break the streak for even one day, the "I don't have to write today either" collapse of the habit starts.

Key point 3: When it's empty, Claude invents the topic itself

When the queue goes empty, the script calls neither an external service nor an API — it calls claude -p running locally, has it auto-plan exactly one next article topic, and inserts it at the head of the queue. The basis it uses is "that day's actual work." Daily briefs, memory files, the automation scripts themselves — Claude finds "things that could become a technical article" from the concrete artifacts piling up every day. Presenting real, existing files and paths as evidence rather than fabricating them is hardcoded into the instructions.

If the generated JSON doesn't pass required-field validation, the script exits without writing the done-marker. launchd's catch-up slot (10:35) automatically re-runs the same script, so "refill failure → retry at the next slot" is part of one and the same mechanism.

How the whole thing flows

ASCII diagram

launchd com.shun.article-daily
├─ 8:00  StartCalendarInterval
└─ 10:35 StartCalendarInterval（キャッチアップ）
         │
         ▼
claude-quota-guard.py  ← Claude Maxトークン枯渇時に即abort
         │
         ▼
run-and-notify.sh      ← 完了/失敗をDiscordへ通知
         │
         ▼
article-daily-stock.sh apply
         │
         ├─ [0] done-marker 確認（当日生成済み？）
         │       YES → audit のみ実行して exit 0
         │
         ├─ [1] 全ストックを audit + 自己修復
         │       サムネ欠落 → gen_note_thumbs.py で再生成
         │       本文不正   → done-queue からネタを取り戻してキュー再投入
         │
         ├─ [2] jq 'length' .topic-queue.json
         │       │
         │       ├─ >= 1 → [4] キュー先頭取得 へ
         │       │
         │       └─ == 0 → [3] 自動立案モード
         │                     │
         │                     ▼
         │                 BRIEF_LATEST（当日のdaily brief md）
         │                 + ~/.claude/memory/ 
         │                 + ~/.claude/scripts/ 実ファイル
         │                     │
         │                     ▼
         │                 claude -p REPLENISH_PROMPT
         │                 --model sonnet --effort high
         │                 --max-turns 20
         │                     │
         │                 JSON検証（slug / title / sources / thumb_title）
         │                     ├─ NG → done-marker 書かず exit 0
         │                     │        ↑ 10:35スロットが拾う
         │                     └─ OK → queue 先頭へ insert
         │
         ├─ [4] キュー先頭 → SLUG / TITLE / EMOJI / NO を取得
         │
         ├─ [5] claude -p で記事執筆（--max-turns 40, 最大1500秒）
         │
         ├─ [6] 検証（title有無・70字以内・1200bytes以上・スタブ語なし）
         │       NG → ファイル破棄 / done-marker 書かず exit 0
         │
         ├─ [7] content/article/articles/ へストック
         ├─ [8] gen_note_thumbs.py でサムネ生成 → thumbnails/
         ├─ [9] coverage.json を upsert
         ├─[10] manifest を ready 化
         ├─[11] queue pop → done-queue へ移動
         └─[12] git push（best-effort・失敗しても done-marker は立つ）

The plist design: why there are two slots

If you look at StartCalendarInterval in ~/Library/LaunchAgents/com.shun.article-daily.plist, the fire times are two slots: 8:00 and 10:35.

<key>StartCalendarInterval</key>
<array>
    <dict>
        <key>Hour</key><integer>8</integer>
        <key>Minute</key><integer>0</integer>
    </dict>
    <dict>
        <key>Hour</key><integer>10</integer>
        <key>Minute</key><integer>35</integer>
    </dict>
</array>

The first (8:00) is the production slot, the second (10:35) is the catch-up slot. Thanks to the done-marker check at the top of the script, if the day's article has already been generated it "runs only audit + self-repair and exits immediately" (lines 51–53 of the actual code).

SKIP_GEN=0
[ "$MODE" = "apply" ] && [ -f "$DONE_MARKER" ] && SKIP_GEN=1

The point of this design is that it forms a double safety valve: "8:00 succeeds → 10:35 ends with audit only" and "8:00 fails → 10:35 takes over as production." The marker file name uses a date-suffix format, ~/.claude/logs/.article-daily-done-YYYYMMDD, and is auto-deleted after 7 days (line 525 of the actual code).

The actual code, from empty-queue detection through refill

Lines 286–346 of the script are the implementation of automatic topic planning. Let me walk through the key parts.

Step 1: Detect empty

QLEN=$(jq 'length' "$QUEUE" 2>/dev/null || echo 0)
if [ "$QLEN" -eq 0 ]; then
  log "queue 空 → 実作業からネタ自動立案"

jq 'length' gets the number of elements in the array. If jq isn't found or there's a syntax error, it falls back to echo 0, so even on error it safely enters topic-planning mode.

Step 2: Gather context

BRIEF_LATEST=$(ls -t "$HOME/Desktop"/*brief* \
  "$HOME/Documents/claude-obsidian/wiki/briefs/daily/"*.md \
  2>/dev/null | head -1)

To ground everything in "that day's actual work," it looks for the day's daily brief in newest-mtime order. With ls -t priority it scans both the brief on the Desktop and the ones inside the Obsidian vault, and uses only the single newest file.

Step 3: The prompt to Claude

The important part inside the prompt (REPLENISH_PROMPT) is the constraints block.

# 制約
- 既出slugは禁止（重複ネタNG）: ${USED}
- 技術記事として1本で完結する具体的な工夫であること（粒度: 1スクリプト/1仕組み）
- 根拠ファイルは必ず実在パスで2〜4個挙げる（~ 表記）

USED holds the list of already-used slugs collected from the existing queue, done-queue, the articles directory, and coverage.json (generated by the used_slugs() function at lines 113–119). If a slug identical to a past article is generated, it gets rejected by the subsequent duplicate check.

The Claude invocation is this code:

TOPIC_JSON=$(run_to 600 "$CLAUDE" -p "$REPLENISH_PROMPT" \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  --model "${ARTICLE_MODEL:-sonnet}" --effort high \
  --output-format text --allowedTools "Read,Grep,Glob,Bash" --max-turns 20 2>>"$LOG")

run_to 600 is a wrapper around gtimeout 600, imposing a maximum 10-minute timeout. --strict-mcp-config --mcp-config '{"mcpServers":{}}' completely disables MCP servers, allowing only the standard Claude Code tools (Read/Grep/Glob/Bash). The model can be overridden with the ARTICLE_MODEL environment variable; when unset it uses sonnet.

Step 4: Strip the JSON fence and validate

TOPIC_JSON=$(printf '%s' "$TOPIC_JSON" | sed -n '/{/,/}/p')
if ! echo "$TOPIC_JSON" | jq -e \
  '.slug and .title and (.sources|length>0) and (.thumb_title|length>0)' \
  >/dev/null 2>&1; then
  log "ABORT: ネタ自動立案に失敗（JSON不正）。marker無しで次スロット再試行"
  notify "ネタ自動立案に失敗。次スロットで再試行。"
  exit 0
fi

Anticipating that Claude will attach code fences or a preamble, sed -n '/{/,/}/p' cuts out everything from the first { to the last }. Validation confirms that the four fields slug, title, sources (at least one), and thumb_title (at least one) exist. On validation failure it exits with exit 0 — but the crucial thing is that it does not write the done-marker. Because there's no done-marker, the 10:35 catch-up slot runs the same script again.

Step 5: Overwrite no and prev_slug on the machine side before inserting

TOPIC_JSON=$(echo "$TOPIC_JSON" | jq -c \
  --arg no "$NO_NEW" --arg prev "$PREV_NEW" \
  '.no=$no | .prev_slug=$prev')
jq --argjson t "$TOPIC_JSON" '[$t] + .' "$QUEUE" > "$QUEUE.tmp" && mv "$QUEUE.tmp" "$QUEUE"
log "ネタ追加: $NEW_SLUG (no=$NO_NEW)"

Because there's a risk the model mixes up no (article number) and prev_slug (the previous article's slug), the shell overwrites them with definitive values after generation. jq '[$t] + .' inserts at the head of the queue, so this topic gets used immediately at the next 8:00 or 10:35 slot. The >.tmp && mv pattern rewrites atomically to prevent the JSON from being corrupted if the script is interrupted mid-write.

The plist's wrapper structure

Looking at ProgramArguments in the plist, it doesn't call the script directly — it goes through two layers of wrappers.

<array>
  <string>~/.claude/scripts/claude-quota-guard.py</string>
  <string>--job</string>
  <string>com.shun.article-daily</string>
  <string>--</string>
  <string>/bin/bash</string>
  <string>~/.discord/run-and-notify.sh</string>
  <string>zenn</string>
  <string>Zenn記事ストック生成</string>
  <string>/bin/bash</string>
  <string>~/.claude/scripts/article-daily-stock.sh</string>
  <string>apply</string>
</array>

claude-quota-guard.py checks in advance whether the Claude Max plan's tokens are exhausted. If the remaining budget is in a critical state (token-budget-advisor.sh --short returns 🔴 or critical), it aborts immediately without launching the script body (lines 94–98 of the actual code). This prevents the situation of "hammering Claude calls while the token budget is bottomed out and burning it for nothing."

run-and-notify.sh is a wrapper that notifies Discord of the result. The arguments "zenn" and "Zenn記事ストック生成" become the channel and the notification title. The script body itself holds no notification logic, and the result arrives in Discord whether it succeeded or failed.

Let me also record the plist settings themselves. Because LowPriorityIO: true and Nice: 10 are set, the Mac doesn't get sluggish to operate while article generation is running. Combined with ProcessType: Background, even during the heavy load right after waking from sleep the OS defers I/O and prioritizes the foreground task. Since RunAtLoad: false, merely loading the plist doesn't start it — it fires only at the specified times.

The details that hold the implementation together

In the first half I traced the core logic of queue refilling. The actual script (528 lines) stacks up a number of supporting design details. Let me take up, one at a time, the places where "why it's written that way" is hard to see.

Solving the launchd PATH problem dynamically

# article-daily-stock.sh 56-58行目
NODE_BIN=$(ls -d "$HOME"/.nvm/versions/node/*/bin 2>/dev/null | sort -V | tail -1)
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"
[ -n "$NODE_BIN" ] && export PATH="${NODE_BIN}:$PATH"

A process launched by launchd reads neither .zshrc nor .bash_profile. Even if the claude command is installed at ~/.nvm/versions/node/v24.13.0/bin/, it isn't in launchd's bare PATH. Despite passing PATH explicitly via the plist's EnvironmentVariables (line 8 of the plist), the script also stacks the NVM bin at the front again. The reason is to avoid pinning a version. A hardcoded /v24.13.0/bin in the plist would need to be manually rewritten every time node is upgraded. By dynamically fetching the latest version's bin with sort -V | tail -1, the design means you never have to touch the plist.

Eradicating "interruption by sleep" with caffeinate

# article-daily-stock.sh 61-63行目
if [ -z "${CAFFEINATED:-}" ]; then
  exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash "$0" "$@"
fi

Article generation, counting the two Claude calls (topic planning + article writing), takes up to 25 minutes. If the Mac sleeps in the middle of that, the claude -p HTTP connection drops and it's treated as a timeout. caffeinate -i suppresses idle sleep and -s suppresses system sleep, and exec replaces the current process, building a structure where "the original script keeps running under caffeinate's umbrella." The CAFFEINATED=1 check is to prevent infinite recursion. Since adding this one block, interruption logs after waking from sleep have been zero.

Preventing double execution with a mkdir-based atomic lock

# article-daily-stock.sh 66-75行目
LOCKDIR="$HOME/.claude/locks/article-daily.lock"
if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then
  oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true)
  if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then
    log "別インスタンス実行中(pid=$oldpid) — skip"; exit 0
  fi
  rm -rf "$LOCKDIR"; /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0
fi
echo $$ > "$LOCKDIR/pid"
trap 'rm -rf "$LOCKDIR"' EXIT INT TERM

When the 8:00 slot's generation runs long, the 10:35 slot launches the same script. The reason I adopted mkdir as the locking mechanism rather than a file lock (flock) is that macOS's mkdir is a POSIX-guaranteed atomic operation. Even if two processes call mkdir simultaneously, only one succeeds. Furthermore, even when a lock directory has been left behind, it applies kill -0 to the PID inside to check whether "the process is really alive," and if it's dead it cleans up the stale lock and continues. This is a "zombie PID check" that avoids the stale lock produced when the script is force-killed.

Getting the title character count accurately with inline Python

Zenn has a constraint that the title must be within 70 characters. The frontmatter_title_chars function (lines 140–165) that checks this is implemented as an inline Python script.

# 141-164行目のPythonインライン（要点抜粋）
if (in_frontmatter or idx < 20) and line.startswith("title:"):
    title = line.split(":", 1)[1].strip()
    if len(title) >= 2 and title[0] == title[-1] and title[0] in ("'", '"'):
        title = title[1:-1]
    print(len(title))
    sys.exit(0)

The reason for not using grep | wc -c is the quoting and multibyte problem. Zenn frontmatter title values are sometimes wrapped in single quotes and sometimes not. wc -c returns byte count, so it counts one Japanese character (3 bytes) as 3 characters. Using wc -m gives you a character count, but then you separately need to strip the quotes, which complicates the code. By designing it to parse the frontmatter in Python and return the len() of the pure string with quotes removed, both Japanese titles and quoted/unquoted forms can be measured accurately in one go.

Running path sanitization before the secret scan

# article-daily-stock.sh 431-446行目（順序が肝）
# ①先にサニタイズ
for _f in "$ART" "$ARTICLES/$SLUG.md"; do
  [ -f "$_f" ] || continue
  /usr/bin/sed -i '' -E 's#/Users/[A-Za-z0-9._-]+/#~/#g' "$_f"
done

# ②その後で秘密スキャン
if grep -nEi 'AKIA[0-9A-Z]{16}|(secret|api_key|...)[[:space:]]*[:=]...' "$ART" >/dev/null; then
  log "ABORT: 秘密らしき値混入 → 中止"
  rm -f "$ART"; exit 1
fi
# ③実ホームパスが残っていたら中止
if grep -q "/Users/" "$ART"; then
  log "ABORT: 実ホームパス(/Users/)混入 → 中止"
  rm -f "$ART"; exit 1
fi

Even when the generation prompt explicitly says "write it with ~ notation," Claude sometimes writes the real path (/Users/…) into the body. In that case the script doesn't ABORT outright — it first attempts mechanical sanitization. Since /Users/ never carries meaning relevant to a published article, replacing it with ~ doesn't break the meaning of the text. It ABORTs only if, after sanitization, an AWS key format (AKIA…) or a secret-assignment pattern still remains. The philosophy is "fix what can be fixed, stop only for what can't," minimizing unnecessary retries.

Why the done-marker is set before git push

# article-daily-stock.sh 509行目と524行目
# キューをpopした直後——git push の前——に立てる
touch "$DONE_MARKER"   # ← 511行目

run_to 120 git push -q 2>>"$LOG" || log "WARN: push失敗（ストックは確保済）"

# push後にも念のため（べき等な重複touchは無害）
touch "$DONE_MARKER"   # ← 524行目

The done-marker being touched twice is intentional. The first one happens before git push. If push fails and the script exits, then without a done-marker the 10:35 slot won't find one and will run article generation again. Regenerating with the same SLUG would overwrite the existing file, and the queue would get popped twice. The moment the file write to the stock completes is "generation complete"; git push is nothing more than best-effort post-processing. This ordering alone completely prevents duplicate-generation accidents caused by push failures.

Three pitfalls I got stuck on

When you're thinking about the design, "do it this way and it works" is clear in your head. But when you actually run it, you get stuck in places you never imagined. Here are three failures I actually hit, in symptom → cause → fix order.

Pitfall ①: Claude always wraps the JSON in a code fence

Symptom

Automatic topic planning ended in ABORT: JSON不正 every single time. Looking at the log, the contents of TOPIC_JSON were as follows.

json
{
"no": "17",
"slug": "caffeinate-wrapper",
...
}

shell

Cause

Even when the claude -p prompt explicitly says "no code fence, output only one JSON object," it wraps the output in three backticks and a json tag with fairly high probability. Passing that straight to jq -e blows up instantly with a parse error.

Fix

TOPIC_JSON=$(printf '%s' "$TOPIC_JSON" | sed -n '/{/,/}/p')

sed -n '/{/,/}/p' outputs "from the line where the first { appears to the line where the last } appears." It strips off the code fence, the preamble, and any trailing text, leaving only the JSON object portion. Even with multiple nested { inside the JSON, the final } is at the end, so it works correctly. Since adding this, ABORTs due to invalid JSON have been near zero. Lesson: never trust a "strictly follow this format" instruction to a model. Always write code that normalizes the output mechanically.

Pitfall ②: 8:00 and 10:35 overlapped and generated the same article twice

Symptom

The next morning, checking ~/content/article/articles/, there were two files with the same sequence number (17-caffeinate-wrapper.md) whose contents differed subtly. Two entries were gone from the queue as well, and done-queue had two entries with the same SLUG. Discord had also received the "complete" notification twice.

Cause

The generation that started at 8:00 took 25 minutes, and was still running when the 10:35 slot launched. In the early version, before the double-execution lock was added, both processes were reading the head of the queue in parallel. The 10:35 process read the queue before the 8:00 process popped it, and as a result both ran separate Claude calls with the same SLUG; the process that wrote its file later overwrote the existing file and left two done-queue entries.

Fix

I added the mkdir-based lock (lines 65–75). mkdir is a POSIX-guaranteed atomic operation, and when two processes call it simultaneously only one succeeds. On top of that, I set up the flow where the done-marker check (lines 51–53) sets SKIP_GEN=1, so that if the day's article has already been generated it runs only the audit and exits immediately. The duplication came to light after publishing, but looking at the generated_at timestamps in done-queue showed the two entries were 2 seconds apart, so identifying the cause was immediate.

Pitfall ③: A broken article kept getting pushed back onto the queue, causing an infinite loop

Symptom

A few days later, generation for the same SLUG was running every morning but all of them ended in ABORT: 生成本文が不完全. done-queue accumulated one entry with the same SLUG per day, and the article file was discarded every time. At the same time, the token consumption log was 3× the normal amount.

Cause

The audit_repair function (lines 230–277) is designed to "reclaim from done-queue any article judged to have an incomplete body and re-enqueue it" (line 261). But every time regeneration ran it ABORTed with the same breakage, and got re-enqueued again... entering a loop. The bump_attempt function was supposed to record the retry count in $ATTEMPTS_FILE and, once it exceeded MAX_ATTEMPTS=2, divert the item into the needhuman array — but the root cause was that $ATTEMPTS_FILE (a JSON file) had been left in a corrupted state by an interruption during a disk write, and was constantly returning n=0.

Fix

I changed bump_attempt's write to the atomic .tmp → mv pattern.

# article-daily-stock.sh 224-226行目
jq --arg s "$slug" --argjson n "$n" '.[$s]=$n' \
  "$ATTEMPTS_FILE" > "$ATTEMPTS_FILE.tmp" 2>>"$LOG" \
  && mv "$ATTEMPTS_FILE.tmp" "$ATTEMPTS_FILE"

I also added an initialization line at line 137, [ -f "$ATTEMPTS_FILE" ] || echo '{}' > "$ATTEMPTS_FILE", so that even if the file is corrupted it can restart from an empty object on the next launch. SLUGs exceeding MAX_ATTEMPTS=2 get stacked into the needhuman array, and writing to _NEEDS-FIX.txt plus a macOS notification (lines 267–268) run as a set. After this fix, the infinite loop disappeared completely.

All three pitfalls were either "timing problems invisible to unit tests" or "problems with how state files break." You can't find them by tracing the flow on paper. If you're building something similar, I recommend first running it manually in dry mode and following the logs of each step. Passing the argument bash article-daily-stock.sh dry means neither the queue pop nor the git push happens — it only writes the stock file and finishes (lines 498–500). Deliberately creating failure cases in this mode before wiring it to production and confirming the behavior for each of "corrupt the JSON / delete the article file / leave a lock file behind" is the shortest route.

Sticking points

In addition to the three covered in p2 — "JSON fences," "double execution," and "the infinite loop" — here is a comprehensive list of the points I got stuck on in actual operation.

① launchd does not expand ~

When I first started writing the plist, I put ~/.claude/scripts/article-daily-stock.sh in ProgramArguments, and launchd decided "the file does not exist" and exited immediately. launchd does not expand ~. The plist must be written with absolute paths of the form /Users/<username>/… (this is why every <string> in the actual plist is an absolute path). Inside the script you can reference home via the $HOME variable, but the values in the plist's ProgramArguments and EnvironmentVariables can only use absolute paths.

② On a Mac without gtimeout, run_to does nothing

The script uses GNU coreutils' gtimeout as a timeout wrapper (lines 82–84). If /opt/homebrew/bin/gtimeout doesn't exist, TIMEOUT_BIN="" and run_to() simply passes "$@" straight through. In an environment where brew install coreutils hasn't been done, the timeout in run_to 1500 claude -p … doesn't function and the Claude call keeps running indefinitely. Confirm with gtimeout --version before wiring things up.

③ I didn't add a network-connectivity wait after waking from sleep

When the MacBook is left asleep and 8:00 the next morning arrives, it wakes from sleep and the script launches — but Wi-Fi reconnection doesn't make it in time and claude -p was erroring out immediately. That's why lines 87–91 of the script contain a connectivity wait that loops nc -z -G 3 1.1.1.1 443 up to 18 times (90 seconds). Audit mode doesn't use the network, so it skips the loop.

④ I kept calling Claude while tokens were exhausted and melted the remaining budget

The Claude Max plan has both a 5-hour window and a 7-day window. When article generation continues into the latter half of the week, the 7-day window gets tight and claude -p starts returning rate limits partway through. At first I ignored the errors and kept retrying at the next slot, so I ended up in a loop of "it doesn't stop → it dies partway every time → it tries again," shaving the remaining budget down further. Now, at lines 93–98, if token-budget-advisor.sh --short returns 🔴 or critical, it exits without launching the job body. Checking the remaining token budget before a Claude call is mandatory preprocessing.

⑤ The stock article file paths broke sort order depending on the digit count of NO

The next_no() function (lines 122–130) reads the maximum number from the file names in ~/content/article/articles/ and does +1. Initially I created numbers as one digit — 1, 2, … — so ls sorting made 10 < 2 and the numbering went wrong. It's resolved by always zero-padding to two digits with printf '%02d'. If it ever goes past 99 articles, it will need to change to three digits.

⑥ article_ok()'s stub-word check produced a false positive

Line 179's grep -qiE 'request timed out|不明な商品|TODO: *本文|$生成失敗$' checks whether stub keywords are contained in the article body. On one occasion I wrote "an article introducing an implementation that detects timeouts via the request timed out error," and the string request timed out appeared in the body, so a correctly generated article got rejected with ABORT: 生成本文が不完全. The stub-word check patterns need to be narrowed as much as possible to "words unlikely to appear as example text in the body."

⑦ There was a period when I made a git push failure exit 1

In the initial implementation, if git push failed I made the whole script exit 1. Since everything counted as a failure regardless of whether the push failed due to "offline" or "conflict," the done-marker didn't get set and the 10:35 slot started regenerating the same article. Now, as at line 518, it's designed as || log "WARN: push失敗…", dropping the error to a warning log and continuing the script. It's important to think about generation completion and git push success as separate things.

⑧ coverage.json bloated and entries already deleted from the queue kept lingering

When I rewrote a few articles and their SLUG changed, the old entries stayed in coverage.json forever. There's a stale-cleanup block near the end of audit_repair() (lines 270–276), but I didn't have it at first, and around the point it exceeded 200 entries the jq processing got heavy. The correct form for cleaning coverage.json is to keep only "files that actually exist in the stock directory + slugs still in the queue" and delete everything else.

⑨ I forgot to initialize ATTEMPTS_FILE and got zombie retries

bump_attempt() records the retry count in ATTEMPTS_FILE (.article-repair-attempts.json). That initialization is done by [ -f "$ATTEMPTS_FILE" ] || echo '{}' > "$ATTEMPTS_FILE" (line 137), but the initial implementation didn't have it, so when the file didn't exist jq returned an error and n was always 0. As a result, MAX_ATTEMPTS=2 could never be exceeded, and the broken article kept getting re-enqueued every morning. Always guarantee the existence of state files at the top of the script.

⑩ I forgot the sips command is macOS-only and it failed in a Linux test environment

The thumb_ok() function (lines 185–188) checks the thumbnail's pixel width with sips -g pixelWidth. This command exists only on macOS, and on Linux or in a Docker container it's command not found. I only noticed when I tried to verify behavior in CI. Since the mechanism presupposes launchd there's no real harm, but if you try it in another environment you'll need a stub for this command.

⑪ The no field was passed as a string and zero-padding conversion failed

When passing the number obtained via NO=$(echo "$TOPIC" | jq -r '.no // ""') into printf '%02d' "$((10#$NO))", if no was an empty string or a string like "01", the shell's arithmetic evaluation sometimes failed to interpret 10#. That's why line 355 has the guard [ -z "$NO" ] || ! [[ "$NO" =~ ^[0-9]+$ ]] && NO=$(next_no) — "if something non-numeric arrives, re-number with next_no()."

Best practices

Here are the design principles verified in actual operation.

1. Keep generation and deployment independent

Don't mix writing to the stock and publishing to Zenn into the same job. Generation completion is judged solely by "was a file written to ~/content/article/articles/?", and git push is treated as best-effort post-processing (as the comment at the top of the script says). A design where generation doesn't stop even if deployment jams is the source of stability in long-term operation.

2. Set the done-marker before git push

"The moment the write to the stock completes" is generation completion. Line 509's touch "$DONE_MARKER" is placed before git push. This way, even if push fails, the catch-up slot doesn't do a duplicate generation. Line 524's second touch is a post-push belt-and-suspenders, but the first one is the linchpin protecting the whole design.

3. Protect state files with the .tmp && mv pattern

Every write to topic-queue.json, coverage.json, and ATTEMPTS_FILE is done with the atomic pattern jq … > "$FILE.tmp" && mv "$FILE.tmp" "$FILE". This prevents half-written JSON from being left behind on interruption and eradicates jq parse errors on the next launch. Careful: if you forget the && and write > file && mv, then even when jq fails you've already overwritten the file with an empty one.

4. Use a mkdir-based atomic lock

flock can't be used in environments where macOS's /bin/flock doesn't exist. /bin/mkdir is a POSIX-guaranteed atomic operation and works reliably on macOS. The pattern of writing the PID inside the lock directory, detecting zombie PIDs with kill -0, and auto-cleaning stale locks (lines 65–75) is the complete form of double-execution prevention.

5. Delegate retries to the OS (launchd)

Rather than retrying yourself with a sleep loop on failure, a design where you exit 0 without writing the done-marker and let the next slot pick it up is simpler. By setting two slots, 8:00 and 10:35, in the plist's StartCalendarInterval, the double safety valve of "production → failure → automatic retry 2 hours 35 minutes later" is completed at the OS level. The script-side code stays minimal.

6. Never trust the model's output — always normalize it

Even when you explicitly say "output only JSON," claude -p attaches code fences and a preamble. Always include the normalization that cuts from the first { to the last } with sed -n '/{/,/}/p' (line 332). Since adding this, invalid-JSON ABORTs have been near zero. Likewise, fields the model easily mixes up, such as no and prev_slug, get overwritten with definitive values on the shell side after generation (lines 342–343).

7. Attempt mechanical repair before ABORTing

Real home paths (/Users/…) creeping in happens even when the prompt instructs against it. ABORTing immediately for that makes retries expensive. The script first attempts automatic correction with sed -i '' -E 's#/Users/[A-Za-z0-9._-]+/#~/#g' (lines 434–437), and only aborts if a secret or AKIA… pattern still remains afterward. The principle of "fix what can be fixed, stop only for what can't" reduces unnecessary retries.

8. Check the remaining token budget at the top of the script

If you start Claude calls and then fail due to insufficient budget, all the preparation up to that point is wasted. As at lines 93–98, check the remaining budget with token-budget-advisor.sh --short before calling claude -p, and make the decision to end the job first if it's 🔴. This avoids the situation of "burning through the window all at once with two calls — replenish + article generation — in the latter half of a week when the budget is low."

9. Protect long-running processes with caffeinate -i -s exec

If the Mac sleeps partway through a process that takes up to 25 minutes (10 minutes of topic planning + 25 minutes of article writing), the HTTP connection drops and it's treated as a timeout. The single line exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash "$0" "$@" (line 62) restarts the script itself under caffeinate's umbrella. Because exec replaces the process, no extra subshells accumulate. The CAFFEINATED=1 check exists to prevent infinite recursion.

10. Explicitly restrict the push destination owner with ALLOWED_OWNER

git push runs only against repositories of the GitHub user matching ALLOWED_OWNER="bokuwalily" (line 39) (lines 512–522). It's a safety valve preventing accidental pushes to a forked repository or one cloned by mistake. Always check this to prevent your own articles from being published to a repository under someone else's name.

11. Manually test failure cases in dry mode before wiring it to launchd

bash article-daily-stock.sh dry causes neither a queue pop nor a git push, and only writes the stock file (lines 496–500). Before launchctl loading the plist, manually trying the three failure cases — "corrupt the JSON," "delete the article file and run audit," "launch with a lock file left behind" — in dry mode is the shortest route to verifying behavior. If you try to discover failure-case behavior after wiring it to production, you end up in a loop of waiting 24 hours per slot.

12. Keep the background quiet with LowPriorityIO + Nice 10

Setting LowPriorityIO: true, Nice: 10, and ProcessType: Background in the plist means your main work doesn't slow down during article generation. Nice 10 lowers CPU priority, and LowPriorityIO makes the OS defer disk I/O. Even with article generation running while I use Claude Code at the same time, I never feel any sluggishness.

13. Reliably reject stub articles with MIN_ARTICLE_BYTES=1200

When Claude times out for some reason, sometimes only a few dozen bytes of error text saying "the article could not be generated" gets written. The article_ok() function checks the byte count with stat -f%z and treats anything under 1200 bytes as a stub (lines 176–177). MIN_ARTICLE_BYTES is defined as a constant at the top of the file, so when adjusting the minimum article quality you only change it there and it applies everywhere.

14. Record the retry count in ATTEMPTS_FILE and escalate to a human

When audit_repair() returns a broken article to the regeneration queue, bump_attempt() records the attempt count. Once it exceeds MAX_ATTEMPTS=2, it gets stacked into the needhuman array, and writing to _NEEDS-FIX.txt plus a macOS notification (lines 266–268) run as a set. Setting an escalation threshold to a human so that auto-repair doesn't cycle forever is a mandatory design element for machine repair loops.

15. Don't hardcode the nvm version in the plist's PATH — resolve it dynamically in the script

If you hardcode /Users/…/.nvm/versions/node/v24.13.0/bin into the plist's EnvironmentVariables.PATH, you'll need to rewrite the plist every time you upgrade node. By dynamically fetching the latest node's bin with ls -d ~/.nvm/versions/node/*/bin | sort -V | tail -1 at lines 56–58 of the script, the design means you write the plist once and never touch it again.

Wrap-up

To summarize the mechanism introduced in this article: "when the queue goes empty, Claude plans the next topic itself and adds it to the queue" is built on a stack of five decisions.

Decouple generation from deployment: if one stops, it doesn't chain
On failure, exit without setting the done-marker: the OS handles retries
Normalize the output before validating it: don't trust the model's output format
Write state files atomically: they don't break on interruption
Cap auto-repair and hand off to a human: build an exit for when the loop doesn't converge

.topic-queue.json currently has 16 topics waiting. Every morning at 8:00 launchd consumes one, and on the morning after it hits 0, Claude plans the next topic grounded in that day's actual work (daily brief, memory, the script collection) and inserts it into the queue. If the generated JSON doesn't pass the four-field validation, it exits without writing the done-marker and the 10:35 catch-up slot picks it up. This "refill fails → retry at the next slot" is part of one and the same set.

Publishing to Zenn is handled by the separate zenn-daily job process. All article-daily-stock.sh knows about is "are the draft in ~/content/article/articles/ and the thumbnail in ~/content/article/thumbnails/ both in place?" Because of this division of labor, this article series has piled up without a single day's gap.

Of the revenue structure behind ¥1.2M a month, "continuous knowledge publishing" in the form of a series is the highest cost-performance means of asset building. The state of "I can't write because I've run out of topics" can be solved with a mechanism.

One question for you: if you've built something like this, which failure finally forced you to add a lock — a duplicated output file, or a corrupted state file?

I've collected the full picture of the mechanism, the breakdown of the ¥1.2M/month, and the 30-day procedure in a paid note.
📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

52 Days of Silent Zeros: The Stop Hook Payload Has No usage Field

Lily — Fri, 28 Aug 2026 05:00:06 +0000

I used to be a college student scraping by on ¥100k a month. Then I was laid off. Six months later, after building an autonomous Claude Code environment, I'm clearing ¥1.2M a month in revenue. What closed that gap wasn't talent or capital — it was continuously growing an environment that thinks and works in my place.

Why this setup works

The first thing I noticed when trying to grow income through solo development: raising the quality of the environment has far better ROI than raising the amount of work. Spending a week building a system that runs autonomously and stacks up deliverables while I'm away beats hammering Claude Code for 8 hours a day — at least when you measure revenue three months out. This series is a record of mass-producing that kind of environment.

Today's topic is cost visibility. When you live on Claude Code, token consumption happens as naturally as breathing. The problem is that you can't optimize a cost you can't see. "How much did I spend this month?" "Which session was heavy?" "How many dollars a month would I save by shifting a bit more Sonnet work to Haiku?" — without answers to these, gross margin doesn't improve even as revenue grows.

Claude Code has a hook system. Shell scripts or Node.js scripts configured in ~/.claude/settings.json run automatically when tied to specific events. The Stop hook is the most important of them, and it fires every time the assistant completes a turn. Not just at session end — once per completed turn (see the comment on line 19 of cost-tracker.js: "Stop fires per assistant response, not per session").

The natural idea here is: "If I record token counts in the Stop hook, I get automatic cost tracking." The implementation looks simple. The hook receives a JSON payload on stdin. Read usage.input_tokens and usage.output_tokens from that payload, append to a JSONL file, done — and building it with that assumption is exactly the mistake the first version made.

The Stop hook payload has no usage field.

Here's what the payload actually looks like:

{
  "session_id": "...",
  "transcript_path": "/path/to/session.jsonl",
  "cwd": "/path/to/workdir",
  "hook_event_name": "Stop"
}

session_id, transcript_path, cwd, hook_event_name — that's it. No model name, no token counts, no cost. Because this isn't spelled out in the docs, if you write code assuming usage exists, you read a nonexistent field and get undefined, Number(undefined) becomes NaN, you keep adding NaN, and 0 gets recorded. No errors. Just silent zeros, piling up every turn.

The comment in my cost-tracker.js preserves the evidence verbatim (lines 12–13):

* The Stop payload does NOT include `usage` or `model` directly. The previous
* version of this hook expected those fields and silently produced zero-filled
* rows (verified: 2,340 rows captured with 0.0% non-zero token rate over 52
* days).

52 days, 2,340 rows, 0.0% non-zero rate. As a tracker, a total failure. And the whole time, the script kept running without complaint, the log file grew steadily, and running cost-summary.sh returned "$0.00 / 0 sess". It looked like it was working while recording nothing.

This isn't merely an implementation bug — it's an architectural mistake stemming from a misunderstanding of what Claude Code's Stop hook is. To fix it, you have to give up on a field that doesn't exist in the payload and go read the place the payload points to — transcript_path.

That switch is the core of this article.

The overall flow

Here's the corrected architecture at a glance:

┌─────────────────────────────────────────────────────┐
│  Claude Code セッション                              │
│                                                     │
│  アシスタントターン完了                              │
│       │                                             │
│       ▼                                             │
│  Stop フック発火                                    │
│       │                                             │
│       ▼ stdin (JSON)                                │
│  { session_id, transcript_path, cwd, ... }          │
│       │                                             │
│       ▼                                             │
│  cost-tracker.js                                    │
│  ┌──────────────────────────────────────────────┐   │
│  │  1. transcript_path を取得                   │   │
│  │  2. JSONL を読み込み                         │   │
│  │  3. type="assistant" の行だけフィルタ        │   │
│  │  4. message.usage を積算                     │   │
│  │  5. モデル名からレートを引いてコスト計算     │   │
│  │  6. ~/.claude/metrics/costs.jsonl に追記     │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

The key is the shift in understanding: "the Stop hook is not a courier delivering cost information." The hook is strictly an event notifier, and all the notification contains is an address (transcript_path) telling you which session transcript to read. The cost information lives inside the transcript.

The structure of the JSONL that transcript_path points to

Claude Code writes every turn of a session into a single JSONL file. Each line corresponds to one message, and the type field distinguishes the kind.

What you need for cost calculation are the type: "assistant" lines. Their structure is as follows:

{
  "type": "assistant",
  "message": {
    "model": "claude-sonnet-4-6",
    "usage": {
      "input_tokens": 12483,
      "output_tokens": 847,
      "cache_creation_input_tokens": 8192,
      "cache_read_input_tokens": 3200
    }
  }
}

Summing the four kinds — input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens — across all assistant turns gives you the session's total token consumption. Multiply by the billing rates and you have cost.

Implementing the sumUsageFromTranscript function

Lines 57–90 of cost-tracker.js implement this JSONL accumulation logic.

function sumUsageFromTranscript(transcriptPath) {
  let content;
  try {
    content = fs.readFileSync(transcriptPath, 'utf8');
  } catch {
    return null;
  }

  let inputTokens = 0;
  let outputTokens = 0;
  let cacheWriteTokens = 0;
  let cacheReadTokens = 0;
  let model = 'unknown';

  for (const line of content.split('\n')) {
    if (!line.trim()) continue;
    let entry;
    try { entry = JSON.parse(line); } catch { continue; }

    if (entry.type !== 'assistant') continue;
    const msg = entry.message;
    if (!msg || !msg.usage) continue;

    const u = msg.usage;
    inputTokens      += toNumber(u.input_tokens);
    outputTokens     += toNumber(u.output_tokens);
    cacheWriteTokens += toNumber(u.cache_creation_input_tokens);
    cacheReadTokens  += toNumber(u.cache_read_input_tokens);

    if (msg.model && msg.model !== 'unknown') model = msg.model;
  }

  return { inputTokens, outputTokens, cacheWriteTokens, cacheReadTokens, model };
}

Three design decisions are worth noting.

Swallow parse errors and continue. Even if a line partway through the JSONL is corrupted, try { entry = JSON.parse(line); } catch { continue; } skips it. The Stop hook must be non-blocking. Having Claude Code's session termination fail because the cost log couldn't be captured would be completely backwards.

Prevent NaN with toNumber(). The reason the old implementation kept writing 0 was NaN propagating from trying to convert a nonexistent field into a number. In the new implementation, a toNumber() helper checks with Number.isFinite() and returns 0 if the value isn't a finite number (lines 47–49).

Use the last model name found. Assuming the model can switch mid-session, it keeps overwriting as long as msg.model isn't 'unknown'. The model from the later part of the session becomes the representative value, but the cost error is generally within acceptable range.

The rate table and cost calculation

Per-model billing rates are hardcoded on lines 34–38.

const RATE_TABLE = {
  haiku:  { in: 0.80,  out: 4.0,  cacheWrite: 1.00,  cacheRead: 0.08 },
  sonnet: { in: 3.00,  out: 15.0, cacheWrite: 3.75,  cacheRead: 0.30 },
  opus:   { in: 15.00, out: 75.0, cacheWrite: 18.75, cacheRead: 1.50 }
};

Units are US dollars per 1M tokens. The getRates() function (lines 40–45) determines haiku, opus, or other (default sonnet) from the model name string.

The cost calculation is contained in lines 128–133.

const estimatedCostUsd = Math.round((
  (inputTokens      / 1e6) * rates.in +
  (outputTokens     / 1e6) * rates.out +
  (cacheWriteTokens / 1e6) * rates.cacheWrite +
  (cacheReadTokens  / 1e6) * rates.cacheRead
) * 1e6) / 1e6;

The / 1e6 * 1e6 round trip is there to round away floating-point error. Rounding at the micro-dollar level keeps errors below 0.000001 out of the result.

The line ultimately appended to ~/.claude/metrics/costs.jsonl looks like this:

{
  "timestamp": "2026-08-18T08:45:22.000Z",
  "session_id": "abc123",
  "transcript_path": "~/.claude/transcripts/abc123.jsonl",
  "model": "claude-sonnet-4-6",
  "input_tokens": 12483,
  "output_tokens": 847,
  "cache_write_tokens": 8192,
  "cache_read_tokens": 3200,
  "estimated_cost_usd": 0.051234
}

Everything up to here is recorded automatically on each completed turn. Every time the Stop hook fires, the session's "cumulative cost up to that point" is appended. If you want the final cost for one session, you take the last line with the same session_id.

In the next part, I'll verify the specific failure patterns of the old implementation that spat out zeros for 52 days, and how the output of cost-summary.sh changed before and after the fix.

Implementation details

The stdin design and the 64KB cap

Input to the Stop hook arrives on stdin. Since Node.js stdin is a stream, data may arrive split across multiple chunks. cost-tracker.js handles this on lines 92–98 as follows:

const MAX_STDIN = 64 * 1024;
let raw = '';

process.stdin.setEncoding('utf8');
process.stdin.on('data', chunk => {
  if (raw.length < MAX_STDIN) raw += chunk.substring(0, MAX_STDIN - raw.length);
});

The 64KB (65,536 byte) cap is a defensive design: the Stop hook payload is a 4-field JSON of a few hundred bytes in practice, but the cap prevents the process from hanging on a future Claude Code spec change or an unexpectedly large payload. The double guard of raw.length < MAX_STDIN and chunk.substring(0, MAX_STDIN - raw.length) ensures the read buffer stops at the cap no matter what input arrives.

If the cap is exceeded and the JSON is cut off partway, JSON.parse(raw) on line 100 throws. But the outer try { ... } catch { } catches it and the hook terminates without incident. What matters is the structure on lines 151–156:

  } catch {
    // Non-blocking — never fail the Stop hook.
  }

  // Pass stdin through (required by ECC hook convention).
  process.stdout.write(raw);
});

process.stdout.write(raw) is outside the try. Even if a parse failure blows away the cost record, the stdin contents always flow to stdout. This is a requirement of the ECC hook convention, and an expression of the design philosophy that delivering input to downstream links in the hook chain takes top priority. Losing one line of cost log versus crashing the entire Stop hook — the magnitudes of impact are literally incomparable.

The three-tier fallback for transcript_path

Lines 104–106 hold the logic for obtaining transcript_path.

const transcriptPath = (typeof input.transcript_path === 'string' && input.transcript_path)
  ? input.transcript_path
  : process.env.CLAUDE_TRANSCRIPT_PATH || null;

The first candidate is the payload's input.transcript_path. It's adopted only after confirming it's a string type and non-empty. The typeof check is there to prevent an exception from touching undefined when the payload falls back to an empty object {} (e.g. stdin was empty).

The second candidate is the environment variable CLAUDE_TRANSCRIPT_PATH. This is for testing and debugging. It's unnecessary via the actual Stop hook, but when you want to run the script standalone by hand and check the log, you can set the environment variable manually and run it.

CLAUDE_TRANSCRIPT_PATH=~/.claude/transcripts/abc.jsonl \
  echo '{}' | node cost-tracker.js

The old implementation, which tried to read directly from the payload, had none of this design — it didn't even have a variable called transcript_path.

The sessionId resolution chain

Session ID retrieval is a three-stage process on lines 108–112.

const sessionId =
  sanitizeSessionId(input.session_id) ||
  sanitizeSessionId(process.env.ECC_SESSION_ID) ||
  sanitizeSessionId(process.env.CLAUDE_SESSION_ID) ||
  'default';

sanitizeSessionId is a utility that validates and sanitizes UUID format. Getting session_id from the payload is ideal, but in an ECC session-management environment there may be ECC_SESSION_ID, and in plain Claude Code there may be CLAUDE_SESSION_ID as environment variables. If none can be obtained, it falls through to 'default'. Rows with 'default' effectively serve as a signal you can check later that "something is completely broken."

Floating-point countermeasures in cost calculation

Repeating the cost calculation from lines 128–133:

const estimatedCostUsd = Math.round((
  (inputTokens      / 1e6) * rates.in +
  (outputTokens     / 1e6) * rates.out +
  (cacheWriteTokens / 1e6) * rates.cacheWrite +
  (cacheReadTokens  / 1e6) * rates.cacheRead
) * 1e6) / 1e6;

The round trip of * 1e6, then Math.round, then / 1e6 exists to erase floating-point error below the micro-dollar level. It prevents JavaScript's famous 0.1 + 0.2 === 0.30000000000000004 trap from leaving damage in the decimals of cost aggregation. You could argue there's no real harm if $0.051234 gets recorded in the JSONL as $0.051234000000000003, but the numbers look ugly when you later aggregate with cost-summary.sh. It's a single line of processing, but it shows that the quality of the record is being taken seriously.

Where I got stuck

The fix wasn't finished by simply rewriting the old zero-writing implementation to read the transcript. Even after rewriting to v2, two separate problems stacked up and turned verification into a maze.

Stuck point ①: Fixed to v2, but summary still says "$0.00 / 0 sess"

The morning after implementing and deploying sumUsageFromTranscript, I ran cost-summary.sh.

=== cost summary (last 7d) ===
  sessions: 0
  total:    $0.00

Thinking "zero again," I first looked directly at ~/.claude/metrics/costs.jsonl.

tail -3 ~/.claude/metrics/costs.jsonl

The file existed, with 3 rows recorded for that day. The estimated_cost_usd values were 0.048291, 0.031457, 0.072139. v2 was working correctly.

The cause was what cost-summary.sh was reading from. Look at line 10 of the script:

LOG="$HOME/.claude/logs/cost-log.jsonl"

That's ~/.claude/logs/cost-log.jsonl. Meanwhile, what cost-tracker.js writes to is ~/.claude/metrics/costs.jsonl. Different directories. logs vs metrics.

The field names didn't match either. The aggregation logic in cost-summary.sh (lines 39–47) looks like this:

total += r.get("cost_usd", 0)
t = datetime.datetime.fromisoformat(r["ts"])

It reads cost_usd and ts. But the fields cost-tracker.js writes are estimated_cost_usd and timestamp.

In other words, at the point I fixed things to v2, two mismatches existed simultaneously: a file path mismatch and a field name mismatch. If it were only one of them, cost-summary.sh would read an empty file and return "0 sess". With both at once the symptom is unchanged, so I nearly misdiagnosed it as "v2 still isn't working either."

What was actually working was the v2 hook. What wasn't working was where the summary script was reading from.

The debugging lesson was "look at the terminal record file directly with tail -f." Rather than trusting the summary script's output and concluding "it's not working," check the raw JSONL with your own eyes. Without the habit of isolating which stage of the pipeline is broken one step at a time, this double mismatch would have stayed unsolved.

Stuck point ②: Why NaN propagation doesn't make the rows look broken

Let me trace a bit more deeply how the v1 code kept writing 2,340 zero rows.

The old implementation looked (in pseudocode) like this:

// v1 (旧実装の想定コード)
const payload = JSON.parse(raw);
const inputTokens  = Number(payload.usage?.input_tokens);   // undefined → NaN
const outputTokens = Number(payload.usage?.output_tokens);  // undefined → NaN
const cost = (inputTokens / 1e6) * rates.in + ...;          // NaN

Number(undefined) returns NaN. Every arithmetic operation using NaN returns NaN. That much is predictable. The problem is the next line.

const row = { estimated_cost_usd: NaN, input_tokens: NaN, ... };
JSON.stringify(row);
// → '{"estimated_cost_usd":null,"input_tokens":null,...}'

JSON.stringify converts NaN to null. That's the JavaScript spec. Not an error — it quietly becomes null. The lines written to the JSONL aren't malformed JSON; they're perfectly well-formed lines. It's just that all the values are null.

On the cost-summary.sh side, the Python code (lines 40–49) is written as r.get("cost_usd", 0), so when the key exists but the value is null, it returns None. In Python, total += None throws a TypeError, but except Exception: continue on line 49 skips all exceptions, so that session's row is ignored and total doesn't move.

No errors. No exceptions. Just silently skipped from aggregation, over and over. The structure was: it wasn't that zeros were being recorded — the recorded rows were being continuously excluded from aggregation. Writing null is the JSON spec; skipping null is Python's try-except. Each is correct behavior on its own, but combined they created a 52-day blind spot.

The new implementation's toNumber() helper (lines 47–49) cuts this propagation off at the source.

function toNumber(v) {
  const n = Number(v);
  return Number.isFinite(n) ? n : 0;
}

Number.isFinite(NaN) returns false. NaN, Infinity, null, and undefined all get converted to 0 here. The path by which null could contaminate a record row is sealed at the very top of the conversion chain.

Stuck point ③: Unexpected cost overstatement from how "`model: unknown`" rows are handled

sumUsageFromTranscript scans the assistant turns of a session JSONL and adopts the last msg.model found as the representative model (line 86: if (msg.model && msg.model !== 'unknown') model = msg.model).

But the Stop hook fires "on each completed assistant turn" (comment line 19: "Stop fires per assistant response, not per session"). When the hook runs right after the session's first turn finishes, the transcript has only one assistant line, and there's no problem as long as the model name was recorded correctly on that turn.

However, in rare cases the assistant message early in a session has no model field (streaming interruption, tool-use-only turns, etc.). In that case model stays 'unknown' and gets passed to getRates('unknown').

function getRates(model) {
  const m = String(model || '').toLowerCase();
  if (m.includes('haiku')) return RATE_TABLE.haiku;
  if (m.includes('opus'))  return RATE_TABLE.opus;
  return RATE_TABLE.sonnet;   // ← fallthrough
}

'unknown' matches none of the conditions, so the sonnet rate is applied. Early-session turns from a session that was using a Haiku model get calculated at the sonnet rate, and the cost is recorded inflated by 3.75× (in: 0.80 vs in: 3.00).

I noticed this discrepancy when I was looking directly at costs.jsonl and saw multiple rows with model: unknown.

{"model":"unknown","input_tokens":4821,"estimated_cost_usd":0.014463}

Calculating 4,821 input tokens at the sonnet rate gives about $0.0145. At the haiku rate it would be about $0.0039. Nearly a 4× difference.

The root fix would be either "defer cost calculation for rows where the model couldn't be obtained" or "retroactively correct once the model becomes known in a later turn," but implementation complexity jumps sharply. In the current implementation, the policy is to approximate unknown rows at the sonnet rate and judge the monthly error to be within acceptable range. What I want from a cost tracker is precision at a granularity usable for business decisions, not matching the Anthropic console down to the yen.

Whether you make that trade-off consciously is, I think, the difference in attitude toward a tool's reliability. Rather than settling for "that's just how it is," it's only once you've traced "the conditions under which this row becomes unknown," "the upper bound of the cost error at that point," and "the impact that has on monthly aggregation" that the numbers become something you can use with confidence.

In the next part, I'll cover how cost-summary.sh started returning non-zero values after the fix, and the surprising consumption patterns that emerged from the daily cost graph.

Pitfalls

The previous two parts dug into four points: "the Stop hook payload has no usage," "the NaN→null JSON.stringify trap," "path/field name mismatches," and "model:unknown cost overstatement." Here I'll cover the actual sticking points that fell outside those. Every one of them was a wall I thought was impossible.

Because the Stop hook fires per turn, dozens of rows accumulate under the same session_id.
If there are 30 assistant turns in one session, 30 rows get written to costs.jsonl. It's stated clearly in comment line 19 of cost-tracker.js ("Stop fires per assistant response, not per session"), but if the aggregation script doesn't know this, it sums all rows and produces a value dozens of times the real cost. I also made the error in the opposite direction: I thought "I spent $90 this month" when it was actually $4. To get per-session cost, you must aggregate only the last row for each session_id (= the maximum cumulative value). The current aggregation logic in cost-summary.sh (lines 37–49) doesn't account for this cumulative structure and adds every row with for line in open(log):. Fixing the hook to v2 isn't enough — the summary script needs fixing too.
Passing a transcript_path with a literal ~ to fs.readFileSync crashes immediately.
Node.js's fs.readFileSync('~/.claude/...') doesn't do shell expansion. ~ is treated as a plain string and it throws because the file doesn't exist. After receiving the path on lines 104–106 of cost-tracker.js, if there's no processing to expand it with os.homedir() or path.resolve(), then depending on the Claude Code execution environment, transcript_path arrives with a tilde. I actually hit this. You should run transcriptPath.replace(/^~/, os.homedir()) before using input.transcript_path.
Writing the hook command in settings.json as just a path means node isn't found.
If you're using nvm-managed Node, the nvm bin directory may not be in PATH by the time the hook is launched through a shell. The same applies to a #!/usr/bin/env node shebang — there are cases where the node that /usr/bin/env finds points to an old system Node at /usr/local/bin/node (v16, etc.). The reliable approach is to specify node's absolute path explicitly in settings.json's command.

  {
    "hooks": {
      "Stop": [{
        "command": "/Users/<you>/.nvm/versions/node/v24.13.0/bin/node ~/.claude/scripts/hooks/cost-tracker.js"
      }]
    }
  }

Adjust the path to your own setup. When a hook "doesn't feel like it's firing," this is the first thing I suspect.

Debugging with console.log breaks the hook chain.
The Stop hook's stdout is exclusively for passing the payload through. cost-tracker.js line 156: process.stdout.write(raw); — downstream scripts in the hook chain receive their input from this stdout. Insert a single console.log('debug:', something) and an arbitrary string contaminates stdout, breaking the downstream hook when it tries to parse the JSON. Always write debug output to process.stderr.write(...) or a dedicated log file.
Not counting cache_creation_input_tokens makes costs come out low.
Because Claude Code makes heavy use of prompt caching, billing for cache_creation_input_tokens accounts for a non-negligible share. Sonnet's cache write rate is $3.75/1M tokens (1.25× normal input) and the read rate is $0.30/1M tokens (0.1×) (see the RATE table on lines 34–38 of cost-tracker.js). If a session's input is 50,000 tokens with 30,000 of those being cache writes, the write portion alone is about $0.11 — 1.5× the cost of normal input. Leaving cache_creation_input_tokens out of aggregation makes 20–40% of your real cost invisible.
Calling process.exit() inside the Stop hook cuts stdin short and loses data.
Node.js stdin is a stream. If you detect an error inside the process.stdin.on('data', ...) callback and call process.exit(1), the process ends before the remaining chunks arrive, and the payload gets cut off. Design hooks so all processing completes inside the process.stdin.on('end', () => { ... }) callback, with no early exit.
Even when transcript_path arrives, the file may not exist.
The Stop hook fires simultaneously with session completion, but there's a very short race window between Claude Code writing the transcript and the hook launching. That's what the existence check on line 115 of cost-tracker.js — if (transcriptPath && fs.existsSync(transcriptPath)) — is for. Omit it and call readFileSync directly, and occasionally you'll throw on "file doesn't exist" rather than "the path exists but can't be read." The outer try-catch will catch the error, but that session's cost is lost entirely.
Trying to match the Anthropic console's numbers is a swamp.
The cost cost-tracker.js calculates is strictly an estimate. Actual billing is based on the token counts Anthropic measures, plus various factors like batch discounts, promotions, and taxes. Start chasing "why is it $2 off from the console" and it will consume infinite time. All I want from this tracker is "monthly trends and identification of high-cost sessions." I decided within ±15% of the Anthropic console is my acceptable range and stopped pursuing precision beyond that.
Verifying behavior using only the summary script's output.
I touched on this in p2, but I'll restate it as a universal anti-pattern. cost-summary.sh returns correct numbers only when the read path, field names, and aggregation logic are all correct. If even one of them doesn't match, you get $0.00 / 0 sess. Whether the hook is working correctly can only be determined by checking ~/.claude/metrics/costs.jsonl directly with tail -5, not by the summary output. When you think "it's not working," your first command should be checking the raw file, not re-running the summary script.

Best practices

Here are 10+ design principles for tracking cost with the Stop hook, solidified through implementation.

① Wrap the entire hook in try-catch and make it absolutely non-blocking.
Even if the cost log can't be captured, it must not obstruct Claude Code's session termination. Lines 151–153 of cost-tracker.js are the core of this design.

} catch {
  // Non-blocking — never fail the Stop hook.
}

Losing the cost log is far more acceptable than crashing the hook.

② Put the stdin pass-through outside the try block.
If you put process.stdout.write(raw) inside the try, input stops reaching downstream scripts in the hook chain when parsing fails. The placement on line 156 (after the try-catch) is deliberate design. The cost log can be lost, but the hook chain stays alive — that's the priority order.

③ Kill NaN at the source with a toNumber() helper.
Number(undefined) === NaN, and JSON.stringify({v: NaN}) becomes {"v":null}. Cutting off this propagation with toNumber() on lines 47–49 seals the path by which null contaminates a record row, at the source.

function toNumber(v) {
  const n = Number(v);
  return Number.isFinite(n) ? n : 0;
}

④ Take the write path and the read path from the same constant.
The logs/cost-log.jsonl vs metrics/costs.jsonl mismatch I wrote about in p2 was born from two scripts writing the path separately. Ideally you place a constant like COSTS_PATH in a shared module and import it from both the hook script and the summary script. Between two JavaScript files you can share it with require('../lib/paths').

⑤ Manage field names with schema constants too.
Name mismatches like ts vs timestamp and cost_usd vs estimated_cost_usd also don't happen if the schema is defined in one place. It structurally guarantees that JSONL writing and reading reference the same keys.

⑥ Verify the terminal file directly with tail.
The summary script's output isn't the truth — it's "the world as the summary script sees it." The most reliable way to confirm the hook is working correctly is this one command:

tail -5 ~/.claude/metrics/costs.jsonl | python3 -m json.tool

If estimated_cost_usd has a non-zero value, the hook is working.

⑦ Prevent hangs with a stdin cap (64KB).
cost-tracker.js line 92: const MAX_STDIN = 64 * 1024;. The current Stop hook payload is a few hundred bytes, but this is a defensive design preventing the process from hanging on a future spec change or unexpectedly large input.

⑧ Check that transcript_path exists before reading.
Forget the fs.existsSync(transcriptPath) check (line 115) and a read will run during the race window when the file doesn't exist, losing that session's cost row entirely.

⑨ Always count cache_creation_input_tokens and cache_read_input_tokens.
Omit these two fields and 20–40% of your real cost becomes invisible on cache-heavy sessions. Because Claude Code automatically caches long prompts, the cache ratio grows the more you use it.

⑩ Use only the last row of each session_id as the per-session cost.
Sum all rows without knowing the cumulative design and you get a value dozens of times the real cost. Always shape the aggregation logic as "group by session_id and take the row with the latest timestamp."

⑪ Make model: unknown rows identifiable to make the location of cost error explicit.
Rows whose model field is 'unknown' are approximated at the sonnet rate (the getRates function's fallthrough). When there are many such rows, the divergence from real cost widens. Displaying the count and computed cost of unknown rows separately during aggregation makes the scale of the error visible.

⑫ Send debug output to stderr or a log file. Don't touch stdout.
A hook's stdout is exclusively for pass-through. console.log is forbidden; use only console.error or appending to a dedicated log file.

⑬ Specify node's absolute path in the hook command.
Specifying /path/to/.nvm/versions/node/vX.Y.Z/bin/node directly ensures the hook launches regardless of whether PATH is expanded. It's mandatory in execution environments where nvm use doesn't take effect.

⑭ State your acceptable error range and stop chasing precision.
What you want from a cost tracker is precision sufficient to "identify high-cost sessions and grasp monthly trends," not matching the Anthropic console to the yen. Deciding on an error you can accept (e.g. ±15%) saves you from spending time on pointless precision improvements.

Summary

52 days, 2,340 rows, 0.0% non-zero rate — the fact that this confession still lives in the code shows how quietly, and for how long, a misunderstanding of the Stop hook can keep doing real damage.

The root cause was simple: the assumption that "the Stop hook payload has a usage field." Trying to read a field that doesn't exist in the payload produced NaN, which JSON.stringify converted to null, which was then silently skipped during aggregation. No errors, no warnings. Just zeros quietly stacking up.

The essence of the fix comes down to one point. The cost information isn't in the payload — it's in the JSONL that transcript_path points to. The Stop hook is "a notifier that tells you which JSONL to read," not "a courier that delivers cost information." That shift in understanding is the entirety of the v1-to-v2 rewrite.

Even if v2 is written correctly, the output stays zero unless cost-summary.sh's read target and field names match. Until you look at the terminal file directly with tail -5, you can't tell which of the two is broken. The principle of pipeline debugging is not "trust the summary output" but "work backwards from the most terminal record."

Once cost becomes visible as numbers, the resolution of your business decisions changes. "How much am I spending on Sonnet per month?" "Which work can be replaced by shifting to Haiku?" "Is the overnight autonomous loop more cost-effective than a morning manual session?" — you can judge these with numbers instead of intuition. Being able to keep optimizing the breakdown of ¥1.2M in monthly revenue is possible because there's a system making gross margin visible.

I've written up the full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day procedure in a paid note.

📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

3 Wrong Guesses About a Chrome Slot Limit: 3 6 Reserved Lanes

Lily — Fri, 28 Aug 2026 00:00:04 +0000

Automation doesn't break when you write it. It breaks in production, weeks later, quietly — and my Chrome concurrency limit broke three separate times before it settled.

Why This Mechanism Matters

Once you run automation seriously as a solo developer, you always hit the same wall: multiple jobs trying to use the same resource at the same time.

In the context of social media automation, that resource is Chrome. Liking, following, posting, note integration — browser-driven jobs run around the clock. In my setup, a launchd job called com.lily.autolike.ig-1 fires every two hours, 12 times a day (0:36, 2:36, 4:36 … 22:36), and drives automatic likes through Chrome. That alone is 12 processes a day, and jobs of the same kind exist across multiple accounts and multiple social networks.

With no control at all, overlapping start times mean nine Chrome instances launch simultaneously. It's not hard to imagine the Mac screaming.

The problem is how it breaks. If one of them crashed from memory exhaustion, that would at least be legible. What actually happens is that every process stays alive but becomes extremely slow, and timeouts pile up. The run looks like it completed, but effectively nothing happened. The logs fill with TIMEOUT: killed after 900s.

The idea of "control this with a cap on concurrent launches" is itself correct. What I got wrong was the basis for the cap value, and how I treated priority across job types.

~/.claude/scripts/browser-slot.sh carries all three mistakes and their fixes, etched into a single file. The comments serve as a change log, and tracing the numbers and dates reveals the structure of the failures.

The Real Problem: Intuitive Resource Blame

When automation gets heavy, your intuition says "something must be eating resources." Chrome sure looks heavy. My decision to set the cap to 3 was based on the hypothesis that Chrome was devouring memory.

When the hypothesis is wrong, the countermeasure is wrong. The cap of 3 was a number born from the misdiagnosis "throttle Chrome and things get lighter." Until I actually measured, I didn't notice that this misdiagnosis was producing 92 skips a day.

Put the other way around: measuring and recording it in a comment makes the misdiagnosis visible. The script's comments (lines 5–8) read as a postmortem report on their own.

# 実測(2026-08-09): Chrome系ジョブ9個で合計0.7GB。swap枯渇の主犯は dasd(47GB)/
# ComfyUI(12GB)/iii(3.5GB)であってブラウザジョブではなかった。上限3は過剰に厳しく
# 1日で92回のskip(全体の30%)を出していたので5に緩める。
SLOT_MAX="${BROWSER_SLOT_MAX:-6}"

The comment says "relax it to 5," but the actual default value of SLOT_MAX is 6. At the time I wrote the comment I was assuming 5, but I ended up setting 6. That very "gap between comment and value" is the fingerprint of a fix made in one sitting.

Lock Files Live or Die by Atomicity

For the semaphore mechanism, browser-slot.sh chose the atomicity of the filesystem's mkdir.

GUARD_DIR="$SLOT_DIR/.guard"

acquire_guard() {
  attempt=0
  while [ "$attempt" -lt 100 ]; do
    if mkdir "$GUARD_DIR" 2>/dev/null; then
      printf '%s\n' "$$" > "$GUARD_DIR/pid"
      GUARD_HELD=1
      return 0
    fi
    # ...ゾンビガードの掃除...
    attempt=$((attempt + 1))
    sleep 0.05
  done
  return 1
}

mkdir is atomic under POSIX. If the directory doesn't exist it is created and succeeds; if it already exists it fails. This mutual exclusion, which doesn't consume a single byte, guarantees that even when multiple jobs call acquire_guard() simultaneously, exactly one gets through.

The process that acquires the guard then scans the *.lock files under ~/.cache/lily-browser-slots/ to count how many are currently running. The first line of a lock file is the PID, and liveness is checked with kill -0 $pid. Lock files belonging to dead processes get cleaned up.

for lock in "$SLOT_DIR"/*.lock; do
  [ -e "$lock" ] || continue
  lock_pid="$(sed -n '1p' "$lock" 2>/dev/null)"
  # ...
  if ! kill -0 "$lock_pid" 2>/dev/null; then
    rm -f "$lock"
    continue
  fi
  running=$((running + 1))

If the running count is below the limit, it writes a .lock file under its own label name, releases the guard, and runs the command. If the count is at or above the limit, it either skips or waits.

The advantage of this approach is that it self-heals even when a process dies abnormally. The lock file of a crashed process gets cleaned up by whoever comes next. Files written to /tmp/ disappear on reboot. And unlike flock, there's no kernel-side FD management involved. For something a shell script has to handle, this "create a file, and clean it up if it dies" approach is the most robust.

Don't Leave Timeouts to the Shell

The timeout for the executed command is controlled by a Perl subprocess rather than the timeout command (lines 277–327). The reason is that the timeout command can kill only the direct child while leaving grandchildren behind.

Perl's descendants() function builds a PID-to-PPID table for all processes via /bin/ps -axo pid=,ppid=, then recursively collects descendants from the root. On timeout it kills every descendant in the order SIGTERM → wait 1 second → SIGKILL.

local $SIG{ALRM} = sub { $timed_out = 1; stop_tree(); };
alarm $timeout;

The exit code on timeout is exit 124. The script side detects this and records it in the log as RESULT="timeout:${TIMEOUT_SEC}s".

com.lily.autolike.ig-1.plist passes AUTOLIKE_TIMEOUT_SEC=3700 (about 61 minutes) as an environment variable, but the TIMEOUT_SEC default in browser-slot.sh is 900 seconds (15 minutes). Unless each job overrides BROWSER_SLOT_TIMEOUT_SEC, it gets killed at 15 minutes.

The Overall Flow

Here's a diagram of how a single auto-like job "borrows one Chrome slot, does its work, and returns it."

launchd
  com.lily.autolike.ig-1  ←── 0:36 / 2:36 / ... / 22:36（12回/日）
         │
         │ /bin/bash ~/dev/social-autolike/scripts/run-account.sh ig-1
         ▼
   run-account.sh ig-1
         │
         │ browser-slot.sh ig-1 --group engage --group-max N -- <chrome-cmd>
         ▼
   browser-slot.sh
         │
         ├─① acquire_guard()
         │     mkdir ~/.cache/lily-browser-slots/.guard  ← アトミック排他
         │
         ├─② *.lock を走査して running を数える
         │     死んだPIDのロックは掃除
         │
         ├─③ 実効上限を決める
         │     reserved group (post)?
         │       YES → effective_max = SLOT_MAX          (= 6)
         │       NO  → effective_max = SLOT_MAX - SLOT_RESERVE_COUNT  (= 5)
         │
         ├─④ running >= effective_max ?
         │     YES + WAIT_SEC > 0 → 15〜45秒ランダム待機してリトライ
         │     YES + 待機限界    → SKIP:global-limit をログに書いて exit 0
         │     NO               → .lock ファイルを書いてガード解放
         │
         ├─⑤ Perl supervisor で <chrome-cmd> を起動
         │     alarm(TIMEOUT_SEC=900)  ← SIGALRM でタイムアウト
         │     子孫PIDを再帰収集して SIGTERM→SIGKILL
         │
         └─⑥ cleanup()
               .lock 削除
               slot.log へ result=exit:0 / timeout:900s 等を記録

Slot acquisition happens only inside acquire_guard. Even with multiple jobs running at once, only one process holds the guard at a time. Because counting the running total and writing the .lock file both complete inside the guard, you never get the race where "I counted and was under the limit, but another process passed through at the same time and pushed us over."

File Layout and Key Variables

SLOT_DIR="${BROWSER_SLOT_DIR:-$HOME/.cache/lily-browser-slots}"
SLOT_MAX="${BROWSER_SLOT_MAX:-6}"
SLOT_RESERVED_GROUPS="${BROWSER_SLOT_RESERVED_GROUPS:-post}"
SLOT_RESERVE_COUNT="${BROWSER_SLOT_RESERVE:-1}"
TIMEOUT_SEC="${BROWSER_SLOT_TIMEOUT_SEC:-900}"
WAIT_SEC="${BROWSER_SLOT_WAIT_SEC:-600}"
SLOT_LOG="${BROWSER_SLOT_LOG:-$HOME/.cache/lily-browser-slots/slot.log}"

Every variable is designed to be overridable via environment variables. Change BROWSER_SLOT_MAX and you change the cap without touching the script at all. It's the same mechanism by which com.lily.autolike.ig-1.plist passes AUTOLIKE_TIMEOUT_SEC=3700 — each job can have its own behavior through launchd's EnvironmentVariables block.

The slot label is the first command argument (e.g. ig-1). It becomes the filename ig-1.lock directly, and a duplicate launch under the same label is prevented with SKIP: slot already held. launchd starts jobs via StartCalendarInterval, but even if the previous run is still alive past 60 minutes, a second one won't run at the next start.

Group limits are the higher-level control. Pass --group engage --group-max 3 and you can layer on a "the engage group gets at most 3" restriction separate from the global cap. Put likes, follows, and note engagement in the same group and you prevent the state where slots are free but too many engage jobs run in parallel.

Logs are appended to slot.log in a one-line format.

2026-08-15T07:42:11+0900 label=xpilot.autopost group=post result=exit:0
2026-08-15T07:43:05+0900 label=ig-1 group=engage result=skip:global-limit:6/5:waited=600s

This format is the raw material for aggregation. Count the result=skip:global-limit lines and you know that day's slot contention. If result=timeout:900s starts increasing, that's a sign something is jammed.

How It's Actually Called

When launchd's plist starts run-account.sh ig-1, that script calls browser-slot.sh internally. The call looks like this.

browser-slot.sh "ig-1" \
  --group engage \
  --group-max 3 \
  -- python3 ~/dev/social-autolike/src/like.py ig-1

ig-1 is the slot label. It fits within the engage group's 3-slot cap, and if it also hits the global cap (effectively 5 slots for engage), it waits. The maximum wait is WAIT_SEC=600 (10 minutes), retrying at random intervals of 15–45 seconds during that window. The randomization exists to avoid re-collision when multiple jobs time out simultaneously and start retrying simultaneously.

retry_delay=$((15 + RANDOM % 31))

If no slot frees up after 600 seconds, it skips and exits 0. It's exit 0 so that launchd doesn't record it as an error. A skip isn't a "failure" — it's "we passed this time," and it gets naturally retried at the next start (two hours later).

Implementation Details

The Crux of `acquire_slot()`: The Effective-Cap Branch

The heart of acquire_slot() is that it doesn't use the global cap as-is. The most important branch in the code is lines 177–188.

# 予約グループはSLOT_MAXまで使える。それ以外は予約分を差し引いた実効上限で止める。
effective_max="$SLOT_MAX"
if ! is_reserved_group "$GROUP"; then
  effective_max=$((SLOT_MAX - SLOT_RESERVE_COUNT))
  [ "$effective_max" -lt 1 ] && effective_max=1
fi
if [ "$running" -ge "$effective_max" ]; then
  SLOT_BLOCK_MESSAGE="SKIP: global limit reached ($running/$effective_max)"
  SLOT_BLOCK_RESULT="skip:global-limit:$running/$effective_max"
  SLOT_BLOCK_WAITABLE=1
  release_guard
  return 1
fi

is_reserved_group() scans the whitespace-separated list in SLOT_RESERVED_GROUPS (default "post") and checks whether the given group name is in it.

is_reserved_group() {
  _needle="${1:-}"
  [ -n "$_needle" ] || return 1
  for _g in $SLOT_RESERVED_GROUPS; do
    [ "$_g" = "$_needle" ] && return 0
  done
  return 1
}

The unquoted $SLOT_RESERVED_GROUPS is expanded directly by for. This is a deliberate use of shell word splitting: if you want to list multiple groups whitespace-separated like "post engage", you just add them to the env variable.

The effective-cap computation is simple. With SLOT_MAX=6 and SLOT_RESERVE_COUNT=1, the engage group's cap becomes 6 - 1 = 5. Only the post group can use all 6. That one-slot difference creates the guarantee that "even with 5 engage jobs running, a post always gets in."

The guard [ "$effective_max" -lt 1 ] && effective_max=1 matters too. It prevents the accident where setting SLOT_RESERVE_COUNT to a value at or above SLOT_MAX drives the effective cap to zero or below and stops every job.

What the Lock File's Three Lines Mean

When a slot is acquired, three lines are written to the lock file named after the label (line 198).

printf '%s\n%s\n%s\n' "$$" "$GROUP" "$(date +%s)" > "$LOCK_FILE"

Line 1 is the PID, line 2 is the group name, line 3 is UNIX time (epoch seconds).

The PID on line 1 plays a dual role. One is the liveness check. When scanning lock files, kill -0 "$lock_pid" 2>/dev/null confirms whether the process is alive. If it's dead, the lock file is deleted and the scan moves on. The other is self-verification at cleanup time. When cleanup() runs, it checks that the PID read with sed -n '1p' matches its own $$ before deleting the file. This prevents the accident of mistakenly removing another process's lock file.

The group name on line 2 is for group-limit counting. During the scan, lock_group="$(sed -n '2p' "$lock")" reads it out, and if it matches your own group, group_running is incremented. Because the lock file itself carries the group information, there's no need to query the running processes.

The epoch seconds on line 3 are for debugging. When the logs alone don't tell you "when did this slot start," you can cat the lock file directly and see the start time.

Why "Random" Works in the Wait Loop

Retries after a failed slot acquisition use a random wait rather than a fixed interval (line 269).

retry_delay=$((15 + RANDOM % 31))

A random wait of 15–45 seconds.

Why not fixed? Suppose five engage jobs end up waiting for a slot at the same time. If they all retry on a fixed 20 seconds, then 20 seconds later all five call acquire_guard() simultaneously. Only one can take the guard with mkdir; the other four are rejected and wait another 20 seconds. Repeat that and you get a periodic scramble over the guard, and throughput drops. Randomizing spreads out the retry timing, producing a flow where slots fill up in sequence. It's the standard technique for avoiding the simultaneous-retry collision known as the "thundering herd."

There's also a comparison against the remaining wait time (lines 271–273).

remaining=$((WAIT_SEC - waited))
if [ "$retry_delay" -gt "$remaining" ]; then
  retry_delay="$remaining"
fi

This prevents a "retry in 30 seconds" setting when only 5 seconds remain on the wait timeout. It's designed to keep making meaningful attempts right up to the edge of the remaining time.

Why Perl Wipes Out the Whole Process Tree

The reason timeout handling isn't left to the shell's timeout command comes down to how grandchild processes are handled.

The Perl supervisor's descendants() function (lines 287–303) builds a system-wide PID-to-PPID table via /bin/ps -axo pid=,ppid= and enumerates all descendants of the given PID with a BFS.

sub descendants {
  my ($root) = @_;
  my %children;
  open my $ps, "-|", "/bin/ps", "-axo", "pid=,ppid=" or return ();
  while (my $line = <$ps>) {
    next unless $line =~ /^\s*(\d+)\s+(\d+)\s*$/;
    push @{$children{$2}}, $1;
  }
  close $ps;
  my @queue = ($root);
  my @found;
  while (@queue) {
    my $parent = shift @queue;
    for my $pid (@{$children{$parent} || []}) {
      push @found, $pid;
      push @queue, $pid;
    }
  }
  return @found;
}

On timeout, stop_tree() sends SIGTERM to the descendants in reverse order (child → parent), waits 1 second, then sends SIGKILL to any process still alive (lines 306–312).

sub stop_tree {
  return if $stopping++;
  my @pids = descendants($child);
  kill "TERM", reverse(@pids), $child;
  select undef, undef, undef, 1;
  kill "KILL", grep { kill 0, $_ } reverse(@pids), $child;
}

return if $stopping++ ensures idempotency. Even if SIGALRM and SIGTERM arrive at the same time and stop_tree() gets called twice, only the first call executes.

Chrome spawns many child processes — renderer, GPU, network service, and so on. Because the shell's timeout command only signals the direct child, grandchildren and below survive. Those leftover Chrome processes created a state where "Chrome is already running" at the next launch, causing a problem where the login session couldn't be obtained. Since switching to Perl killing all descendants together, this zombie-Chrome problem has stopped appearing.

Don't Let launchd See a Skip as an Error

When a slot can't be acquired, the script ends with exit 0 (lines 257–260).

if [ "$SLOT_BLOCK_WAITABLE" -ne 1 ] || [ "$WAIT_SEC" -eq 0 ]; then
  echo "$SLOT_BLOCK_MESSAGE"
  RESULT="$SLOT_BLOCK_RESULT"
  exit 0
fi

com.lily.autolike.ig-1.plist uses StartCalendarInterval for fixed-time launches (0:36, 2:36 … 22:36). launchd treats a non-zero exit code as a "failed job" and in some cases imposes a ThrottleInterval penalty.

A skip is "passing this time," not a failure. It gets naturally retried at the next start two hours later. The log records result=skip:global-limit, so a human can tell the difference. To launchd, it looks like a normal exit. Reconciling both is the reason for exit 0.

Where I Got Stuck

Miscalculation 1: "Chrome Is Heavy" Was Just an Assumption

When I first built the slot management, I set the cap to 3. The basis was a feeling. Chrome sure looks heavy, so let's hold it to less than half — that was the judgment.

The symptoms started quietly. It felt like the number of auto-likes processed was dropping. Aggregating the logs showed 92 result=skip:global-limit entries a day: about 30% of all runs finished without even acquiring a slot.

But I had a confirmation bias — "even if it's skipped, the next run should cover it" — and left it alone for a while. It'll try again in two hours, so we're fine.

The confirmation bias collapsed when I actually measured with vm_stat and top. The result is preserved verbatim in the script's comments.

# 実測(2026-08-09): Chrome系ジョブ9個で合計0.7GB。swap枯渇の主犯は dasd(47GB)/
# ComfyUI(12GB)/iii(3.5GB)であってブラウザジョブではなかった。上限3は過剰に厳しく
# 1日で92回のskip(全体の30%)を出していたので5に緩める。
SLOT_MAX="${BROWSER_SLOT_MAX:-6}"

Against 0.7GB total for nine Chrome-family jobs, dasd (Apple's log daemon) was using 47GB and ComfyUI (an image-generation AI server) 12GB. Chrome's 0.7GB was within the margin of error. The hypothesis that Chrome was the culprit was completely wrong.

The comment says "relax it to 5" while the variable's default is 6. That's the fingerprint of measuring, thinking "I figured 5 would do, but 6 is fine," changing my mind midway, fixing only the variable, and leaving the comment half-written. When you make a fix you're not confident about, you hesitate while writing the comment. Write "5," then reconsider with "6 would be fine too," change only the variable — and this gap is born.

The core of the fix was in the variable design. Because I'd built BROWSER_SLOT_MAX to be overridable via an environment variable, I could change the cap without touching the script at all. Just add one line, BROWSER_SLOT_MAX=5, to launchd's plist. If I'd hardcoded the number inside the script, every change would have meant editing and redeploying the file.

Miscalculation 2: I Raised the Cap Back to 6, and the Post Lane Jammed 9 Times

Six days after relaxing the cap, on 2026-08-15, a new problem appeared.

Checking the morning logs, I found 9 consecutive result=skip:global-limit:6/5 entries for the post job (xpilot.autopost). The slot cap was supposed to be 6, but the record said it jammed at "6/5" — at first I thought it was an aggregation bug.

The record is preserved verbatim in the script's comments (lines 9–14).

# 2026-08-15: 朝の7本同時timeoutで枠6が死んだrunに占有され、投稿レーン(xpilot.autopost
# 等)が global-limit で9回skipした。いいね/フォローは1回落ちても翌回で取り返せるが、
# 投稿はその時間帯の枠が消えると二度と埋まらない。そこで投稿用に枠を予約し、
# engage系(いいね/フォロー/note等)はSLOT_MAXより1つ少ない実効上限で動かす。

Here's the cause. Seven engage-group jobs started simultaneously in the morning window, and each process jammed at TIMEOUT_SEC=900 (15 minutes). For those 15 minutes until timeout, the slots were legitimately occupied. With all 6 slots full, the post job tried to enter, but running=6 is at or above effective_max=6, so it was rejected. This repeated 9 times.

The core of the problem is this: "a like can be recovered on the next run, but once a post's window in that time slot is gone, it never gets filled."

Auto-likes get 12 launch opportunities per day. One skip doesn't change the total processed count much. But a posting schedule is a time specification — "publish at 9 AM." Miss that window and the content becomes either "posted a day late" or "a missing number." Because posting in a specific time slot directly affects engagement rate, the cost of a skip is completely different from that of a like.

Even with the same skip:global-limit symptom, the severity differed entirely depending on job type.

The fix: I added two env variables to introduce a reserved-slot mechanism. I set up SLOT_RESERVED_GROUPS=post (the reserved group name) and SLOT_RESERVE_COUNT=1 (the reserve count), making the engage group's effective_max = 6 - 1 = 5. This guarantees "even with 5 engage jobs running, post always gets the 6th."

The change to the script was minimal. I only added is_reserved_group() and inserted a 4-line block into the cap check in acquire_slot(). It takes effect either by having the calling shell script pass --group post, or by setting BROWSER_SLOT_RESERVED_GROUPS=post in the plist's EnvironmentVariables block.

Miscalculation 3: Timeout Values Managed in Two Places

com.lily.autolike.ig-1.plist has the following environment variable set.

<key>EnvironmentVariables</key>
<dict>
  <key>AUTOLIKE_TIMEOUT_SEC</key>
  <string>3700</string>
</dict>

AUTOLIKE_TIMEOUT_SEC=3700 (about 61 minutes). Meanwhile, the variable browser-slot.sh uses for the Perl supervisor's timeout is BROWSER_SLOT_TIMEOUT_SEC, defaulting to 900 seconds (15 minutes). The variable names differ.

AUTOLIKE_TIMEOUT_SEC is the variable run-account.sh reads, controlling the processing timeout on the Python script side. TIMEOUT_SEC in browser-slot.sh is a separate-layer timeout for when the Perl supervisor force-terminates.

This is where I got stuck. The Python script is built on the premise that it "can take up to 61 minutes," but if the Perl supervisor is left set to "force-terminate at 15 minutes," anything that doesn't finish within that window times out. Conversely, if you forget to set BROWSER_SLOT_TIMEOUT_SEC and run at the 900-second default, account operations that would have completed get killed at 15 minutes. This misconfiguration was hard to notice because an exit-124 log only records "timeout" — it doesn't tell you "the timeout cap is too small."

The plist's Nice: 10 and LowPriorityIO: true settings are also involved.

<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>10</integer>
<key>ProcessType</key>
<string>Background</string>

A nice value of 10 lowers CPU scheduling priority, and launchd aggressively defers this job's work when the system is under load. It's the right setting for system stability, but it has the side effect that "low priority makes processing slower, which in turn makes it easier to hit the 900-second timeout." It's one of the reasons you need to set a timeout value with headroom.

The fix: I added one line inside run-account.sh that reads AUTOLIKE_TIMEOUT_SEC and passes it to browser-slot.sh as BROWSER_SLOT_TIMEOUT_SEC. Having "the timeout Python uses" and "the timeout the Perl supervisor uses" be separate variables is unavoidable, but writing the handoff from the former to the latter explicitly in code eliminated the state of "I configured it but it isn't taking effect." Env variables in general — not just timeouts — make it hard to trace the route of where they're passed and where they're used. Checking once with printenv which process a value written in launchd's plist actually reaches is the reliable move.

Pitfalls

Miscalculations 1–3 (the Chrome misdiagnosis, the post-lane jam, the duplicated timeout management) were covered in detail above. Here I'll list the other, finer-grained traps I actually stepped on.

A gap between comment and variable value is the fingerprint of mid-change hesitation

The comment at lines 5–8 of the script says "relax it to 5," but the actual variable is SLOT_MAX="${BROWSER_SLOT_MAX:-6}".

# 実測(2026-08-09): Chrome系ジョブ9個で合計0.7GB。swap枯渇の主犯は dasd(47GB)/
# ComfyUI(12GB)/iii(3.5GB)であってブラウザジョブではなかった。上限3は過剰に厳しく
# 1日で92回のskip(全体の30%)を出していたので5に緩める。
SLOT_MAX="${BROWSER_SLOT_MAX:-6}"

This isn't a bug — it's the fingerprint of "wrote 5, reconsidered that 6 would be fine, changed only the variable, and left the comment half-written." The smaller the one-line change, the more likely you forget to update the comment. Reading this comment myself later, I spent 5 minutes debugging "it should be 5 but it's 6 — is this a bug?" You need the habit of fixing the comment first when you change a variable.

I underestimated the side effects of Nice: 10 and LowPriorityIO: true

The plist has these three set together.

<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>10</integer>
<key>ProcessType</key>
<string>Background</string>

It's the correct setting for a design that doesn't pressure the system, but it has side effects. When high-load processes like dasd or ComfyUI are running, macOS aggressively defers nice-10 jobs. You get a state where "work that normally finishes in 5–8 minutes takes 12–14 minutes," and the headroom on TIMEOUT_SEC=900 (15 minutes) effectively shrinks to 1–3 minutes. Timeout values need headroom that factors in nice-induced delay. This is often the cause behind the symptom "the settings are right but timeouts keep happening."

There's intent behind the zombie-guard cleanup timing

Inside acquire_guard(), when the guard directory exists but the contents of the PID file are invalid, cleanup waits until attempt reaches 20 or more (script lines 110–126).

case "$guard_pid" in
  ''|*[!0-9]*)
    if [ "$attempt" -ge 20 ]; then
      rm -f "$GUARD_DIR/pid"
      rmdir "$GUARD_DIR" 2>/dev/null || true
    fi

The reason it doesn't clean up immediately from attempt=0 is to protect "the tiny window between acquiring the guard with mkdir and writing with printf '%s\n' "$$" > "$GUARD_DIR/pid"." If another process peeks during that gap, the PID file looks empty or nonexistent. It waits about 1 second (0.05s × 20) before deciding it's "genuinely an orphaned zombie guard." Not knowing this intent, I started debugging "why can't I take the guard immediately?" and burned 30 minutes.

Specifying only one of --group and --group-max exits 1

The argument-parsing section of the script (lines 233–239) has the following interdependency check.

if [ -n "$GROUP" ] && [ -z "$GROUP_MAX" ]; then
  echo "--group requires --group-max" >&2
  exit 1
fi
if [ -z "$GROUP" ] && [ -n "$GROUP_MAX" ]; then
  echo "--group-max requires --group" >&2
  exit 1
fi

Write only --group engage and forget --group-max 3, and the script dies with exit 1. launchd records this as an error exit and may apply a ThrottleInterval penalty. The error message itself is clear, but you won't notice until you check ~/dev/social-autolike/logs/ig-1.launchd.log. Setting only BROWSER_SLOT_GROUP in the plist's EnvironmentVariables and forgetting BROWSER_SLOT_GROUP_MAX produces the same symptom.

SLOT_RESERVE_COUNT >= SLOT_MAX makes the effective cap 1

In the computation effective_max = SLOT_MAX - SLOT_RESERVE_COUNT, mistakenly setting SLOT_RESERVE_COUNT at or above SLOT_MAX drives the effective cap to zero or below. The script has the fallback [ "$effective_max" -lt 1 ] && effective_max=1, so it doesn't stop completely, but you end up in a state where "the entire engage group is limited to 1 slot." Skips spike, but since it returns exit 0 to launchd, you won't find out unless you aggregate slot.log. Always set SLOT_RESERVE_COUNT to a value smaller than SLOT_MAX.

The character set usable in slot labels is limited

In the script's argument validation (lines 207–209), a label containing anything other than alphanumerics, dots, underscores, colons, and hyphens dies with exit 1.

case "$SLOT_LABEL" in
  *[!A-Za-z0-9._:-]*) echo "invalid slot label: $SLOT_LABEL" >&2; exit 1 ;;
esac

Because the lock filename becomes $SLOT_LABEL.lock, the design rejects characters that would cause filesystem trouble. I got stuck trying to use an account identifier containing an at sign, like @ig_account, directly as a label. You need to convert it to a simple identifier like ig-1 before passing it.

The morning "7 simultaneous timeouts" is hard to avoid by plist design

Looking at StartCalendarInterval in com.lily.autolike.ig-1.plist, this job launches daily at 6:36 (Hour=6, Minute=36). If multiple engage jobs are set to the same Minute=36, launchd tries to start them simultaneously. launchd does not guarantee start order.

If 7 engage-family jobs all start at 6:36 and each holds its slot until reaching the TIMEOUT_SEC=900 limit, then for the 75 minutes from 6:36 to 7:51, all 6 slots are occupied by processes that are "running but effectively doing nothing." You could also solve this with a design change that staggers start times by 2–5 minutes per job, but against the cost of rewriting every plist, the reserved slot — which takes just two env variables (BROWSER_SLOT_RESERVED_GROUPS=post and BROWSER_SLOT_RESERVE=1) — was the cheaper fix.

exit-0 skips are invisible without log aggregation

A skip from failing to get a slot exits 0. Nothing is left in launchd's error log. Unless you aggregate slot.log, you have no idea "how many were skipped today."

For the first week, a 30% skip rate was happening and I thought "no errors in launchd, so it's fine." I later added a cron that runs grep 'result=skip' ~/.cache/lily-browser-slots/slot.log | wc -l every morning, but I should have set that up from day one of the slot management. Automation without monitoring keeps you believing "it's not broken" while it's broken.

Best Practices

Here are guidelines, distilled from the implementation and the failures, for building a semaphore mechanism of this kind.

1. Make every variable overridable via environment variables

Write things in the form SLOT_MAX="${BROWSER_SLOT_MAX:-6}" and you can change behavior from launchd's EnvironmentVariables block without touching the script at all. No more editing and re-copying the file every time you change the cap. You can flexibly handle requirements like "I want the cap to be 3 only in this environment" or "I want a longer timeout for this one job." With the four variables BROWSER_SLOT_MAX, BROWSER_SLOT_TIMEOUT_SEC, BROWSER_SLOT_RESERVED_GROUPS, and BROWSER_SLOT_RESERVE in place, per-environment tuning is complete with zero changes to the script body.

2. Skips should exit 0 so launchd doesn't see an error

"We passed this time" is not a job failure. launchd records a non-zero exit code as a failure and may apply a ThrottleInterval penalty. Design skips to exit 0 and leave the human-facing record in slot.log. You need two layers of recording: "normal exit for launchd" and "details in slot.log."

3. Write the measurement date and concrete numbers in comments

# 実測(2026-08-09): Chrome系ジョブ9個で合計0.7GB。swap枯渇の主犯は dasd(47GB)/ComfyUI(12GB)
SLOT_MAX="${BROWSER_SLOT_MAX:-6}"

Write the measured values and the date next to the variable and you can trace "why this number" later. The comment stands in for a change log, and it also creates motivation to update it at the next measurement. A "number decided by feel" has no update criterion, but a "number decided by measurement" automatically comes with the condition "measure again next time and change it if it exceeds this value."

4. Separate timeout layers explicitly by variable name

The application-side timeout (AUTOLIKE_TIMEOUT_SEC) and the Perl supervisor's timeout (BROWSER_SLOT_TIMEOUT_SEC) serve different roles. You need to write code in run-account.sh that explicitly connects the two.

export BROWSER_SLOT_TIMEOUT_SEC="${AUTOLIKE_TIMEOUT_SEC:-900}"

Without this one line, you stay in the state "I set 3700 in the plist but Perl kills at 900 seconds." For variable handoffs, checking once with printenv "which process does the value I wrote here actually reach" is the reliable move.

5. Build a two-tier structure of group limits and global limits

The global cap (SLOT_MAX=6) alone gives you coarse control. Pass --group engage --group-max 3 and you get fine-grained control: "engage gets at most 3 even when under the global cap." Effective when you want to prevent jobs of the same kind from over-paralleling and hitting rate limits. Because group limits are evaluated independently of the global limit, you can design it so that in a state of "2 engage, 1 note, 3 total, with global cap 6 to spare," only the individual group limit fires first.

6. Add reserved slots only when jobs of differing weight are mixed

SLOT_RESERVED_GROUPS=post and SLOT_RESERVE_COUNT=1 reserve one slot exclusively for the post group. Treat "a like that can be recovered the next day" and "a post that can never be filled once its time slot is gone" as equals, and posts get skipped repeatedly. If job weights are uniform, adding reserved slots only complicates the configuration. Introduce it only "when jobs that must not be skipped are mixed with jobs that can be covered on the next run."

7. Randomize wait intervals to avoid the thundering herd

retry_delay=$((15 + RANDOM % 31))

Retry on a fixed interval and, when multiple jobs are waiting simultaneously, they all fight over the guard at the same moment. Only one process can take the guard with mkdir, so the rest are rejected and retry again after the same fixed interval. This collision repeats periodically and throughput drops. Randomizing to 15–45 seconds spreads out retry timing and produces a flow where slots fill up in sequence.

8. Always include a comparison against the remaining wait time

remaining=$((WAIT_SEC - waited))
if [ "$retry_delay" -gt "$remaining" ]; then
  retry_delay="$remaining"
fi

This prevents a "retry in 30 seconds" setting when only 5 seconds remain on the wait timeout. Without this comparison, you miss opportunities where "one more attempt was worth it" while the remaining time runs out.

9. Killing an entire process tree requires Perl or Python

The shell's timeout command only signals the direct child process. Chrome spawns many descendant processes, such as renderer and GPU processes. Kill only the parent with timeout and grandchildren and below survive, perpetuating the state "Chrome is already running at the next launch." You need an implementation that, like Perl's descendants() function, builds a PID-to-PPID table with /bin/ps -axo pid=,ppid=, enumerates all descendants with a BFS, and sends SIGTERM → SIGKILL. The reason for choosing Perl is that it ships with macOS by default.

10. Write three lines — PID, group name, epoch seconds — to the lock file

printf '%s\n%s\n%s\n' "$$" "$GROUP" "$(date +%s)" > "$LOCK_FILE"

The PID on line 1 lets you do liveness checks with kill -0. The group name on line 2 lets you count for group limits. The epoch seconds on line 3 let you debug "when did this slot start." Just cat the lock file and you know everything about that slot's state. Giving the lock file the group information removes the need to query other running processes.

11. Always include zombie-guard cleanup

If the process holding the guard dies abnormally, the guard directory sticks around forever. The next process can't take the guard, and all slot management stops. Without logic that checks the guard's PID with kill -0 and cleans it up with rmdir if dead, you get an implementation with zero fault tolerance where "the moment one process crashes, every job jams." Guard cleanup is the single most important part of self-healing.

12. Set up slot.log aggregation from day one of the automation

# cronで毎朝実行
skip_count=$(grep "result=skip:global-limit" ~/.cache/lily-browser-slots/slot.log | wc -l)
echo "today skip:global-limit count = $skip_count"

Look only at launchd's logs and everything appears "normal, all exit 0." The skip rate of slot management is only knowable by aggregating slot.log. To prevent the situation "92 skips a day were happening and I didn't notice for a week," automate the aggregation from the start. A skip rate above 15% is a sign to revisit the cap or the reserved slots.

13. Stagger start times across jobs by a few minutes each

If multiple jobs share the same Minute value in StartCalendarInterval, launchd starts them simultaneously. When simultaneously started jobs enter a timeout race, all 6 slots stay occupied for a long time. Just staggering each job's Minute by 2–5 minutes naturally distributes the contention over slots. Combined with reserved slots, post-lane jams drop further. If you have many plist entries there's a bulk-rewrite cost, but it's worth keeping in mind every time you add a new job.

Summary

There were three miscalculations before browser-slot.sh reached its current form.

The first was the hypothesis that "Chrome is heavy." It's preserved verbatim in the script's comments. A single vm_stat revealed the culprits were dasd (47GB) and ComfyUI (12GB), and the 0.7GB total across 9 Chrome instances was noise. The 92 skips a day the cap of 3 was producing was a problem solvable with 5 minutes of measurement.

The second was the post-lane jam after raising the cap back to 6. Because engage and post were treated as equals, simultaneous morning timeouts caused 9 consecutive post skips. "A like that can be recovered on the next run" and "a post that can't be recovered once its time slot is gone" should never have competed under the same slot cap. The reserved slot — two env variables — was the solution, and the script change was just adding is_reserved_group() and inserting 4 lines into acquire_slot().

The third was managing the timeout variable in two places. The AUTOLIKE_TIMEOUT_SEC=3700 written in the plist wasn't reaching the Perl supervisor's BROWSER_SLOT_TIMEOUT_SEC, and it ran that way for several days. A one-line handoff in run-account.sh solved it, but the symptom "I configured it but it isn't taking effect" is especially slow to discover because the logs don't honestly tell you the cause.

What all three share is the pattern "configure by intuition → no measurement → quiet malfunction." Chrome's weight, slot contention, the timeout connection — every one presented as "running but not actually functioning," and none surfaced until I aggregated the logs.

Automation is something that "breaks in production after the code works." The quieter the failure mode, the more measurement and log aggregation become the only diagnostic tools. Deciding up front "what to measure" and "what to aggregate" is a shorter path to automation that runs stably for a long time than the semaphore mechanism itself.

The full picture of the system, the breakdown behind ¥1.2M/month, and the 30-day walkthrough are collected in a paid note.
📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

DEV Community: Lily

The Morning My cron Jobs Went Silent: A 97-Line Script That Migrated Everything to launchd

Why This Approach Works

"It Should Be Running" Is the Most Dangerous Kind of Confidence

Why launchd Is the Right Answer

What It Means to Invest in the Environment, Not the Work

The Overall Flow

Map of the Processing

Dissecting the Script (All 97 Lines)

Implementation Details

set -uo pipefail — Why -e Was Left Out

The Label-Generation Regex and Its Absolute-Path Dependency

How the */N Format Breaks — Traced Through the Code

The Intent and Limits of the /bin/zsh -c Wrapper

Where I Got Stuck

Stuck 1: */15 * * * * Failed Silently

Stuck 2: ~ Paths in the crontab Caused Label Collisions and Overwrote Old plists

Stuck 3: node and python Were command not found

Stuck 4: cron + launchd Were Double-Running the Same Script

Stuck 5: The plist I Checked in dry Was a Different File From the One apply Deployed

Pitfalls

What Can and Can't Be Automated

Pitfall List (With Real Code)

Best Practices

Summary

28 Hours of Green Logs, Zero Replies: How a Single `echo` Line Swallowed Every Exit Code

Why This Mechanism Works

Scenarios Readers Will Hit

Not "No Failure Occurred," but "The Failure Code Never Arrived"

The Overall Flow

The Full Call Chain

The Actual Code of run-with-flags-shell.sh

What Actually Happens Along the Chain

The Actual Incident Numbers

What's Happening at the Code Level

Implementation Details

Why set -euo pipefail Alone Doesn't Save the "Outside"

Pattern 1: Save $? Before Any $(...)

Pattern 2: Use PIPESTATUS to Capture Every Command's Code

Pattern 3: Rewrite the JS Dispatcher to Call Directly

Verify the Guard Lands With a Minimal exit 1 Repro

Where I Got Stuck

Case 1: 28 Hours of All-Green Logs, Zero Actual Replies

Case 2: The note Paid-Bonus ZIP Attachment Failure Went Unnoticed for 3 Weeks

Case 3: YouTrust Was Failing in "Complete Silence" — A Library's process.exit()

The Common Pattern Across All 3

Pitfalls

Best Practices

Summary

8 Runs a Day Became 3: Building a launchd plist Integrity Guard in 73 Lines

Why this mechanism works

The problem: you can't tell when automation has stopped

The value of designing for "record," not just "restore"

The structural weak point in a launchd plist

The overall flow

Architecture overview

Core logic: the plist reading part

Core logic: generating and comparing expected values

Core logic: the restore part

Implementing the forensic log

Implementation details

set -uo pipefail — why I dropped -e

Explicit PATH and the log function

Guarding against a missing plist

Naming the backups

What the self-repair job's plist teaches you

Where I got stuck

Snag ①: the python heredoc broke from variable expansion

Snag ②: StartCalendarInterval returns a dict when there's a single entry

Snag ③: not knowing ${expected:+,} and getting a leading comma every time

Snag ④: launchctl bootstrap returned an error immediately

Snag ⑤: why rest2 was needed in read -r p rest2 in the ps pipe

Gotchas

Best practices

Wrapping up

Broken Retries: Why 'Just Try Again' Kept 5 Automation Lanes Dead for 63 Minutes

Why this design works

The overall flow

(1) Wait — the skeleton of the implementation

(2) Give up — always place a terminator

`set -uo pipefail` — Why `-e` Was Left Out

How the `*/N` Format Breaks — Traced Through the Code

The Intent and Limits of the `/bin/zsh -c` Wrapper

Stuck 1: `/15 * * *` Failed Silently

Stuck 2: `~` Paths in the crontab Caused Label Collisions and Overwrote Old plists

Stuck 3: node and python Were `command not found`

Stuck 5: The plist I Checked in `dry` Was a Different File From the One `apply` Deployed

The Actual Code of `run-with-flags-shell.sh`

Why `set -euo pipefail` Alone Doesn't Save the "Outside"

Pattern 1: Save `$?` Before Any `$(...)`

Pattern 2: Use `PIPESTATUS` to Capture Every Command's Code

Verify the Guard Lands With a Minimal `exit 1` Repro

Case 3: YouTrust Was Failing in "Complete Silence" — A Library's `process.exit()`

`set -uo pipefail` — why I dropped `-e`

Snag ②: `StartCalendarInterval` returns a dict when there's a single entry

Snag ③: not knowing `${expected:+,}` and getting a leading comma every time

Snag ④: `launchctl bootstrap` returned an error immediately

Snag ⑤: why `rest2` was needed in `read -r p rest2` in the ps pipe

Why You Must Never Mix `count_today()` and `has_today()`

`count_dead_runs()` — Catching Runs That Finished but Produced Nothing

The Three-Stage Pipeline of `sum_likes_today()`

What `trap cleanup EXIT` protects

Before I added `set -euo pipefail`

Before I added `--color never`