The fix had comments. It had logs. It even had an audit note saying "measured 4m00s, exit 0." It still failed every Sunday for three weeks straight, and nobody noticed.
Last time, I wrote about handling external drive mounts and file identity. This one is less flashy but harder to deal with. It's about an automation I thought I had fixed that kept failing silently for three weeks, while the comments and logs saying it was fixed stayed in place.
The job itself is boring. It's npx skills@latest update -g, the CLI that updates Claude Code skills, run weekly by launchd. I had stacked three layers on top of it: a fix for unauthenticated GitHub API rate limits, a guard against hangs, and failure detection. After that I assumed it was fixed. Then I opened the actual logs, and the last three Sunday runs had all failed for the same reason.
The problem: npx skills@latest update -g is fragile against the unauthenticated GitHub API
~/Library/LaunchAgents/com.lily.skills-update.plist is a plist whose only job is to launch the skill update script every Sunday at 4:40.
<key>Label</key><string>com.lily.skills-update</string>
<key>ProgramArguments</key><array><string>/bin/zsh</string><string>-lc</string><string>~/.claude/scripts/skills-auto-update.sh</string></array>
<key>StartCalendarInterval</key><dict><key>Weekday</key><integer>0</integer><key>Hour</key><integer>4</integer><key>Minute</key><integer>40</integer></dict>
The skills-auto-update.sh it calls basically just runs npx skills@latest update -g --yes. The catch is that this CLI routinely hits GitHub's API rate limit when unauthenticated, and depending on network conditions, its fetch can hang. The more skills you have, the more often this happens. Leave it alone and one day you notice nothing has updated in months.
Design: three layers of self-healing
To handle this, the script has three defenses. Here are the relevant parts of the actual file.
# skills CLI は未認証で GitHub API を叩くため、スキル数が増えるとレート上限に当たる。
# gh は認証済み(5000/h)なのでトークンを渡す。取れなくても従来どおり動く。
if command -v gh >/dev/null 2>&1; then
GH_TOKEN="$(gh auth token 2>/dev/null)"
if [[ -n "$GH_TOKEN" ]]; then export GH_TOKEN GITHUB_TOKEN="$GH_TOKEN"; fi
fi
# timeout 上限: npx の network fetch がハングするのを gtimeout で頭打ち(async hook なので非ブロッキング)。
# 45s だった頃はスキル数が増えて毎回タイムアウトし、5,833行の失敗ログを吐きながら
# 一度も更新できていなかった(2026-08-08 監査で実測4分00秒・exit 0 を確認)。
# STAMP は成功時のみ更新する。旧実装は失敗でも touch していたため、
# ネットワーク不調時に「更新が静かに失敗し続ける」状態になっていた(2026-06-11 監査)。
if command -v gtimeout >/dev/null 2>&1; then
if gtimeout 600 "$NPXBIN" skills@latest update -g --yes >/dev/null 2>>"$ERRLOG"; then
touch "$STAMP"
else
echo "[$(date '+%Y-%m-%d %H:%M:%S')] skills update failed (timeout or error) — STAMP not updated, will retry next session" >> "$ERRLOG"
fi
The three layers work like this:
-
Pass
gh auth token: moves from unauthenticated (60/h) to authenticated (5000/h) to avoid the rate limit. -
gtimeout 600: 45 seconds had already proven too short, and runs stalled more as the skill count grew, so the job is capped at 10 minutes. -
Update STAMP only on success: the old implementation ran
toucheven on failure, so "last run" kept advancing when runs failed and problems stayed invisible. To fix this,STAMPis now updated only on success, and failures are written toskills-update.errwith a reason.
According to the comments, the 2026-08-08 audit confirmed "measured 4m00s, exit 0," so this design was verified to work at least once.
Note
It's good design that the code comments record why each fix was made (the failed 45-second timeout, the 5,833-line log, the 2026-08-08 measurement). But those are only facts verified at that point in time. They don't prove the job keeps working afterward. Forget that difference and you'll read the comments and assume it's fixed.
Reading the real logs: "timeout or error" three weeks in a row
That's the design intent. So what actually happened? Here is ~/.claude/logs/skills-update.err:
[2026-09-06 04:40:38] skills update failed (timeout or error) — STAMP not updated, will retry next session
npm warn exec The following package was not found and will be installed: skills@1.5.26
[2026-09-13 05:27:06] skills update failed (timeout or error) — STAMP not updated, will retry next session
npm warn exec The following package was not found and will be installed: skills@1.7.0
[2026-09-20 04:41:35] skills update failed (timeout or error) — STAMP not updated, will retry next session
9/6, 9/13, 9/20: all Sundays, all around the plist's start time (about 4:40). So for these three weeks, the Sunday job started every time, failed every time, and logged the same reason every time. The npm warn exec lines in between show that npx's own bootstrap (fetching skills@1.5.26 → skills@1.7.0) was getting through. So the failure isn't before npx launches. It's further in, in skills update itself, exactly where you'd suspect rate limiting or a network hang.
A contradiction in the STAMP file
Now another real file: ~/.claude/skills-last-update, which should only be updated on success.
birth: Aug 8 11:54:13 2026 / mtime: Sep 10 17:02:10 2026
It was created on 2026-08-08, the same date as the audit behind the "measured 4m00s, exit 0" design comment. That part adds up.
The mtime is the problem: 2026-09-10 17:02. That doesn't match any slot in the plist's schedule (every Sunday at 4:40). It's not a Sunday, and it's not the morning. So the run that last updated this STAMP most likely did not come from launchd's weekly automated run.
Every automated run recorded after that (9/6, 9/13, 9/20) failed. So for at least these three weeks, the automated path alone never succeeded once. The STAMP had stopped moving three weeks earlier, and nobody went to look.
Why the "fix" didn't fix anything
This is the part I most wanted to write about. Each of the three self-healing layers is working as intended.
-
gtimeout 600works: the exit code came back within 600 seconds and the script fell into theelsebranch. If it were truly hanging, the timing of the[timeout]entries would be more irregular, but all three were logged at regular points within 1 to 47 minutes of launch. - Success-only STAMP works: nothing succeeded, so the STAMP didn't advance. That's the correct, designed behavior.
-
echo ... >> "$ERRLOG"works: three weeks of entries are all there.
So none of the safety mechanisms is broken. Yet updates failed three weeks running. What the design changed was going from "a failure tells you nothing" to "you can see when a failure happened and why." It never fixed the failure itself. All the "should work" code really did was turn a silent failure into a logged one. The root cause was untouched.
The most suspicious variable left is gh auth token. The script comment says "it still works as before even if the token can't be obtained." Read the other way, that means nobody checks whether the token was obtained. Running gh auth status locally, I confirmed that besides the authenticated account, there is an account whose keyring access times out. gh auth token itself currently exits normally (exit 0), but nothing guarantees it resolves the same way every time in launchd's non-interactive, 4:40 a.m. context. If keychain access stalls there and GH_TOKEN is passed empty, the script gives no warning, falls back to the unauthenticated path, and hits the same rate limit it was meant to avoid. These logs alone can't prove that, so I'm leaving it as the next hypothesis to dig into.
How this ties to the macos-hook-path-timeout skill
The gtimeout dependency has some history. ~/.claude/skills/auto/macos-hook-path-timeout/SKILL.md is an existing skill that was auto-generated based on this very script.
### 症状2: スクリプトの `timeout 60 ...` が効かない(`timeout: command not found`)
macOSは GNU coreutils の `timeout` を**標準搭載しない**。ECCのhooks.md等が前提にしていても不発。
**修正**: `brew install coreutils` で `gtimeout` が入る。スクリプトは
`command -v gtimeout >/dev/null && TIMEOUT_CMD="gtimeout 60"` の形で優先検出させる
(`post_tsc_check.sh` / `skills-auto-update.sh` はこの形。coreutils導入だけで設計通りに起動する)。
(Translation: Symptom 2: a script's timeout 60 ... doesn't work (timeout: command not found). macOS does not ship GNU coreutils' timeout, so it fails even when docs like ECC's hooks.md assume it exists. Fix: brew install coreutils provides gtimeout. Scripts should detect it first with command -v gtimeout >/dev/null && TIMEOUT_CMD="gtimeout 60". post_tsc_check.sh and skills-auto-update.sh use this form, so installing coreutils alone makes them start as designed.)
This skill's Verification section only goes as far as checking that the gtimeout command exists.
command -v gtimeout && echo "post_tsc_check 等の timeout 分岐が起動する"
So all this skill guarantees is that "if gtimeout is installed, the timeout branch runs." Whether the job actually succeeds once it's in that branch is out of scope. The three-week failure streak happened in exactly that gap: gtimeout started correctly, capped correctly, and logged correctly, and skills update itself still didn't get through.
Pitfalls I hit
- Raising the 45-second timeout to 600 seconds "should have" fixed it, but it didn't fix the network or rate-limit problem itself → A longer timeout only helps with hangs. If the cause is something else (rate limiting, or GH_TOKEN not being obtained), it will fail in the same place however far you extend it.
- Success-only STAMP didn't fix failures. It only made them visible → Unless something regularly reads the logs (cron monitoring, a statusline integration), nobody reads the visible failures and you're back to "nothing has updated in months."
-
Whether
npm warn execshows up tells you where the failure is → If that line is there,npx's bootstrap succeeded, so the cause is insideskills update(network/API side). -
"It still works as before even if the token can't be obtained" is the weakest fallback → An empty
GH_TOKENquietly drops to the unauthenticated path. If you don't log whether auth succeeded, you can't tell when a supposed fix has quietly stopped doing anything. - Compare the STAMP mtime against the cron schedule → If the update time doesn't match the job's start time, the success probably didn't come from the automated path.
- An existing auto-skill's Verification section may only check that a command exists → "Does the command exist?" and "Does the job keep succeeding afterward?" are separate things to verify.
Summary
- All three self-healing layers (GH_TOKEN injection, gtimeout, success-only STAMP) behaved as designed, both in the code and in the run logs.
- Even so, the last three weekly runs (9/6, 9/13, 9/20) all failed for the same reason, and the STAMP stayed stuck at 2026-09-10.
- The safety mechanisms only turned a "silent failure" into a "logged failure." They did not fix the root cause.
- "Should work" (design, past measurements) and "actually worked" (raw logs, STAMP mtime) are different things, and you need to check them against each other regularly.
- Whether
gh auth tokenresolves every time in launchd's non-interactive context is still unverified. That's the next hypothesis to rule out.
Next time, I plan to check whether gh auth token really resolves every time in launchd's context (non-interactive, early morning, keychain not unlocked) and report whether that turned out to be the root cause.
What about you: how do you check that your "fixed" background jobs are still succeeding, and not just logging their failures neatly?
Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*
Top comments (0)