DEV Community

Mashi Mashi
Mashi Mashi

Posted on

launchd is not cron: five failure modes that silently killed my daily jobs

#macos #automation #devops #productivity

I run about 40 scheduled jobs on a Mac mini: content pipelines, backups, PDCA report generators, and a few posting bots. All of them are macOS launchd agents in ~/Library/LaunchAgents.

Coming from cron, I assumed launchd was cron with XML. It isn't. Over the past months every one of these bit me, and in each case the job did not report an error in an obvious place. Here they are, with the fix.

1. exit 78 (EX_CONFIG) means launchd could not open your log file

One morning three jobs stopped. No log output at all — not a single line, not even a partial run. Running the same script by hand worked perfectly.

The cause: a cleanup had moved those project directories onto an external volume and left symlinks behind. Their plists still pointed StandardOutPath at a path that now resolved under /Volumes/....

launchd refuses to start a process when it cannot open the redirect targets, and it dies before your script exists. That is what exit 78 means. Because the failure is before exec, there is nothing in the log — the log is the thing that failed.

<!-- bad: log target can disappear when the volume unmounts -->
<key>StandardOutPath</key>
<string>/Volumes/BigDisk/project/logs/job.log</string>

<!-- good: internal disk, always mounted -->
<key>StandardOutPath</key>
<string>/Users/me/Library/Logs/MyBrand/job.log</string>
Enter fullscreen mode Exit fullscreen mode

Rule I now follow: launchd's own redirect always goes to the internal disk. The script's own detailed log can live wherever the project lives.

2. The plist filename and the Label key are different things

launchctl list matches the Label key inside the plist, not the filename. I once spent half an hour concluding a job was "not loaded" because I grepped for the filename. It had been running fine for weeks under a different label.

# what you actually want
/usr/libexec/PlistBuddy -c 'Print :Label' ~/Library/LaunchAgents/whatever.plist
launchctl list | grep "$LABEL"
Enter fullscreen mode Exit fullscreen mode

3. launchd gives you almost no PATH

A launchd job runs with a minimal environment. node, python3, pnpm installed via Homebrew or a version manager are simply not on PATH.

The nasty part is what that looks like downstream. My script called node bare, got command not found, and the wrapper interpreted the empty output as "nothing to do yet" — so the run was reported as waiting, not failing. Every single item was skipped for days while the job reported success.

Two fixes, both required:

# 1. give the job a real PATH
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin"

# 2. fail closed - never let a missing binary look like "idle"
command -v node >/dev/null || { echo "FATAL: node not found" >&2; exit 1; }
Enter fullscreen mode Exit fullscreen mode

If a job can report "nothing to do", make sure the only way to reach that state is a real empty queue.

4. A stale-lock timeout shorter than the job destroys live locks

Several of my jobs drive one browser profile, so they take a mutex. The original implementation treated a lock older than 60 seconds as stale and broke it.

The jobs routinely hold the lock for 5–20 minutes. So the second job would break a perfectly live lock, both processes would drive the same browser, and they killed each other with errors like Target page, context or browser has been closed. It looked like a browser bug. It was a lock bug.

The fix that made it stop:

  • the holder touches the lock file every 20 seconds (a heartbeat)
  • "stale" means no heartbeat for 120 seconds, not "created 60 seconds ago"
  • the waiter waits up to 30 minutes instead of 5 seconds — it queues instead of stealing

5. Exact scheduled times look like a bot

All of my posting jobs used to fire on round numbers: 07:00, 11:00, 19:00, every day, to the second. One account was flagged the day after it started this pattern.

Now each job sleeps a random 0–15 minutes at the start — but only when launchd started it:

post_time_jitter() {
  [ -t 1 ] && return 0            # interactive run: don't wait
  [ "${DRY_RUN:-0}" = "1" ] && return 0
  sleep $(( RANDOM % ${JITTER_MAX_SEC:-900} ))
}
Enter fullscreen mode Exit fullscreen mode

Side benefit: jobs that used to fire at exactly the same minute now spread out, which relieved the lock contention from #4.

The debug loop I use now 

# every non-Apple job with a non-zero last exit status
launchctl list | grep -v apple | awk '$2 != 0'
Enter fullscreen mode Exit fullscreen mode

Then read StandardErrorPath from the plist rather than from memory, because it moves.

One of the jobs on this box publishes five short daily readings for 星詠み AI占い, a Japanese astrology and tarot site — five renders and five uploads, every morning, unattended. It has been the best stress test for all of the above: anything that fails silently shows up as a missing post the next day, which is very hard to ignore.

If you take one thing from this: launchd's silent failures are almost always about environment, paths, and locks — not about your code. Make each of those loud.

Top comments (0)