A scheduled job that quietly stops is worse than one that crashes loudly: you keep assuming the pipeline is running while your content quietly goes stale.
For macOS users, cron is often the first choice for recurring scripts, but it has a dangerous weakness. A failed job can disappear into the background without notifying you, and a LaunchAgent can be unloaded without any obvious warning. The result is a content pipeline that appears healthy while producing nothing.
This tutorial builds a self-healing pipeline with macOS launchd and LaunchAgents. The setup combines:
- A scheduled worker for your content script.
-
KeepAlivesolaunchdrestarts a worker after a failed run. - A watchdog LaunchAgent that detects missing or unhealthy workers.
- Idempotent scripts so restarting a job does not create duplicate content.
- Log files and health markers that make failures visible.
By the end, you will have a pipeline that recovers from common failures and gives you a clear signal when something needs attention.
Prerequisites
You will need:
- macOS, where
launchdmanages background agents. - A script you want to run on a schedule, such as a Python, Bash, or Ruby script.
- Basic familiarity with the command line and shell scripting.
- A user account with permission to create files in your home directory.
The examples use these paths:
~/scripts/site-cron.py
~/scripts/site-watchdog.sh
~/Library/LaunchAgents/com.example.site-cron.plist
~/Library/LaunchAgents/com.example.site-watchdog.plist
~/Library/Logs/site-cron/
Replace username in the examples with the username on your Mac.
Step 1: Start with a simple LaunchAgent
Create a plist file in ~/Library/LaunchAgents/. The following LaunchAgent runs a Python script every day at 9:00 AM:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.site-cron</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/python3</string>
<string>/Users/username/scripts/site-cron.py</string>
</array>
<key>StartCalendarInterval</key>
<dict>
<key>Hour</key>
<integer>9</integer>
<key>Minute</key>
<integer>0</integer>
</dict>
<key>StandardOutPath</key>
<string>/Users/username/Library/Logs/site-cron/worker.out.log</string>
<key>StandardErrorPath</key>
<string>/Users/username/Library/Logs/site-cron/worker.err.log</string>
</dict>
</plist>
Create the log directory before loading the agent:
mkdir -p ~/Library/Logs/site-cron
The important fields are:
-
Label: A unique identifier for the agent. -
ProgramArguments: The command and its arguments. -
StartCalendarInterval: A cron-like schedule. -
StandardOutPath: Where normal output is written. -
StandardErrorPath: Where errors are written.
Load the agent into your user session:
launchctl bootstrap gui/$(id -u) \
~/Library/LaunchAgents/com.example.site-cron.plist
The gui/$(id -u) portion identifies the current user's graphical login domain.
Verify that it is loaded:
launchctl list | grep com.example.site-cron
You can also inspect the full launchd state:
launchctl print gui/$(id -u)/com.example.site-cron
Step 2: Why the naive setup fails
Suppose your script has an import error:
# ~/scripts/site-cron.py
import nonexistent
print("Site cron running")
The job fails immediately. The error may be written to worker.err.log, but nothing tells you that the pipeline has stopped.
There are two separate failure modes to address:
The process exits unsuccessfully.
launchdshould restart the worker automatically.The agent is no longer loaded.
A watchdog should detect the missing agent and load it again.
A reliable solution needs to handle both cases.
Step 3: Make the worker restartable
Add KeepAlive to the worker plist:
<key>KeepAlive</key>
<dict>
<key>SuccessfulExit</key>
<false/>
</dict>
<key>ThrottleInterval</key>
<integer>30</integer>
The resulting plist looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.site-cron</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/python3</string>
<string>/Users/username/scripts/site-cron.py</string>
</array>
<key>StartCalendarInterval</key>
<dict>
<key>Hour</key>
<integer>9</integer>
<key>Minute</key>
<integer>0</integer>
</dict>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<dict>
<key>SuccessfulExit</key>
<false/>
</dict>
<key>ThrottleInterval</key>
<integer>30</integer>
<key>StandardOutPath</key>
<string>/Users/username/Library/Logs/site-cron/worker.out.log</string>
<key>StandardErrorPath</key>
<string>/Users/username/Library/Logs/site-cron/worker.err.log</string>
</dict>
</plist>
KeepAlive tells launchd to restart the job whenever it exits unsuccessfully. ThrottleInterval prevents a broken script from consuming resources in a tight restart loop.
Reload the agent after changing its plist:
launchctl bootout gui/$(id -u)/com.example.site-cron
launchctl bootstrap gui/$(id -u) \
~/Library/LaunchAgents/com.example.site-cron.plist
You can test the restart behavior by temporarily making the script exit with a nonzero status:
python3 -c 'import sys; sys.exit(1)'
The agent should be restarted by launchd rather than remaining failed and inactive.
Step 4: Add a health marker
A restart policy protects against process failures, but it does not tell you whether the script is successfully doing its work. Your script might start successfully, return zero, and still fail to publish the expected content.
Have the worker write a timestamp after each successful run:
# ~/scripts/site-cron.py
from datetime import datetime, timezone
from pathlib import Path
import os
LOG_DIR = Path.home() / "Library" / "Logs" / "site-cron"
HEALTH_FILE = LOG_DIR / "last-success"
LOG_DIR.mkdir(parents=True, exist_ok=True)
# Run your content pipeline here.
print("Site cron running")
timestamp = datetime.now(timezone.utc).isoformat()
temporary_file = HEALTH_FILE.with_suffix(".tmp")
temporary_file.write_text(timestamp + "\n")
os.replace(temporary_file, HEALTH_FILE)
The health file should only be updated after the useful work has completed. Otherwise, a script can look healthy while producing broken output.
Step 5: Build the watchdog
The watchdog checks two things:
- Whether the worker LaunchAgent is still loaded.
- Whether the worker has completed successfully recently.
Create ~/scripts/site-watchdog.sh:
#!/bin/bash
set -u
UID_VALUE="$(id -u)"
DOMAIN="gui/${UID_VALUE}"
LABEL="com.example.site-cron"
PLIST="${HOME}/Library/LaunchAgents/${LABEL}.plist"
LOG_DIR="${HOME}/Library/Logs/site-cron"
HEALTH_FILE="${LOG_DIR}/last-success"
MAX_AGE_SECONDS=90000
mkdir -p "${LOG_DIR}"
log() {
printf '%s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*"
}
if ! launchctl print "${DOMAIN}/${LABEL}" >/dev/null 2>&1; then
log "Worker is not loaded; bootstrapping ${LABEL}"
if launchctl bootstrap "${DOMAIN}" "${PLIST}"; then
log "Worker loaded successfully"
else
log "ERROR: could not load worker"
exit 1
fi
exit 0
fi
if [[ ! -f "${HEALTH_FILE}" ]]; then
log "No health marker found; requesting a worker run"
launchctl kickstart -k "${DOMAIN}/${LABEL}"
exit $?
fi
now="$(date +%s)"
last_success="$(stat -f %m "${HEALTH_FILE}")"
age=$((now - last_success))
if (( age > MAX_AGE_SECONDS )); then
log "Worker health marker is stale (${age}s); restarting worker"
launchctl kickstart -k "${DOMAIN}/${LABEL}"
else
log "Worker is loaded and healthy (last success ${age}s ago)"
fi
Make it executable:
chmod +x ~/scripts/site-watchdog.sh
The MAX_AGE_SECONDS value should be longer than the normal interval between successful runs. The example uses roughly 25 hours for a daily job. If your job runs every hour, use a smaller threshold, such as 7200 seconds.
The watchdog should only restart the worker when it has evidence that the worker is unhealthy. Restarting a healthy job every few minutes can cause unnecessary work and duplicate side effects.
Step 6: Run the watchdog as a LaunchAgent
Create ~/Library/LaunchAgents/com.example.site-watchdog.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.site-watchdog</string>
<key>ProgramArguments</key>
<array>
<string>/bin/bash</string>
<string>/Users/username/scripts/site-watchdog.sh</string>
</array>
<key>StartInterval</key>
<integer>300</integer>
<key>RunAtLoad</key>
<true/>
<key>StandardOutPath</key>
<string>/Users/username/Library/Logs/site-cron/watchdog.out.log</string>
<key>StandardErrorPath</key>
<string>/Users/username/Library/Logs/site-cron/watchdog.err.log</string>
</dict>
</plist>
This runs the watchdog every five minutes and once immediately after loading.
Load it:
launchctl bootstrap gui/$(id -u) \
~/Library/LaunchAgents/com.example.site-watchdog.plist
Verify both agents:
launchctl list | grep com.example.site-cron
The output should contain entries for the worker and watchdog.
Step 7: Make your scripts idempotent
Self-healing means jobs may run more than once. A retry should not publish the same article twice, duplicate records, or overwrite valid output with stale data.
Idempotency means that running the same operation multiple times produces the same correct result as running it once.
For example, your publishing script might use a stable content ID:
content_id = "post-2025-01-15"
# Use the ID as an idempotency key when writing to your database
# or publishing to your CMS.
publish(content_id, title="My article")
Other useful techniques include:
- Upserting records with a unique key.
- Checking whether the content already exists before creating it.
- Writing temporary files and atomically renaming them into place.
- Using a lock to prevent concurrent runs.
- Making retries safe by preserving a processing state.
For example, a simple lock can prevent two worker instances from running simultaneously:
import fcntl
from pathlib import Path
lock_path = Path.home() / "Library" / "Logs" / "site-cron" / "worker.lock"
with lock_path.open("w") as lock_file:
try:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
print("Another worker is already running")
raise SystemExit(0)
run_content_pipeline()
Step 8: Monitor the pipeline
A self-healing system should still be observable. Check the logs regularly:
tail -f ~/Library/Logs/site-cron/worker.err.log
You can also inspect the worker and watchdog output:
tail -f ~/Library/Logs/site-cron/watchdog.out.log
Check the current state of an agent with:
launchctl print gui/$(id -u)/com.example.site-cron
Useful health checks include:
# Check the last successful run
cat ~/Library/Logs/site-cron/last-success
# Check whether the worker is loaded
launchctl print gui/$(id -u)/com.example.site-cron
# Check the watchdog
launchctl print gui/$(id -u)/com.example.site-watchdog
For production workflows, send an alert when the watchdog detects a missing agent, a stale health marker, or repeated restart failures. A simple command-line notification can be the first step:
osascript -e 'display notification "Content pipeline needs attention"'
Step 9: Test the recovery paths
Do not wait for a real outage to discover that recovery is broken. Test each failure independently.
Test a process failure
Temporarily make the worker fail:
raise RuntimeError("simulated failure")
Confirm that launchd restarts it and that the error appears in the log.
Test an unloaded agent
Unload the worker:
launchctl bootout gui/$(id -u)/com.example.site-cron
The watchdog should detect that it is missing and bootstrap it again.
Test a stale health marker
Set the health marker to an old timestamp:
touch -t 202001010000 ~/Library/Logs/site-cron/last-success
The watchdog should detect the stale marker and restart the worker.
Test duplicate execution
Run the worker manually several times and confirm that your content is not duplicated:
launchctl kickstart -k gui/$(id -u)/com.example.site-cron
A resilient pipeline is not one that never fails. It is one that fails predictably, recovers automatically, and leaves enough evidence for you to understand what happened.
Start with the basic LaunchAgent, add KeepAlive, then introduce a watchdog and idempotent scripts. That combination gives you a macOS content pipeline that can survive both process failures and silently unloaded agents.
This was first published on Sol AI — https://thesolai.github.io
Top comments (0)