DEV Community

Amre
Amre

Posted on Originally published at thesolai.github.io

Build a Self-Healing Content Pipeline with macOS LaunchAgents

#ai

A scheduled job that quietly stops is worse than one that crashes loudly: you keep assuming the pipeline is running while your content quietly goes stale.

For macOS users, cron is often the first choice for recurring scripts, but it has a dangerous weakness. A failed job can disappear into the background without notifying you, and a LaunchAgent can be unloaded without any obvious warning. The result is a content pipeline that appears healthy while producing nothing.

This tutorial builds a self-healing pipeline with macOS launchd and LaunchAgents. The setup combines:

  • A scheduled worker for your content script.
  • KeepAlive so launchd restarts a worker after a failed run.
  • A watchdog LaunchAgent that detects missing or unhealthy workers.
  • Idempotent scripts so restarting a job does not create duplicate content.
  • Log files and health markers that make failures visible.

By the end, you will have a pipeline that recovers from common failures and gives you a clear signal when something needs attention.

Prerequisites

You will need:

  • macOS, where launchd manages background agents.
  • A script you want to run on a schedule, such as a Python, Bash, or Ruby script.
  • Basic familiarity with the command line and shell scripting.
  • A user account with permission to create files in your home directory.

The examples use these paths:

~/scripts/site-cron.py
~/scripts/site-watchdog.sh
~/Library/LaunchAgents/com.example.site-cron.plist
~/Library/LaunchAgents/com.example.site-watchdog.plist
~/Library/Logs/site-cron/
Enter fullscreen mode Exit fullscreen mode

Replace username in the examples with the username on your Mac.

Step 1: Start with a simple LaunchAgent

Create a plist file in ~/Library/LaunchAgents/. The following LaunchAgent runs a Python script every day at 9:00 AM:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.site-cron</string>

<key>ProgramArguments</key>
    <array>
        <string>/usr/bin/python3</string>
        <string>/Users/username/scripts/site-cron.py</string>
    </array>

<key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>9</integer>
        <key>Minute</key>
        <integer>0</integer>
    </dict>

<key>StandardOutPath</key>
    <string>/Users/username/Library/Logs/site-cron/worker.out.log</string>

<key>StandardErrorPath</key>
    <string>/Users/username/Library/Logs/site-cron/worker.err.log</string>
</dict>
</plist>
Enter fullscreen mode Exit fullscreen mode

Create the log directory before loading the agent:

mkdir -p ~/Library/Logs/site-cron
Enter fullscreen mode Exit fullscreen mode

The important fields are:

  • Label: A unique identifier for the agent.
  • ProgramArguments: The command and its arguments.
  • StartCalendarInterval: A cron-like schedule.
  • StandardOutPath: Where normal output is written.
  • StandardErrorPath: Where errors are written.

Load the agent into your user session:

launchctl bootstrap gui/$(id -u) \
  ~/Library/LaunchAgents/com.example.site-cron.plist
Enter fullscreen mode Exit fullscreen mode

The gui/$(id -u) portion identifies the current user's graphical login domain.

Verify that it is loaded:

launchctl list | grep com.example.site-cron
Enter fullscreen mode Exit fullscreen mode

You can also inspect the full launchd state:

launchctl print gui/$(id -u)/com.example.site-cron
Enter fullscreen mode Exit fullscreen mode

Step 2: Why the naive setup fails

Suppose your script has an import error:

# ~/scripts/site-cron.py
import nonexistent

print("Site cron running")
Enter fullscreen mode Exit fullscreen mode

The job fails immediately. The error may be written to worker.err.log, but nothing tells you that the pipeline has stopped.

There are two separate failure modes to address:

  1. The process exits unsuccessfully.

    launchd should restart the worker automatically.

  2. The agent is no longer loaded.

    A watchdog should detect the missing agent and load it again.

A reliable solution needs to handle both cases.

Step 3: Make the worker restartable

Add KeepAlive to the worker plist:

<key>KeepAlive</key>
<dict>
    <key>SuccessfulExit</key>
    <false/>
</dict>

<key>ThrottleInterval</key>
<integer>30</integer>
Enter fullscreen mode Exit fullscreen mode

The resulting plist looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.site-cron</string>

<key>ProgramArguments</key>
    <array>
        <string>/usr/bin/python3</string>
        <string>/Users/username/scripts/site-cron.py</string>
    </array>

<key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>9</integer>
        <key>Minute</key>
        <integer>0</integer>
    </dict>

<key>RunAtLoad</key>
    <true/>

<key>KeepAlive</key>
    <dict>
        <key>SuccessfulExit</key>
        <false/>
    </dict>

<key>ThrottleInterval</key>
    <integer>30</integer>

<key>StandardOutPath</key>
    <string>/Users/username/Library/Logs/site-cron/worker.out.log</string>

<key>StandardErrorPath</key>
    <string>/Users/username/Library/Logs/site-cron/worker.err.log</string>
</dict>
</plist>
Enter fullscreen mode Exit fullscreen mode

KeepAlive tells launchd to restart the job whenever it exits unsuccessfully. ThrottleInterval prevents a broken script from consuming resources in a tight restart loop.

Reload the agent after changing its plist:

launchctl bootout gui/$(id -u)/com.example.site-cron
launchctl bootstrap gui/$(id -u) \
  ~/Library/LaunchAgents/com.example.site-cron.plist
Enter fullscreen mode Exit fullscreen mode

You can test the restart behavior by temporarily making the script exit with a nonzero status:

python3 -c 'import sys; sys.exit(1)'
Enter fullscreen mode Exit fullscreen mode

The agent should be restarted by launchd rather than remaining failed and inactive.

Step 4: Add a health marker

A restart policy protects against process failures, but it does not tell you whether the script is successfully doing its work. Your script might start successfully, return zero, and still fail to publish the expected content.

Have the worker write a timestamp after each successful run:

# ~/scripts/site-cron.py
from datetime import datetime, timezone
from pathlib import Path
import os

LOG_DIR = Path.home() / "Library" / "Logs" / "site-cron"
HEALTH_FILE = LOG_DIR / "last-success"

LOG_DIR.mkdir(parents=True, exist_ok=True)

# Run your content pipeline here.
print("Site cron running")

timestamp = datetime.now(timezone.utc).isoformat()
temporary_file = HEALTH_FILE.with_suffix(".tmp")

temporary_file.write_text(timestamp + "\n")
os.replace(temporary_file, HEALTH_FILE)
Enter fullscreen mode Exit fullscreen mode

The health file should only be updated after the useful work has completed. Otherwise, a script can look healthy while producing broken output.

Step 5: Build the watchdog

The watchdog checks two things:

  • Whether the worker LaunchAgent is still loaded.
  • Whether the worker has completed successfully recently.

Create ~/scripts/site-watchdog.sh:

#!/bin/bash

set -u

UID_VALUE="$(id -u)"
DOMAIN="gui/${UID_VALUE}"
LABEL="com.example.site-cron"
PLIST="${HOME}/Library/LaunchAgents/${LABEL}.plist"
LOG_DIR="${HOME}/Library/Logs/site-cron"
HEALTH_FILE="${LOG_DIR}/last-success"
MAX_AGE_SECONDS=90000

mkdir -p "${LOG_DIR}"

log() {
    printf '%s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*"
}

if ! launchctl print "${DOMAIN}/${LABEL}" >/dev/null 2>&1; then
    log "Worker is not loaded; bootstrapping ${LABEL}"

if launchctl bootstrap "${DOMAIN}" "${PLIST}"; then
        log "Worker loaded successfully"
    else
        log "ERROR: could not load worker"
        exit 1
    fi

exit 0
fi

if [[ ! -f "${HEALTH_FILE}" ]]; then
    log "No health marker found; requesting a worker run"
    launchctl kickstart -k "${DOMAIN}/${LABEL}"
    exit $?
fi

now="$(date +%s)"
last_success="$(stat -f %m "${HEALTH_FILE}")"
age=$((now - last_success))

if (( age > MAX_AGE_SECONDS )); then
    log "Worker health marker is stale (${age}s); restarting worker"
    launchctl kickstart -k "${DOMAIN}/${LABEL}"
else
    log "Worker is loaded and healthy (last success ${age}s ago)"
fi
Enter fullscreen mode Exit fullscreen mode

Make it executable:

chmod +x ~/scripts/site-watchdog.sh
Enter fullscreen mode Exit fullscreen mode

The MAX_AGE_SECONDS value should be longer than the normal interval between successful runs. The example uses roughly 25 hours for a daily job. If your job runs every hour, use a smaller threshold, such as 7200 seconds.

The watchdog should only restart the worker when it has evidence that the worker is unhealthy. Restarting a healthy job every few minutes can cause unnecessary work and duplicate side effects.

Step 6: Run the watchdog as a LaunchAgent

Create ~/Library/LaunchAgents/com.example.site-watchdog.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.site-watchdog</string>

<key>ProgramArguments</key>
    <array>
        <string>/bin/bash</string>
        <string>/Users/username/scripts/site-watchdog.sh</string>
    </array>

<key>StartInterval</key>
    <integer>300</integer>

<key>RunAtLoad</key>
    <true/>

<key>StandardOutPath</key>
    <string>/Users/username/Library/Logs/site-cron/watchdog.out.log</string>

<key>StandardErrorPath</key>
    <string>/Users/username/Library/Logs/site-cron/watchdog.err.log</string>
</dict>
</plist>
Enter fullscreen mode Exit fullscreen mode

This runs the watchdog every five minutes and once immediately after loading.

Load it:

launchctl bootstrap gui/$(id -u) \
  ~/Library/LaunchAgents/com.example.site-watchdog.plist
Enter fullscreen mode Exit fullscreen mode

Verify both agents:

launchctl list | grep com.example.site-cron
Enter fullscreen mode Exit fullscreen mode

The output should contain entries for the worker and watchdog.

Step 7: Make your scripts idempotent

Self-healing means jobs may run more than once. A retry should not publish the same article twice, duplicate records, or overwrite valid output with stale data.

Idempotency means that running the same operation multiple times produces the same correct result as running it once.

For example, your publishing script might use a stable content ID:

content_id = "post-2025-01-15"

# Use the ID as an idempotency key when writing to your database
# or publishing to your CMS.
publish(content_id, title="My article")
Enter fullscreen mode Exit fullscreen mode

Other useful techniques include:

  • Upserting records with a unique key.
  • Checking whether the content already exists before creating it.
  • Writing temporary files and atomically renaming them into place.
  • Using a lock to prevent concurrent runs.
  • Making retries safe by preserving a processing state.

For example, a simple lock can prevent two worker instances from running simultaneously:

import fcntl
from pathlib import Path

lock_path = Path.home() / "Library" / "Logs" / "site-cron" / "worker.lock"

with lock_path.open("w") as lock_file:
    try:
        fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        print("Another worker is already running")
        raise SystemExit(0)

run_content_pipeline()
Enter fullscreen mode Exit fullscreen mode

Step 8: Monitor the pipeline

A self-healing system should still be observable. Check the logs regularly:

tail -f ~/Library/Logs/site-cron/worker.err.log
Enter fullscreen mode Exit fullscreen mode

You can also inspect the worker and watchdog output:

tail -f ~/Library/Logs/site-cron/watchdog.out.log
Enter fullscreen mode Exit fullscreen mode

Check the current state of an agent with:

launchctl print gui/$(id -u)/com.example.site-cron
Enter fullscreen mode Exit fullscreen mode

Useful health checks include:

# Check the last successful run
cat ~/Library/Logs/site-cron/last-success

# Check whether the worker is loaded
launchctl print gui/$(id -u)/com.example.site-cron

# Check the watchdog
launchctl print gui/$(id -u)/com.example.site-watchdog
Enter fullscreen mode Exit fullscreen mode

For production workflows, send an alert when the watchdog detects a missing agent, a stale health marker, or repeated restart failures. A simple command-line notification can be the first step:

osascript -e 'display notification "Content pipeline needs attention"'
Enter fullscreen mode Exit fullscreen mode

Step 9: Test the recovery paths

Do not wait for a real outage to discover that recovery is broken. Test each failure independently.

Test a process failure

Temporarily make the worker fail:

raise RuntimeError("simulated failure")
Enter fullscreen mode Exit fullscreen mode

Confirm that launchd restarts it and that the error appears in the log.

Test an unloaded agent

Unload the worker:

launchctl bootout gui/$(id -u)/com.example.site-cron
Enter fullscreen mode Exit fullscreen mode

The watchdog should detect that it is missing and bootstrap it again.

Test a stale health marker

Set the health marker to an old timestamp:

touch -t 202001010000 ~/Library/Logs/site-cron/last-success
Enter fullscreen mode Exit fullscreen mode

The watchdog should detect the stale marker and restart the worker.

Test duplicate execution

Run the worker manually several times and confirm that your content is not duplicated:

launchctl kickstart -k gui/$(id -u)/com.example.site-cron
Enter fullscreen mode Exit fullscreen mode

A resilient pipeline is not one that never fails. It is one that fails predictably, recovers automatically, and leaves enough evidence for you to understand what happened.

Start with the basic LaunchAgent, add KeepAlive, then introduce a watchdog and idempotent scripts. That combination gives you a macOS content pipeline that can survive both process failures and silently unloaded agents.

This was first published on Sol AI — https://thesolai.github.io

Top comments (0)