My OpenClaw Gateway Went Down at 3 AM. It Fixed Itself Before I Woke Up.

Something I've learned maintaining OpenClaw agents in production: the agent goes down when you least expect it. Midnight. 3 AM. During a backup. Right before a critical task.

I've tried the naive approach — a simple systemd unit or launchd plist. It works until it doesn't. When the gateway crashes repeatedly, you get a restart loop that hammers your server. When it crashes with a config error, a blind restart just re-crashes. And if you're not watching, you miss it entirely.

So I built a watchdog that handles all of this automatically. Here's what I learned building it.

The Problem: Blind Restarts Make Things Worse

A basic watchdog looks like this in theory: "process is dead → restart it." But in practice, that's not enough.

When my gateway started crashing with a config schema mismatch after an update, the restart loop was immediate:

Crash → restart → crash in 2s → restart → crash in 2s → ...

The crash counter never reset. The gateway never recovered. And I woke up to a system that had been pounding itself for six hours.

The real problems are:

1. No backoff — restart loops happen in seconds, accomplishing nothing
2. No crash decay — a temporary glitch locks you into permanent failure state
3. No error awareness — a config error needs a config fix, not a restart
4. No escalation — some failures need more than a simple restart

The Watchdog Architecture

The watchdog I built runs as a bash script, scheduled every few minutes via launchd. It has four layers:

Layer 1: PID + HTTP Health Check
Layer 2: Exponential Backoff + Crash Counter Decay  
Layer 3: Config Auto-Fix (exit_1 + config error pattern)
Layer 4: Level 3 Emergency Recovery (30min persistent failure)

Layer 1: Detecting Real Failures

The watchdog checks two things: is the launchd process running, and does the gateway respond to HTTP?

check_pid_status() {
    local status=$(launchctl list | grep "$LAUNCHD_SERVICE" || echo "")
    # ...
    if [[ "$pid" != "-" ]] && [[ "$pid" -gt 0 ]]; then
        echo "PID:$pid"
    elif [[ "$exit_code" -lt 0 ]]; then
        echo "CRASHED:signal_$exit_code"
    else
        echo "STOPPED:exit_$exit_code"
    fi
}

check_http_health() {
    local response=$(curl -s -o /dev/null -w "%{http_code}" \
                  --max-time 5 "http://127.0.0.1:$GATEWAY_PORT/health")
    [[ "$response" == "200" ]] && echo "OK" || echo "HTTP_$response"
}

Why both? Because a process can be running (launchd shows it) but the gateway inside can be hung. And conversely, the launchd PID can be gone but the gateway child process is still responding. Checking both gives an accurate picture.

Layer 2: Exponential Backoff with Crash Decay

This is where the "self-healing" actually happens. The crash counter tracks failures, and the backoff delay grows with each crash:

# Backoff delays in seconds
BACKOFF_DELAYS=(10 30 90 180 300 600)

get_backoff_delay() {
    local index=$(($(get_crash_count) - 1))
    [[ $index -lt 0 ]] && index=0
    [[ $index -ge ${#BACKOFF_DELAYS[@]} ]] && index=$((${#BACKOFF_DELAYS[@]} - 1))
    echo "${BACKOFF_DELAYS[$index]}"
}

After a successful health check, the counter decays automatically:

check_crash_decay() {
    local elapsed=$(($(date +%s) - $(cat "$CRASH_TIMESTAMP_FILE")))
    if [[ $elapsed -ge $((CRASH_DECAY_HOURS * 3600)) ]]; then
        echo "0" > "$CRASH_COUNTER_FILE"  # Reset after 6 hours idle
    fi
}

The key insight: the system gets another chance automatically after sitting idle for 6 hours. A temporary glitch that caused one crash doesn't permanently mark the gateway as broken.

Layer 3: Config Auto-Fix

When the gateway exits with code 1 and the error log shows a config issue, the watchdog runs openclaw doctor --fix:

is_config_error() {
    tail -50 "$GATEWAY_ERR_LOG" | grep -q "Config invalid\|Unrecognized keys"
}

try_config_fix() {
    local fix_count=$(cat "$CONFIG_FIX_COUNTER_FILE" 2>/dev/null || echo "0")
    [[ $fix_count -ge 2 ]] && return 1  # Max 2 attempts

    openclaw doctor --fix
    echo $((fix_count + 1)) > "$CONFIG_FIX_COUNTER_FILE"
}

This prevents the restart loop from happening when the actual problem is a bad config entry, not the gateway itself.

Layer 4: Emergency Recovery Escalation

If the gateway has been failing for more than 30 minutes straight, the watchdog triggers a full emergency recovery script:

if [[ $crash_count -ge $MAX_TOTAL_RETRIES ]]; then
    if [[ $elapsed -ge 1800 ]]; then  # 30 minutes
        bash "$EMERGENCY_RECOVERY_SCRIPT" &
    fi
fi

The Level 3 script does deeper diagnostics — checks disk space, memory, port conflicts, and can trigger a full gateway reinstall if needed.

The Result

After implementing all four layers, my gateway has survived:

• A config schema mismatch after an update (Layer 3 auto-fix)
• A restart loop from a bad model load (Layer 2 backoff prevented cascading)
• A port conflict from another service grabbing port 18789 (Layer 4 emergency recovery)
• Multiple overnight crashes that resolved before I checked my phone

The 3 AM failure that prompted this whole system? It was fixed by Layer 2 — the crash counter had decayed overnight, and by the time the gateway tried again, the temporary condition had cleared.

What I Learned

1. Self-healing isn't one script — it's a state machine.

The watchdog doesn't just "try again." It tracks state (crash count, timestamps, backoff delays) and makes decisions based on that state. A restart after a 10-second backoff is a different action than a restart after a 10-minute backoff.

2. Decay is as important as escalation.

Most failure detection systems only escalate. But a crashed gateway that's been stable for 6 hours shouldn't be treated like one that's crashing every 30 seconds. Crash decay handles this — the system gradually forgets temporary failures.

3. Error awareness changes the recovery path.

A process crash and a config error require different fixes. If your watchdog can't distinguish between them, it will waste recovery attempts on the wrong problem. The config auto-fix layer alone saved me from at least a dozen useless restart cycles.

4. Automation that alerts you is better than automation you don't know about.

The watchdog sends a Telegram alert on every recovery attempt. I'm informed when it heals itself, which means I can review the logs and improve the system. Full autonomy with observability beats silent automation that you only discover when it fails.

---

The gateway that went down at 3 AM? It was up again in under a minute. I didn't wake up, check my phone, or scramble to a terminal. The watchdog handled it, and I found out about it in the morning log.

That's the goal. Build systems that need you only when something genuinely needs you.