The Watchdog Pattern: Keeping a Long-Running AI Agent Alive on a Bare VPS

Most tutorials on running an AI agent stop at python agent.py and a happy-path demo. Nobody talks about what happens six days later when the process has silently died, the VPS rebooted after a kernel update, or the agent OOM-killed itself mid-write and left a corrupt state file behind. If your agent is supposed to run continuously — polling a queue, watching a mailbox, executing a trading loop, whatever — staying alive is the actual engineering problem, and it's mostly ignored because it isn't glamorous.

This is the pattern I've settled on after running a long-lived agent daemon on a single small VPS for months: no Kubernetes, no managed queue, just a watchdog script, cron, and a handful of Unix primitives that are older than most of us. It's boring, and that's exactly why it works.

Why not just use systemd?

systemd is the right answer if you're already managing a fleet or you want proper Restart=on-failure, resource limits via cgroups, and journal integration. Use it when you have it.

But a lot of small agent deployments live on a single cheap VPS where you don't control the base image, you're iterating fast, and you want the restart logic to be inspectable in thirty seconds by reading a shell script, not by learning unit-file syntax. A cron-driven watchdog trades some elegance for something valuable: anyone on the team can read watchdog.sh top to bottom and understand exactly what happens when the process dies. That legibility matters more than it sounds like it should when you're debugging a 3 a.m. failure from your phone.

The two approaches aren't mutually exclusive either — you can wrap the same watchdog script in a systemd unit later and lose nothing.

The core loop

The watchdog's only job is: is the process actually running, and if not, why did it stop, and is it safe to start it again right now.

#!/usr/bin/env bash
set -euo pipefail

APP_DIR=/opt/agent
PID_FILE="$APP_DIR/agent.pid"
LOCK_FILE="$APP_DIR/watchdog.lock"
LOG_FILE="$APP_DIR/logs/agent.log"

exec 200>"$LOCK_FILE"
flock -n 200 || exit 0   # another watchdog run is already in flight

is_running() {
  [[ -f "$PID_FILE" ]] || return 1
  local pid
  pid=$(cat "$PID_FILE")
  kill -0 "$pid" 2>/dev/null
}

if is_running; then
  exit 0
fi

echo "$(date -Iseconds) restarting agent" >> "$LOG_FILE"
cd "$APP_DIR"
nohup python3 agent.py >> "$LOG_FILE" 2>&1 &
echo $! > "$PID_FILE"

Wire it into cron every five minutes:

*/5 * * * * /opt/agent/watchdog.sh
@reboot sleep 30 && /opt/agent/watchdog.sh

Three details here matter more than the happy-path logic:

flock -n 200 prevents a self-inflicted race. Cron doesn't wait for the previous invocation to finish before starting the next one. If your agent takes longer than five minutes to start up (loading a model, warming a cache, running migrations), you can end up with two watchdogs both deciding the process is dead and both launching a copy — now you have two agents fighting over the same state file. The flock on a dedicated lock file makes each watchdog run mutually exclusive with a single exit 0 if it can't acquire the lock, no retry loop needed.

kill -0 checks liveness, not health. A process can be running and still be useless — deadlocked, stuck in a retry storm against a dead API, or spinning on a corrupted state file. kill -0 only tells you the PID exists. If your agent can hang without exiting, add a heartbeat: have the agent touch a timestamp file every N seconds, and have the watchdog treat a stale heartbeat the same as a missing PID.

HEARTBEAT_FILE="$APP_DIR/heartbeat"
stale=$(( $(date +%s) - $(stat -f %m "$HEARTBEAT_FILE" 2>/dev/null || echo 0) ))
if [[ $stale -gt 600 ]]; then
  kill "$(cat "$PID_FILE")" 2>/dev/null || true
  rm -f "$PID_FILE"
fi

@reboot needs a delay. After a VPS reboot, cron fires @reboot jobs before networking, DNS, and disk mounts are necessarily ready. A bare @reboot /opt/agent/watchdog.sh will sometimes start the agent against a network that isn't up yet, and depending on how your agent handles startup failures, that can leave it in a bad state that the next five-minute tick won't cleanly recover from. A flat sleep 30 before the watchdog runs is crude but reliable.

State needs to survive the crash, not just the process

A watchdog that restarts the process but hands it corrupted state has made things worse, not better. Two rules cover most of it.

First, write state atomically. Never write directly to the file the agent reads on startup — write to a temp file and rename:

import json, os, tempfile

def save_state(path, data):
    dir_ = os.path.dirname(path)
    fd, tmp = tempfile.mkstemp(dir=dir_)
    with os.fdopen(fd, "w") as f:
        json.dump(data, f)
    os.rename(tmp, path)  # atomic on the same filesystem

os.rename on the same filesystem is atomic at the OS level — a crash mid-write leaves either the old file or the new one, never a half-written JSON blob that throws a parse error on the next boot. This one change eliminates the single most common failure mode I've seen in agent daemons: a SIGKILL or power loss during a state write bricking the next startup.

Second, if you're using SQLite for agent state (a good default for a single-process daemon — no server to manage, and it's plenty fast), turn on WAL mode:

PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;

WAL mode lets readers and the writer operate without blocking each other, and critically, it makes the database far more resilient to being killed mid-transaction — the write-ahead log gets replayed or discarded cleanly on next open instead of leaving the main database file in an inconsistent state. synchronous=NORMAL trades a small amount of durability (you could lose the last transaction in a full power-loss scenario) for meaningfully better throughput, which is the right trade for most agent workloads that aren't handling financial transactions directly.

Log rotation is not optional

An agent that runs forever writes logs forever. On a small VPS, an unrotated log file will eventually fill the disk, and a full disk causes far stranger failures than a missing log line — SQLite writes fail, temp files can't be created, and the watchdog's own lock file operations can start misbehaving. Don't reach for a logging framework here; logrotate is already installed on virtually every Linux distribution and needs about six lines:

/opt/agent/logs/agent.log {
  daily
  rotate 14
  compress
  missingok
  notifempty
  copytruncate
}

copytruncate matters specifically for a process you don't control the shutdown of cleanly — it truncates the log in place rather than renaming it, so your long-running Python process doesn't need to know it happened or reopen a file handle.

What this buys you

None of this is sophisticated. It's a shell script, a cron table, an atomic rename, two SQLite pragmas, and a logrotate config — maybe forty lines of configuration in total. But together they turn "the agent died overnight and nobody noticed until a report was two days late" into a five-minute self-heal with a log line marking exactly when and why it happened. For a single-VPS deployment, that's most of the reliability engineering you actually need — the rest is watching the logs and fixing what keeps the watchdog busy in the first place.