DEV Community

matteo adorni
matteo adorni

Posted on

Keep any long-running Python process alive on Linux for $0 (the /proc watchdog pattern)

#bash #python #linux #devops

Anything you leave running will eventually stop running. A scraper, a trading bot, an agent, a queue worker — it does not matter how good the code is. What matters is whether you find out in five minutes or in three weeks.

This is the pattern I use to make long-running processes self-healing, with no orchestrator, no systemd gymnastics, and no paid monitoring. It is a single bash script and a cron line. It works for anything: it does not need to know what your process does, only what it looks like.

The trap everyone falls into first

The obvious watchdog is this:

if ! pgrep -f "mybot" > /dev/null; then
  restart_mybot
fi
Enter fullscreen mode Exit fullscreen mode

It looks correct. It is a landmine.

The problem: pgrep -f matches against the full command line, and your watchdog's own command line contains the pattern you are searching for. Depending on how the script is invoked (especially from cron or from a parent that echoes the command), the script can match itself and conclude the process is alive — or worse, pkill -f mybot will kill the shell that is running the watchdog.

I hit this exact bug: the watchdog ran, matched its own invocation, and killed itself before doing anything. The log looked like the bot was fine. It was not.

There is a second, subtler failure: matching a name is not matching a process. Any unrelated binary with the same substring satisfies the check.

The robust check: scan /proc, and require two things

Instead of matching a string, walk /proc and require both:

  1. the command line contains your exact start command, and
  2. the process's resolved executable is the one from your virtualenv. 
VENV_PY=$(readlink -f "$BASE/.venv/bin/python")

find_bot() {
  for p in /proc/[0-9]*; do
    pid=${p#/proc/}
    [ -r "$p/cmdline" ] || continue
    cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null) || continue
    case "$cmd" in
      *"freqtrade trade --config"*)
        exe=$(readlink -f "$p/exe" 2>/dev/null)
        [ "$exe" = "$VENV_PY" ] && echo "$pid"
        ;;
    esac
  done
}
Enter fullscreen mode Exit fullscreen mode

Why this is safe:

  • /proc/<pid>/cmdline is the process's actual argv, NUL-separated — not a shell's mangled string.
  • readlink /proc/<pid>/exe resolves to the real binary. Requiring it to equal your venv's interpreter means you can never match a stray system Python, a shell, or the watchdog itself.
  • A process that has already died leaves no /proc/<pid> entry, so there is no stale-match window.

Adapt the case pattern and the exe path and this same function works for a Node app, a Go binary, a scraper — anything.

The full watchdog

This is production-shaped: it restarts the process, records a health snapshot on every run, enforces a safety rule, and stays silent unless it acts.

#!/bin/bash
# Generic self-healing watchdog. Silent on success.
set -u
BASE=/home/kali/myapp
VENV_PY=$(readlink -f "$BASE/.venv/bin/python")
cd "$BASE" || { echo "watchdog: cannot cd $BASE"; exit 1; }
LOG=$BASE/logs/watchdog.log
ts() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }

find_proc() {
  for p in /proc/[0-9]*; do
    pid=${p#/proc/}
    [ -r "$p/cmdline" ] || continue
    cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null) || continue
    case "$cmd" in
      *"myapp run --config"*)
        exe=$(readlink -f "$p/exe" 2>/dev/null)
        [ "$exe" = "$VENV_PY" ] && echo "$pid"
        ;;
    esac
  done
}

MSG=""
if [ -z "$(find_proc)" ]; then
  echo "$(ts) process not running -> restarting" >> "$LOG"
  nohup "$BASE/.venv/bin/python" -m myapp run --config "$BASE/config.json" \
      >> "$BASE/logs/app_stdout.log" 2>&1 &
  sleep 30                     # give it time to come up
  if [ -n "$(find_proc)" ]; then
    MSG="watchdog: restarted at $(ts)"
  else
    echo "$(ts) restart FAILED" >> "$LOG"
    exit 1                     # exit non-zero so cron mails / your notifier fires
  fi
fi

# health snapshot + safety rule (exit codes: 0 ok, 2 breach, 3 unreachable)
OUT=$("$VENV_PY" "$BASE/health.py" 2>&1); RC=$?
if [ $RC -eq 2 ]; then
  for pid in $(find_proc); do kill "$pid" 2>/dev/null; done
  echo "$(ts) SAFETY RULE TRIPPED -> stopping process" >> "$LOG"
  exit 2
fi
echo "$OUT" >> "$LOG"
[ -n "$MSG" ] && echo "$MSG"       # speak only when something happened
exit 0
Enter fullscreen mode Exit fullscreen mode

Why "silent on success" is the important part

A watchdog that messages you every fifteen minutes gets muted within a day — and a muted watchdog is worse than none. Emit output only when you restarted something or a rule tripped, and pipe that output to whatever notifier you already use (a Discord webhook, mail, a Telegram bot). Silence means health.

The safety rule lives outside your app

Whatever invariant protects you from disaster — a loss limit for a trading bot, an error-rate ceiling for a scraper, a queue-depth threshold — put it in the watchdog's health script, in one place. Your app's internal guards protect a request; this protects the account. One threshold, one exit code, one kill switch.

Scheduling 

*/15 * * * * /home/kali/myapp/scripts/watchdog.sh
Enter fullscreen mode Exit fullscreen mode

Fifteen minutes is a good default: fast enough to matter, slow enough to be invisible. Make the script idempotent — two runs must not start two copies. The two-condition check above guarantees that: if the process exists, the function returns a PID and nothing is started.

Test it, or you do not have one

A watchdog you have never seen fire is a watchdog you do not have. Do this once, immediately:

# 1. kill the app
for pid in $(pgrep -f "myapp run --config"); do kill "$pid"; done
# 2. run the watchdog by hand
bash scripts/watchdog.sh
# 3. confirm BOTH the restart and the log line
tail -3 logs/watchdog.log
Enter fullscreen mode Exit fullscreen mode

If it does not restart, you have a path or an exe mismatch — fix it now, not at 03:00.

The cost

Zero. A cron line, a bash script, and the filesystem you already have. No agent to install, no dashboard, no token, no monthly fee. For the 95% of solo projects that just need "it keeps running and tells me when it doesn't", this pattern beats a full orchestrator on every axis that matters: it is 60 lines you can read and reason about completely.

The two ideas worth keeping, whatever you build: identify processes by /proc + exe, never by name, and let the watchdog speak only when it acts.

Want this pre-built? I packaged the watchdog, the P&L logger, the backtest runner and the config
into a drop-in kit — Freqtrade Self-Healing Starter Kit ($9).
It is the exact code from this article plus a setup guide with the /proc gotcha already solved,
so you can go from zero to a self-healing bot in about five minutes.

Related in this series:

The watchdog from this post is open source: **https://github.com/moon-hacks/proc-watchdog* (MIT).*

Prefer someone to just do it? I also offer a **done-for-you setup* (https://tntofficial.gumroad.com/l/vweza) — Freqtrade installed, backtested and left running in dry-run with the watchdog active.*

Top comments (0)