Meet Watchdog: A Small, Safe Bash Watchdog for Linux Services

A service does not have to be large to become operationally important. A small API behind a reverse proxy, a background worker on a virtual machine, or a database listener used by a side project can all fail at inconvenient moments. In many of those environments, the missing piece is not a sprawling observability program. It is a reliable way to answer a narrower question: is the service healthy right now, and what should this host do if it is not?

Watchdog is an open-source, dependency-light Bash program built for exactly that job. It checks websites and services from a YAML configuration and can run a defined remediation sequence after a target remains unavailable through its configured retries. It is deliberately a one-shot tool: invoke it from a systemd timer, cron, or an existing scheduler instead of keeping another permanent daemon alive. The project is MIT-licensed and targets Linux systems with Bash 4.3 or newer. 1

That design is compelling when you own the host, understand the recovery action, and want the behavior to stay visible in a versioned configuration file. Watchdog is not trying to replace metrics, distributed tracing, external uptime checks, or a full incident-management platform. It is a focused, host-level building block for detection, bounded recovery, and signal-rich notifications.

One configuration, three useful health signals

Different outages need different evidence. An HTTP success code says something different from an open TCP socket, and a local process manager may already expose the most authoritative health signal. Watchdog supports all three common forms of check, with shared timeout, retry, and retry-delay controls. It stops retrying as soon as one attempt succeeds; only an exhausted sequence is treated as unavailable. 3

Check type	What it validates	Strong fit
HTTP/HTTPS	A request completes and returns an accepted status code; by default, the final status must be in the 2xx range.	Public APIs, web apps, reverse proxies, and /health or /readyz endpoints.
TCP	A connection to host:port opens before the configured timeout.	Databases, caches, brokers, and other services that must accept connections.
Command	Every configured local command exits successfully, in order.	systemd-managed workers, queue probes, or domain-specific local checks.

The distinction matters. A TCP connection is inexpensive and useful, but it does not prove that an application can authenticate, process a query, or serve real traffic. For a public service, an HTTP readiness endpoint is often the better customer-facing signal. For a local worker, systemctl is-active --quiet or a domain-specific probe can be more meaningful. Watchdog lets each service use the check that best represents its actual definition of healthy. 3

Safe automation starts with how commands are represented

The feature that deserves special attention is not merely that Watchdog can restart something. It is how it runs the command. Configured commands are YAML argument arrays rather than shell strings, and the implementation invokes them directly rather than interpreting a configured command through eval or bash -c. This preserves argument boundaries and avoids turning routine configuration text into accidental shell syntax. 1

In practical terms, a remediation action is declared as an executable plus its arguments—not as a fragment of shell code to be re-parsed later. That makes configuration easier to review and reduces a class of quoting and interpolation mistakes in operational workflows. 1

Each command can also have an explicit timeout and, when needed, a working directory. Remediation commands run in order and stop at the first non-zero result. That behavior makes the sequence readable: the YAML describes exactly which corrective steps the host will try, in what order, and for how long. 1

From failed check to verified recovery

Good automation should be deliberate, especially when the corrective action is a restart. Watchdog separates detection from remediation. A service can retry its check before it is declared unavailable; then, if commands are configured and the per-service cooldown permits it, Watchdog executes the ordered remediation sequence. After an optional verify_after delay, it runs the full health check again rather than assuming that a successful restart command automatically means the service recovered. 4

The project also persists per-service state. This allows notifications and hooks to be transition-aware: one failure event appears when a service moves from unknown or healthy to unavailable, repeated runs during the same ongoing outage do not produce duplicate failure events, and a later return to health produces a recovery event. A continuing outage may still receive a later remediation attempt after its cooldown, but it does not flood operators with the same alert every scheduler interval. 4

Control	Why it is useful in production
attempts** + ***retry_delay*	Filters short-lived network or startup blips before marking a service unavailable.
cooldown	Sets a minimum interval between remediation attempts for one service, helping to avoid restart loops.
verify_after	Leaves time for a restart to settle, then verifies the service with the actual health check.
Persistent state	Makes failure and recovery actions transition-based instead of repetitive.
Non-blocking global lock	Prevents overlapping scheduled invocations when a previous run is still active.

This is a pragmatic balance for small operations: the configuration does not pretend an outage is solved just because a command returned zero, and the alerting model does not confuse persistence with importance. 4

A practical Docker Compose example

Imagine a public API whose health endpoint should return either 200 or 204. The service runs in Docker Compose, and a restart should be attempted only after two failed checks. The following pattern comes directly from Watchdog’s documented examples, with values that you should replace for your own host. 6

services:
  - name: api
    check:
      type: http
      url: https://api.example.com/health
      method: GET
      follow_redirects: true
      success_status: [200, 204]
      timeout: 10
      attempts: 2
      retry_delay: 2

    actions:
      cooldown: 300
      verify_after: 5
      commands:
        - command: [docker, compose, restart, api]
          working_directory: /srv/example-api
          timeout: 120
        - command: [docker, compose, restart, nginx]
          working_directory: /srv/example-api
          timeout: 120

The operational logic is easy to read. First, test the endpoint that users or upstream systems rely on. If the check still fails after the retry policy, attempt a controlled restart from the directory containing the Compose file. Wait five seconds, then check the endpoint again. The cooldown means this should not become a restart-on-every-minute loop during a prolonged dependency failure. For a PostgreSQL listener, a Redis instance, or a message broker, the same model can use a TCP check. For a systemd worker, it can use a command check and an explicit systemctl restart action. 3

Notifications that tell you what changed

Watchdog includes built-in SMTP email and can also execute local failure and recovery hooks. Both mechanisms are transition-based. That lets a small team receive an alert when an incident begins, a recovery message when availability returns, and no redundant notification every time the scheduler observes the same unresolved fault. 5

For SMTP, the documented configuration supports templated subjects and bodies as well as an environment-variable password field. With the packaged systemd service, secrets can be kept in a root-owned environment file rather than committed in YAML. For integrations that belong outside the main configuration, hooks expose contextual environment variables such as the service name, event, check type, diagnostic detail, HTTP status, and timestamp. 5

This approach stays intentionally modest. Instead of embedding every chat or incident-management provider into the watchdog itself, the tool can hand a clean, bounded event to a local wrapper that follows your team’s preferred integration path. That is a useful boundary for a Bash utility: keep the health and remediation engine predictable, while allowing local automation to adapt it to the surrounding environment.

Start conservatively, then schedule it

Watchdog’s recommended workflow is refreshingly operational. Configure one target, validate the configuration in dry-run mode, test a controlled failure and recovery on a non-production service, and only then enable the schedule. The -n flag still validates the configuration and runs health checks, but it skips state changes, remediation, hooks, and transition notifications. The -s option narrows a run to a single configured service, which makes first tests safer and easier to interpret. 4

# Validate one service without changing state or restarting anything.
sudo /opt/service-watchdog/service-watchdog.sh \
  -c /etc/service-watchdog/config.yaml \
  -s api \
  -n

# Once validated, enable the packaged systemd timer.
sudo systemctl enable --now service-watchdog.timer

The repository ships a systemd oneshot service and timer, while cron is also supported when it is the established deployment standard. The packaged timer is configured to run every minute and uses a persistent timer; the paired unit treats Watchdog’s ordinary service-outage/remediation exit result as expected while preserving configuration or environment errors as failures that require investigation. 7

Before setting an interval, take a realistic look at the worst-case time spent in checks, retries, corrective commands, and verification. Watchdog has a non-blocking lock, so it will not overlap invocations; however, repeatedly skipped runs are still a useful signal that cadence or timeout settings need revision. 4

A small tool with a clear operational contract

There is real value in infrastructure tools that have a narrow purpose and state their boundaries clearly. Watchdog gives Linux operators a compact contract: declare health checks in YAML, decide the recovery commands in advance, use retries and cooldowns to keep the actions bounded, verify recovery, and receive notification only when availability actually changes.

If you operate a Docker Compose application, a systemd-managed worker, a local database listener, or a small fleet of Linux services, that can be the right amount of automation. Explore the repository, read the Wiki, and begin with the ready-to-adapt examples. Test against a controlled, non-production target first, then make Watchdog part of a deliberate host-level reliability routine.

References