DEV Community

Hive80-lab
Hive80-lab

Posted on

Stop paying for monitoring: the 20-line heartbeat script that catches outages before your customers do

#devops #monitoring #bash #sre

Most small teams don't have a monitoring problem. They have a notification problem: something died, and the first person to know was a customer on Twitter.

You don't need a $200/month observability platform to fix that. You need a heartbeat.

The idea

Every important automation you run should write one line somewhere you actually look:

$(date -u +%FT%TZ) nightly-sync OK
Enter fullscreen mode Exit fullscreen mode

Then a second, tiny script checks that file. If the last heartbeat is older than N hours, it screams. That's it. That's the whole system.

#!/usr/bin/env bash
# heartbeat-check.sh - alert if any heartbeat is stale
MAX_AGE_HOURS=6
while IFS='|' read -r name file; do
  [ -f "$file" ] || { echo "ALERT: $name missing ($file)"; continue; }
  last=$(stat -f %m "$file" 2>/dev/null || stat -c %Y "$file")
  now=$(date +%s)
  age_h=$(( (now - last) / 3600 ))
  [ "$age_h" -ge "$MAX_AGE_HOURS" ] && echo "ALERT: $name silent for ${age_h}h"
done < heartbeats.txt
Enter fullscreen mode Exit fullscreen mode

Run it from cron every 15 minutes and pipe output to anything that can reach a human: an email, a Slack webhook, a Telegram bot. Twenty lines. Zero vendor.

Why this beats dashboards

Dashboards are pull. You have to remember to look, and you look at them after you suspect something. A heartbeat is push: silence itself is the alarm. When the "everything fine" signal stops arriving, you know — even if you never opened the dashboard.

Three rules that make it stick:

  1. Write the heartbeat inside the job, not around it. A wrapper script that logs "job ran" even when the job failed is worse than nothing.
  2. Check output, not exit codes. Assert a row count or a checksum. A script that exits 0 every day while syncing zero rows has been lying to you for weeks.
  3. Test the alarm path monthly. Kill a heartbeat on purpose and confirm the alert actually reaches your phone. An untested alert is a rumor.

Where this leads

Once you have heartbeats, the natural next layer is the runbook: for each job, one page saying what it does, where it lives, what it depends on, and how to restart it. Then a simple retry policy: exponential backoff, hard cap, distinct alert when the cap trips. These three layers — heartbeat, runbook, retry rules — solve most of what teams hope an expensive monitoring product will solve for them.

If you want the full checklist system — runbook templates, heartbeat scripts, the daily 10-minute ops drill, and the first-30-minutes-after-an-outage playbook — it's all in the Ops Starter Kit:

👉 Ops Starter Kit — $14, yours forever, updates included. The first-30-minutes outage playbook inside it is free to grab on its own.

Ship the boring safety layer this week. It's the highest-ROI hour your team will spend this month.

Top comments (0)