DEV Community

Hive80-lab
Hive80-lab

Posted on

47 of 50 Small Teams Learned Their Site Was Down From a Customer

#monitoring #devops #webdev #sre

I keep an informal tally from every ops engagement I take: "how did you FIRST find out your site was down last time?" Across the last 50 answers: 47 said a customer told them. Two said a status-page service. One said their own monitoring.

One. Out of fifty.

Why the dashboard lies

Almost every internal monitor watches the inside of the system: the API endpoint, the database, the queue depth. Those checks run from inside the network. When DNS breaks, when the cert expires, when the load balancer dies, when the CDN serves a cached error page — every internal check stays green, because the inside of your system is perfectly healthy. It's the front door that's gone.

Customers don't hit your API. They hit your homepage. Monitor what your customers hit, from where your customers are (outside your network).

The 15-line external heartbeat

No vendor needed. A cron job on any machine that ISN'T your web server:

#!/bin/bash
# external-heartbeat.sh — run every 5 min
URL="https://yoursite.com"
LOG="$HOME/uptime.log"
CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 15 "$URL")
echo "$(date -u +%FT%TZ) $CODE" >> "$LOG"
if [ "$CODE" != "200" ]; then
  curl -s -X POST -H 'Content-type: application/json' \
    --data "{\"text\":\"FRONT DOOR DOWN: $URL returned $CODE at $(date -u +%FT%TZ)\"}" \
    "$YOUR_SLACK_WEBHOOK"
fi
Enter fullscreen mode Exit fullscreen mode

Two design rules that make it work:

  1. It must run from outside your infrastructure. Same network = same blind spot.
  2. Alert on 2 consecutive failures, not 1. One blip is a deploy; two is an outage. (Track the last code in a file if you want zero false alarms.)

Three upgrades in the same file

  • Cert expiry: echo | openssl s_client -servername yoursite.com -connect yoursite.com:443 2>/dev/null | openssl x509 -noout -enddate — alert when under 14 days.
  • DNS resolution: dig +short yoursite.com — alert when empty or changed unexpectedly.
  • Keyword check: status 200 with the word "error" in the body is still a down site. curl -s "$URL" | grep -q "error" && alert.

That's the whole front-door stack: external, cheap, and it measures the thing customers actually experience. Twenty minutes to deploy, and you stop being one of the 47.

I package the exact version I deploy for clients — external heartbeat, cert/DNS/keyword checks, threshold tables and the weekly evidence report — in the Ops Starter Kit ($19, one-time). If you want the full ops layer including automation and agent ops at bundle pricing, that's the Ops Mega Bundle. Full catalog at hive80lab.gumroad.com.

Question for the comments: what did YOUR last outage teach you that no monitor caught? Mine: a CDN serving a cached 200 page full of error text — every check said "up" for 6 hours.

Top comments (0)