DEV Community

Hive80-lab
Hive80-lab

Posted on

Our dashboard was 47 checks green. Checkout had been down for 20 minutes.

#devops #sre #monitoring #observability

Every monitoring stack I've inherited has the same disease: dozens of green checks that measure whether servers are awake, while the thing that actually makes money silently dies.

CPU at 12%? Green. Memory fine? Green. Load balancer healthy? Green. The Stripe webhook endpoint returning 500s for 20 minutes? Nobody noticed until a customer emailed "does checkout work?"

The gap: synthetic vs. business monitoring

Your infra checks answer "is the machine alive?" Customers only care about "can I give you money?" Those are different questions, and the second one needs its own probes.

Here's the minimum business-coverage set for any small team:

  1. One real end-to-end purchase attempt — a test product, real API path, sandbox gateway. Every 5 minutes. If this fails, nothing else matters.
  2. Signup funnel probe — POST a test user, verify the welcome email flow fires.
  3. Critical cron heartbeat — the nightly billing/export/report job must check in ("dead man's switch"). No check-in = page, even though nothing "errored."
  4. Webhook receiver probe — fire a test webhook at your own endpoint, assert it processed.

Wiring it up cheaply

You don't need a $29/seat monitor for this. A 50-line Python script on a $4 VPS with cron and a Telegram/Discord webhook covers all four probes:

# probe.py — dead man's switch example
import os, requests
try:
    r = requests.post(SIGNUP_URL, json=TEST_PAYLOAD, timeout=10)
    assert r.status_code == 201, f"signup returned {r.status_code}"
    assert r.json().get("id"), "no user id in response"
except Exception as e:
    send_alert(f"SIGNUP BROKEN: {e}")  # -> discord webhook
Enter fullscreen mode Exit fullscreen mode

Then the rule that makes it stick: every alert must map to a runbook, or the alert gets deleted. An alert without a documented first response is just anxiety on a schedule.

The checklist we run before any launch

  • [ ] One probe per revenue path (signup, checkout, webhook, report)
  • [ ] Dead man's switch on every critical cron
  • [ ] Each alert links to a runbook with first 3 commands
  • [ ] Monthly restore drill: restore one backup, diff it, delete
  • [ ] Alert volume review: anything that fired >3x without action gets demoted to digest

The full kit — probe scripts, dead-man's-switch cron, restore-drill checklist, and the 2 AM runbook templates — is in the Ops Starter Kit: https://hive80lab.gumroad.com/l/ops-starter-kit

Green dashboards measure effort. Business probes measure money. Monitor the money.

Top comments (0)