DEV Community

Hive80-lab
Hive80-lab

Posted on

99.99% Uptime, 0% Logins: The Outage Your Dashboard Will Never Show You

Every postmortem I've run has a moment where someone says: "but the monitoring was green." It was. That's the problem.

Here's the outage class your dashboard is structurally blind to: every server answers HTTP 200 while zero customers can do the one thing they came to do — log in, load the product, hit Pay.

Why green dashboards lie

Standard monitors check liveness: does the box answer 200. They don't check utility: can a fresh user complete the critical path. I've watched:

  • Auth service healthy, session store dead → every page loads, every login fails.
  • Payment page renders, webhook queue jammed → paid, no access, no ticket, just silent churn.
  • SSL cert still valid per monitor, but 12 hours from expiry → browsers start refusing tomorrow at 9am.

None of that lights up a 200-check. All of it lights up refund requests.

The fix: synthetic money-path checks

One rule: for every step a paying customer takes, a machine performs it on a schedule.

  1. Login → assert a real session cookie comes back (not a 200 — a session).
  2. Add-to-cart / start-checkout → assert the checkout URL is returned.
  3. Payment page loads → assert the payment iframe or form field exists.
  4. One sandbox payment path → assert the success webhook fires. (Sandbox, weekly. It's free.)

Ten lines of curl + jq per step, or one Playwright script. Runs every 5 minutes. On failure it posts to the channel people actually read, with the exact failed step: "step 2/4: checkout URL missing."

The two additions that catch the sneaky ones

  • A fresh session, never a cached one. Cached logins mask dead session stores for weeks.
  • Cert-expiry check with days-left math. Alert at 21 days, page at 7. Expiry is the only outage you can see a month in advance — yet it still happens to thousands of teams every month.

Why this still doesn't get done

Same reason as everything in ops: the script is 30 minutes, but deciding which steps are the money path, writing the assertions, wiring the alerts, and keeping the checks from rotting is the actual work. That's the gap between "we have monitoring" and "we had monitoring."

If you want the pattern deployed rather than described: I put the exact kit — dead-man's-switch, money-path synthetics, cert tripwires — into Agent Ops 24/7. The Automation Starter Pack has the alert-routing templates, and First 30 Minutes is the emergency runbook for when the check screams at 2am.

Green dashboards that hide dead checkouts are worse than no dashboards — they're alibis. Ship the check before the outage writes your postmortem for you.

Running a stack like this? Our Ops Starter Kit ships the runbooks, checklists, and monitoring patterns behind these posts — grab it and cut your next incident short.

Top comments (0)