DEV Community

yanlong wang
yanlong wang

Posted on Originally published at yunshao.aicreditsapi.com

I got tired of 3 AM pages, so I built monitoring that fixes the problem first

#devops #monitoring #sideprojects #python

Every solo developer knows this loop:

  1. UptimeRobot emails you at 3 AM: "site down"
  2. You SSH in, half asleep
  3. systemctl restart nginx
  4. Back to bed, wide awake

The fix took 30 seconds. The waking up was the expensive part.

The failure modes behind those pages are almost always the same four things:

  • A service crashed and needs a restart
  • A disk filled up with logs
  • An SSL cert expired
  • A process is eating all the RAM

So I built Wakeless around a simple idea: run the fix first, and only page the human when the fix fails.

The architecture (agentless)

I deliberately did NOT build a server agent. Instead:

  • A cloud controller connects over SSH every 5 minutes
  • Checks run remotely: HTTP/TCP probes from 4 regions, plus disk/memory/SSL expiry
  • Fixes run remotely too, from a hard-coded whitelist: restart a service, clean logs, renew a cert
  • Every SSH login and every fix action is logged to an audit trail in the dashboard

No agent means nothing to install, nothing to update, nothing to compromise on the target server. The tradeoff: the controller needs SSH access — which brings me to the part everyone asks about.

"Wait, you want my SSH credentials?"

Fair question, and honestly the #1 objection. The design answer:

  1. The installer creates a dedicated restricted user (@@WAKELESS_USER@@)
  2. Its sudoers file allows exactly four things: systemctl, journalctl, df, free
  3. Auto-fix commands come from a fixed whitelist — arbitrary commands cannot execute
  4. Root is never required
  5. Every action is logged, and an uninstall script removes everything

Circuit breakers (because auto-fix can be wrong)

An automated restart can make things worse — a crash loop restarting nginx every 30 seconds is worse than nginx being down. So:

  • Max 3 auto-fix attempts per hour per server, then it stops and alerts
  • A canary phase: new heal policies run on one server before expanding
  • If a fix fails, the alert includes the exact diagnostics — no "something is wrong, good luck"

Where it is now

It monitors my own servers, including its own public status page. Free tier: 1 server, 30 monitors, diagnosis-only (no auto-fix). I'm building in public and every "I would never let this touch my server because..." comment is genuinely useful.

What's the first failure you'd want automated — and the one you'd never trust to automation?

Top comments (0)