DEV Community

slawekluzny
slawekluzny

Posted on • Originally published at sentinel-ai.info

Why I Built a Self-Healing Server Agent (And Why It's Not Magic)

Why I Built a Self-Healing Server Agent (And Why It's Not Magic)

Sentinel can detect and contain suspicious activity while surfacing database performance issues without requiring human intervention.

This is what happens when you stop treating servers like pets and start treating them like cattle with PhDs.

The Problem That Started It All

The post-mortem on recurring production issues revealed two uncomfortable truths:

  1. Monitoring tools lie by omission - They'll happily show you green status lights while your database quietly suffocates from unoptimized queries
  2. Alert fatigue is real - When everything is "urgent", nothing is

Traditional monitoring gives you dashboards. What you actually need is a system that:

  • Understands normal behavior for YOUR stack
  • Can distinguish between "interesting" and "critical"
  • Takes safe actions without human hand-holding

How Sentinel Actually Works (Without the Hype)

The Monitoring Layer That Doesn't Sleep

Sentinel's monitoring isn't revolutionary—it's just thorough where others aren't:

  • Process watching via direct systemd integration (not just PID checks)
  • SSL monitoring that actually performs TLS handshakes (not just date checks)
  • Database analysis that reads slow logs AND EXPLAIN plans
  • Attack detection correlating Fail2Ban, CrowdSec and UFW

When it detects failed SSH attempts from an IP, it blocks that IP at the firewall level.

The Database Autopilot That Earns Its Keep

Here's what our MySQL optimization pipeline looks like today:

  1. Slow query analysis - Identifies queries taking above a configurable threshold
  2. EXPLAIN plan review - Flags full table scans, temp tables, bad joins
  3. Index suggestion - Proposes specific multi-column indexes with expected impact
  4. Safe application - Automatically creates indexes marked high confidence

The Security Model That Fights Back

Most security tools are museum curators—they catalog attacks after they happen. Sentinel operates more like a bouncer with photographic memory:

  • IP reputation sharing - Banned IPs sync to our central "Queen" server
  • Behavioral blocking - Detects probing patterns before full attacks launch
  • Automatic containment - Isolates compromised services without killing entire boxes

The Architecture Choices That Made It Possible

Why Python + Flask Instead of Go

We get asked this constantly. For a monitoring tool, Go seems obvious—but we chose Python because:

  • SQLAlchemy gives us ORM flexibility across SQLite (state) and monitored DBs
  • APScheduler handles background jobs without k8s complexity
  • Prototyping speed mattered more than raw performance initially

The Queen-Sentinel Protocol

Every Sentinel node phones home to Queen with:

  • Aggregated metrics (never raw logs)
  • New security signatures
  • Anonymized query patterns

Queen responds with:

  • Updated threat intelligence
  • Cross-fleet performance recommendations
  • Safe action approvals (for Enterprise tier)

This keeps bandwidth minimal while enabling fleet-wide learning.

Pricing That Doesn't Require an MBA to Understand

Tier Servers Price/Mo Key Features
Basic 1 Free Monitoring, security alerts
Pro 5 $49 DB autopilot, email alerts
Enterprise 10 $149 AI chat, cross-server IP blocking

The economics are simple:

  • Pro pays for itself if it prevents engineer time
  • Enterprise makes sense when managing multiple production workloads

Installation

Enter your email at sentinel-ai.info to receive the Queen-vault installer. This:

  1. Creates /opt/sentinel with isolated Python env
  2. Sets up systemd services
  3. Installs Fail2Ban filters (if missing)
  4. Surfaces the admin-UI access URL

No Docker. No k8s manifests. Just a binary watchdog that works.

The Road Ahead

PostgreSQL query analysis is now generally available. The philosophy remains unchanged: your servers should defend, heal and optimize themselves. Because at 3 AM, you'd rather be sleeping than debugging PHP-FPM.

Try Sentinel free on one server—no credit card required. The worst that can happen is you'll stop getting paged for problems that fix themselves.

Top comments (0)