DEV Community

Cover image for My nginx died on a Saturday, so I taught it to fix itself
Suresh Babu Narayanan
Suresh Babu Narayanan

Posted on • Edited on

My nginx died on a Saturday, so I taught it to fix itself

A few months ago I set myself a weekend challenge: could I make a service restart itself when it dies, using only free open-source tools, with zero manual steps in between?

Not because restarting nginx is hard. It's one command. But I've spent thirteen years in infrastructure, and the pattern I keep seeing is this: the alert fires at 2am, a human wakes up, squints at a phone, VPNs in, and runs the one command. The gap between "we knew" and "we acted" is where the pain lives. I wanted to close that gap on my own laptop, end to end, so I properly understood every link in the chain - not just the theory.

Here's the whole thing: Prometheus notices nginx is down, Alertmanager fires a webhook, and a small Python service restarts the container. Total cost: $0. Total code I had to write: about 50 lines.

What we're building 

nginx  →  nginx-exporter  →  Prometheus  →  Alertmanager  →  responder (Flask)  →  docker restart nginx
         (status→metrics)    (scrape+rule)   (route alert)     (receive+act)
Enter fullscreen mode Exit fullscreen mode

Five containers, one docker-compose file. The full repo is at the end - everything below is copy-pasteable, but I'd genuinely suggest typing the alert rule and the responder yourself. That's where the learning is.

Prerequisites: a laptop with Docker, basic YAML tolerance, and about an hour. It took me longer the first time, and I'll show you exactly where I lost the time, because that part taught me more than the parts that worked.

Step 1 - nginx and its exporter

nginx doesn't speak Prometheus natively, so we pair it with the official exporter, which reads nginx's stub_status page and republishes it as metrics - including the one we care about: nginx_up, which is 1 when nginx answers and 0 when it doesn't.

The nginx config just adds a status listener:

server {
    listen 8081;
    location /stub_status {
        stub_status;
        access_log off;
    }
}
Enter fullscreen mode Exit fullscreen mode

And in docker-compose, the exporter points at it:

nginx-exporter:
  image: nginx/nginx-prometheus-exporter:1.1.0
  command:
    - --nginx.scrape-uri=http://nginx:8081/stub_status
Enter fullscreen mode Exit fullscreen mode

Why an exporter instead of just letting Prometheus scrape nginx directly? Because Prometheus's own up metric tells you whether the scrape target answered - and if you scrape the exporter, up stays 1 even while nginx is dead, because the exporter itself is fine. nginx_up is the truth. This distinction bit me once in a real environment years ago, and it's the kind of thing you only internalise by getting it wrong.

Step 2 - Prometheus and the alert rule

Prometheus scrapes the exporter every 5 seconds (aggressive, but this is a demo and I wanted fast feedback loops):

global:
  scrape_interval: 5s
  evaluation_interval: 5s
Enter fullscreen mode Exit fullscreen mode

The alert rule is four lines that matter:

- alert: NginxDown
  expr: nginx_up == 0
  for: 30s
  labels:
    severity: critical
    action: auto_restart
Enter fullscreen mode Exit fullscreen mode

The for: 30s is doing more work than it looks like. Without it, a single blipped scrape - a container pausing for GC, a network hiccup - fires the alert and restarts a perfectly healthy service. With it, nginx has to be continuously down for 30 seconds before anything happens. Every flapping-alert horror story I've heard traces back to someone skipping this line. In production I'd set it longer; for a demo, 30 seconds keeps the feedback loop tight.

The action: auto_restart label is a habit from working with event-driven automation at scale: route on labels, not alert names. Today one alert triggers a restart; later, twenty alerts can share the same remediation path just by carrying the label.

Step 3 - Alertmanager, and where I lost an hour

Alertmanager's job is routing: this alert goes to that webhook.

route:
  receiver: auto-remediation
  group_by: ["alertname"]
  group_wait: 10s
  repeat_interval: 5m

receivers:
  - name: auto-remediation
    webhook_configs:
      - url: http://responder:5000/alert
        send_resolved: true
Enter fullscreen mode Exit fullscreen mode

This is where my Saturday went sideways. My first version had the webhook URL pointing at localhost:5000 - which inside the Alertmanager container means Alertmanager itself, not my responder. Prometheus showed the alert firing, Alertmanager showed it received, and… nothing happened. No errors anywhere obvious, because from Alertmanager's point of view nothing was wrong; it was dutifully delivering webhooks into the void.

The fix is obvious in hindsight - use the compose service name, responder:5000 - but I mention it because "alert fires, action doesn't happen, no visible error" is exactly the failure mode you must know how to debug before trusting automation to act for you. docker logs alertmanager and a curl from inside the container found it in the end.

The other setting worth a sentence: repeat_interval: 5m means if the alert keeps firing, the webhook re-fires every 5 minutes. That sounds helpful until you realise it means your restart action will retry forever against a service that's never coming back. Hold that thought.

Step 4 - the responder: 50 lines of Flask

The receiving end is deliberately boring - a Flask app that validates the alert and restarts the container via the Docker API:

@app.route("/alert", methods=["POST"])
def alert():
    payload = request.get_json(silent=True) or {}
    for a in payload.get("alerts", []):
        name = a.get("labels", {}).get("alertname")
        if name != "NginxDown" or a.get("status") != "firing":
            continue
        if not allowed_to_restart():
            log.warning("circuit breaker OPEN - escalate to a human")
            continue
        client.containers.get("nginx").restart(timeout=10)
        restart_history.append(time.time())
Enter fullscreen mode Exit fullscreen mode

The part I care about is allowed_to_restart() - a circuit breaker that refuses more than 3 restarts in 10 minutes:

def allowed_to_restart() -> bool:
    now = time.time()
    while restart_history and now - restart_history[0] > WINDOW_SECONDS:
        restart_history.pop(0)
    return len(restart_history) < MAX_RESTARTS
Enter fullscreen mode Exit fullscreen mode

Why bother, in a toy? Because auto-remediation without a stop condition isn't self-healing, it's self-harm with good intentions. If nginx is crashing because of a broken config or a full disk, restarting it in a loop fixes nothing and hides the real problem behind a wall of "remediation succeeded" logs. Three strikes, then a human gets paged. In a real setup this is where you'd escalate to Slack or PagerDuty; in the demo it just logs loudly.

Step 5 - kill it and watch it heal

The payoff. With everything running:

docker stop nginx
Enter fullscreen mode Exit fullscreen mode

Then I sat and watched three windows: the Prometheus alerts page, docker logs -f responder, and docker ps.

The sequence: the alert went pending at the next scrape, firing after the 30-second hold, Alertmanager delivered after its 10-second group wait, and the responder log printed restarted container 'nginx'. Then - my favourite line ever - docker ps showing nginx with an uptime of a few seconds while everything else showed twenty minutes.

End to end, dead-to-healed took on my machine. Nobody touched anything.

I ran it a few more times just to enjoy it, which promptly tripped my own circuit breaker. Satisfying to be refused by my own safety rail.

The irony that wrote itself

Here's the part I didn't plan. The same week I finished this write-up, I deployed a small side project - a cron expression converter - to a well-known static hosting platform. Single HTML file, simplest deployment imaginable.

The deployment got stuck in "Queued". For twenty minutes. Then the retry got stuck too. It turned out to be a known platform-side queue jam that community threads report lasting anywhere from hours to over a day, and there was nothing I could do from my side but wait or fail over. I dragged the same file onto a different host and was live in ninety seconds.

Which is the whole point of this article, really. Everything fails: my nginx, their deploy queue, your thing next Tuesday. The difference between a bad day and a non-event isn't preventing failure - it's how much of the response you've already built before it happens. My pipeline had a restart path ready; my deployment had a second host ready. Same principle, different layer.

Where I'd take this next

If I were promoting this beyond the laptop, in rough order:

  1. Deduplication and idempotency - Alertmanager retries webhooks; the responder should handle the same alert twice gracefully (it mostly does now, but "mostly" is doing heavy lifting).
  2. Escalation - after the circuit breaker opens, page a human with the restart history attached, so they arrive with context instead of a cold start.
  3. Audit trail - every automated action written somewhere durable. When automation acts on your behalf, "what did it do and when" must never be a mystery.
  4. Smarter checks than up/down - restart-on-death is the "hello world" of remediation. The interesting versions act on latency, error rates, and saturation before the service dies.

There are, of course, mature event-driven automation platforms that do all of this at enterprise scale with rulebooks and workflow engines. Having now built the primitive version by hand, I understand far better what those platforms are actually doing under the hood - which was the point of the weekend.

Try it yourself

The full repo - compose file, configs, responder, and a README with the exact break-it steps - is here: https://github.com/suredares/Self-healing-Demo

Clone it, run docker compose up -d --build, then docker stop nginx and watch your infrastructure quietly refuse to stay broken.

If you build it and something behaves differently on your setup, tell me in the comments - genuinely curious what breaks on other machines. And if you've built auto-remediation that went wrong in an interesting way, I definitely want to hear that story.

*I'm a DevOps engineer with 13 years across AWS, Azure, Kubernetes, Terraform and Ansible, writing about automation that actually works. I also built CronPort - a free tool that converts cron expressions between crontab, Kubernetes, GitHub Actions, AWS EventBridge and Terraform.
I also wrote up Five Terraform state mistakes I keep seeing

I also sell a Terraform AWS Starter Kit if you want production-ready foundations without the setup day.

If it saved you time, you can buy me a coffee Buymeacoffee*

Top comments (0)