I Built a Self-Healing Web App (And It Took a Tab Character to Break My Will to Live) Or: a love letter to restart: always

The idea
What if your web app could detect that it's dead and just... come back? No PagerDuty. No 3am phone call. No "sorry, deploying a fix." Just a container that notices it's down and says "nah, I'm fine now" within seconds.

That's what I built. And I'm going to walk you through the whole thing including the parts where I'm an idiot.

self-healing-app/
├── app/              # Flask app (the thing that dies)
│   ├── app.py
│   └── Dockerfile
├── nginx/            # Reverse proxy
│   └── nginx.conf
├── prometheus/       # Monitoring + alerting
│   ├── prometheus.yml
│   └── alert.rules.yml
└── docker-compose.yml   

`Four containers on a bridge network:

Container Job
web-app Flask app with /health and /metrics endpoints
nginx-proxy Routes traffic to the app
prometheus Scrapes metrics every 5s, fires alerts
grafana Makes it look like you know what you're doing

`

Five seconds of up == 0 and the alert fires. No flapping on a single missed scrape.

The debugging journey (a.k.a. the real tutorial)

Problem 1: docker-compose not found.
Fix: It's docker compose (no hyphen) now. The old v1 binary is deprecated. Install docker-compose-plugin.

**Problem 2: **Port 9090 already in use.
Fix: sudo lsof -i :9090 → it was a leftover process. Killed it.

Problem 3: Port 80 already in use.
Fix: System nginx was running. sudo systemctl stop nginx && sudo systemctl disable nginx.

Problem 4: Prometheus target stuck at health: down with error received unsupported Content-Type "text/html".
Fix: Flask returns text/html for string responses. Changed to:

return Response("up 1\n", content_type="text/plain; version=0.0.4; charset=utf-8")

Problem 5: A tab character in alert.rules.yml.
Fix: sed -i 's/\t/ /g' alert.rules.yml. I will never use tabs in YAML again.

P*roblem 6*: I pasted a YAML healthcheck block into my Python file.
Fix: I don't talk about this one.

Testing the self-healing

Kill the app

docker kill web-app

Watch Prometheus (http://localhost:9090)

- Target goes red

- up metric drops to 0

- Alert fires after 5s

- Docker restarts the container

- Target goes green

- Alert resolves

Total downtime: ~5-10 seconds. No human required.

Running it

docker compose up --build -d

App: http://localhost
Prometheus: http://localhost:9090
Grafana: http://localhost:3000 (admin/admin)
Add Prometheus as a data source in Grafana (URL: http://prometheus:9090), import dashboard ID 1860, and you've got a monitoring wall that would make your DevOps friend jealous.

What's next

• Migrate to Azure Container Apps (managed, serverless-ish, auto-scaling) Add Alertmanager for actual notifications (Slack, email, PagerDuty) Automate failure injection (no more manual docker kill) Add node_exporter for host-level metrics Maybe a Terraform module for the whole thing Final thoughts The "self-healing" part is the easiest bit. The hard part is making sure your monitoring is actually monitoring — that the scrape works, the content type is right, the network is correct, the YAML doesn't have a rogue tab.

The system doesn't heal itself. You heal it first. Then it heals itself. Then you sleep.