A power cut rebooted my home servers one June morning, and the reverse proxy didn't come back. Every site I run goes through it, and I only found out that evening. The alert had fired on time, but it reached my phone through a public URL, and that URL went through the proxy that was down.
TL;DR I built an auto-remediation chain for my homelab: Prometheus → Alertmanager → n8n → a policy gate → an executor that can only run actions from a closed list. Restarting a service is one line. The real work was teaching the bot when to refuse. Then I timed it against a human, and it won by 17 seconds.
A watchdog now restarts the proxy if it dies. It can't see a frozen one: a paused container stays "running" with its port open while no traffic gets through. That quiet kind of failure is the one worth automating.
Why a ten-line restart script is dangerous
The obvious fix is a script that restarts a service when it stops responding. Then the NAS goes down, and it restarts the thirty services that depend on it, in a loop, all night. It can't fix anything because the cause is elsewhere, and by morning one diagnosable failure is buried under a pile of restarts. A remediation bot that can't say no turns small outages into big ones. Everything below is about keeping the bot from acting when it shouldn't.
The architecture: four roles, strictly separated
Prometheus ─► Alertmanager ─► n8n ─────────► hsd/gate ─────────► rb-exec
detects routes filters, decides: act or runs ONE action
asks one refuse, where, from a closed
question with which action per-host catalogue
│
└─► re-checks recovery in Prometheus
n8n runs on its own small VM and does very little on purpose. Alertmanager pushes it an alert, not a command. n8n drops anything no longer current (a grouped webhook payload also carries alerts that just resolved), asks the gate "here's the alert and its labels, what do we do?", and logs the answer.
Handing n8n the executor's credentials would be the classic confused deputy. So the secrets vault gives n8n its own credential and the one for calling the gate, and refuses it the executor's. Only the gate gets that one. Someone who takes over n8n can ask for a restart, not run one.
The gate answers with one of nine verdicts. One means "act". The other eight are refusals: frozen by a deeper cause, circuit breaker open, unknown alert, ambiguous match, host not resolved, not armable, undecidable (Prometheus is unreachable, so it can't tell whether something bigger is going on), and maintenance. Every refusal pages me except the two that are expected: frozen and maintenance.
What a catalogue entry looks like
Here's the first service I armed, my ebook library (Calibre-Web), lightly sanitized. The executor's catalogue declares the action and the only container it may touch:
restart-container:
args:
name:
type: enum
values: [calibre-web]
# "Is it ALREADY in the desired state?" Exit 0 means nothing to do (noop).
precondition: "test \"$(docker inspect -f '{{.State.Health.Status}}' {name} 2>/dev/null)\" = healthy"
command: "docker restart {name}"
proof: "test \"$(docker inspect -f '{{.State.Health.Status}}' {name} 2>/dev/null)\" = healthy"
On the gate side, the service's runbook maps the alert to that action and gives the PromQL proof of recovery:
### S1 — UI frozen while the container is Up
- alert: CalibreWebDown
- proof: `probe_success{instance="<calibre-login-url>"} == 1`
- fix_ref: F1 # F1: rb-exec restart-container name=calibre-web
The gate re-checks that proof for up to 120 seconds after the action. If it doesn't hold by then, the attempt is logged as a failure and I'm paged. The entry can run at most three times an hour, and two failures in a row disarm it.
Guardrail 1: don't treat the symptom of a deeper cause
When the NAS goes down, five services alert at once and none of them is broken. So the gate keeps a cause hierarchy: NAS down freezes the five services that depend on it, a server down freezes everything on it, the internet box down freezes everything. Frozen means the alert is real, the gate knows it, and it doesn't act, because the alert for the real cause is already firing.
One entry makes me smile: if the machine hosting the gate goes down, everything freezes, because the gate's own view of the world can't be trusted anymore.
Guardrail 2: a circuit breaker that survives restarts
A wrong bot is wrong fast, so it has hard ceilings:
| Limit | Value | Prevents |
|---|---|---|
| Actions per service per hour | 3 | A service going round in circles |
| Actions across the fleet per hour | 10 | A fleet-wide runaway |
| 24 h failure rate that disarms everything | 30% | A bot that has started getting things wrong |
| Consecutive failures that disarm an entry | 2 | Retrying all night |
| Actions per day on a critical component | 1 | Trial and error on, say, the proxy |
The counters live in SQLite on disk: in memory they'd reset right after the crash a runaway remediation caused. There's no reset button either. A noop ("nothing to do") counts toward the hourly budgets but never toward the failure rate, so 38 noops and 2 failures is two failures out of two, not 5%.
The breaker tripped on its own on day one, after four attempts in the same hour. A better test than any I'd have written.
Guardrail 3: verify recovery, don't trust the exit code
"Restart done" only means exit code 0, so the gate asks Prometheus whether the service's probe answers again. The proxy needs two proofs: its admin endpoint, and a public site served through it with its real hostname and certificate.
Subtler: restarting a paused container unpauses it first, and the old process answers for a few seconds while shutting down. In four of my first eight real failures, the gate closed on a sample taken before the restart finished. Now every sample a proof reads must postdate the action.
And watch the wording. "Is the symptom present?" and "has the service recovered?" sound alike, but one is true when things are broken and the other when they're fine. Swap them and the chain reports success no matter what.
The canary: proving the chain is still alive
How do you know a safety mechanism works when it almost never fires? One metric holds the timestamp of the last successful end-to-end run (remediation_canary_dernier_passage; dernier passage is French for "last run"). Past 20 hours, a "wake up" alert goes into the chain itself, which runs a harmless canary-touch action that refreshes the timestamp, and the alert resolves. The chain proves it works by clearing the alert that called it. Past 26 hours, a second alert goes to my phone.
My first version of that second rule had a hole: "older than 26 h" never fires if the series disappears, because comparing against a missing series returns nothing rather than false. The real rule:
- alert: RemediationCanaryStale
expr: >-
(time() - remediation_canary_dernier_passage > 93600)
or absent_over_time(remediation_canary_dernier_passage[2h])
for: 15m
labels:
severity: critical
Breaking it for real
I started with the ebook library, where downtime costs nothing and which had once frozen unnoticed for twenty-five minutes. I reproduced that (process paused, container up) and touched nothing.
| Event | Time |
|---|---|
| Freeze induced | 14:06:34 |
Alert fires (after its 3 min for:) |
14:09:49 |
| Service answers again | 14:10:22 |
| Alert to recovery | 33 s |
| Total outage | 3 min 48 s |
Look at the precondition above: the executor won't act while the container reports healthy, and right after the freeze it still does. Its healthcheck takes 95 seconds to flag the problem, the alert 180 to fire. That's 85 seconds of margin, luckily in the right direction. Reversed, the chain would have found a frozen service claiming to be healthy and answered "nothing to do". I now time that gap for every new service.
Seventeen seconds
How long would a human have taken? Eight minutes earlier I'd replayed the same failure with nothing armed and an operator at the screen, reacting within a second. That's the best case for a human, not a realistic one. It's also a single run on each side, so read it as an order of magnitude, not a benchmark.
| Alerted human | Automated chain | |
|---|---|---|
| Alert to recovery | 50 s | 33 s |
| Total outage | 4 min 13 s | 3 min 48 s |
Seventeen seconds. I'd hoped for much more. But my long outages came from nobody knowing the service was down: in June because the alert never reached me, for the library because there was no probe at all. Most of the gain arrived the day I added the probe and the alert. The value was in the smoke detector, not the fire truck.
The argument for the chain is different, then. Those 33 seconds are the same at 3 a.m. on a Sunday or while I'm on holiday, so the fair comparison is against a human who isn't there, and I can't measure that without a real night-time outage. Every repair also leaves a record of when, which action and which proof.
Traps I nearly shipped
Some mechanisms only looked alive. The monitoring registration tool printed registered n8n (6 VMs total) into a directory Prometheus no longer read, and n8n's health endpoint answers before its workflows are active, so a deployment check passed while nothing listened.
Planned stops were another gap. When I stop a service to upgrade it, the chain would restart it three minutes later, maybe mid-migration. Docker doesn't expose whether a stop was intended, so I declare it. A maintenance mode covers a whole host, carries an end time rather than a switch I'd forget to flip back, and holds even if I forgot to silence the alert. If its file becomes unreadable, maintenance is treated as active, for at most 24 hours.
The password vault (Vaultwarden) taught me the most. First, something was already repairing it: an older tool restarts any container that reports unhealthy, after three and a half minutes, while the vault's alert waits ten. The new chain would have arrived six and a half minutes late with nothing to do. "What already repairs this?" is now a step in the procedure.
Second, the generic precondition read the container's internal healthcheck, while the alert's probe hits the vault from outside. When those disagree is exactly when a restart would help, and the precondition would have said "nothing to do". The vault's precondition is now the same HTTP request as its proof. My check for any new entry: break what the proof measures, and confirm the precondition turns false.
Third, "it answers" isn't "it works". A vault with a corrupted database still answers and can't decrypt a single password. An endpoint returning an account's key-derivation parameters looked like a good proof, until a non-existent address got the same response, by design, so accounts can't be enumerated. The real proof is a full end-to-end login. Building it showed that nothing had been watching this at all. That new detection may be the biggest win of the project.
The front door, frozen on purpose
Last came the frozen proxy, where a botched restart takes every site down. The watchdog handles a dead proxy; the chain covers a frozen one. The executor got two reasons to refuse. It validates the config in a throwaway container with no network first, because restarting on an invalid file would leave everything down. Sanitized:
docker run --rm --network none --entrypoint caddy \
-v "$CADDYFILE":/etc/caddy/Caddyfile:ro "$IMAGE" \
adapt --config /etc/caddy/Caddyfile --adapter caddyfile >/dev/null 2>&1 \
|| { echo 'invalid Caddyfile: restart refused' >&2; exit 3; }
It also refuses if a network address the proxy binds to has disappeared, since Docker would then fail to start it. And it gets one action per day.
I froze the proxy for real one late morning while polling six public sites about every five seconds.
| Event | Time |
|---|---|
| Proxy frozen | 11:55:17 |
Alert fires (after its 2 min for:) |
around 11:57:45 |
| Proxy restarted by the chain | 11:58:07 |
| All six sites answer again, usual status codes | 11:58:11 |
| Outage seen from outside | about 2 min 50 s |
In June the front door stayed shut until evening. This time it was under three minutes, almost all of it the alert's confirmation delay. The restart itself took about ten seconds.
Where it stands
About twenty services are armed now, from the canary to the proxy, including the vault, the wiki and the code forge. Each went through a full acceptance run with an induced failure first.
I expected to fix things much faster, and the stopwatch said otherwise. What I got is more modest: things get fixed even when I'm not around.
Do you let automation restart things in your homelab? Where do you draw the line, and how do you stop a dead NAS from turning into thirty pointless restarts? I'd like to hear it in the comments.
Prefer the story without the config? There's a shorter non-technical version on Medium: I taught my house to fix itself. The hard part was teaching it when not to.
Top comments (0)