I built a system that detects failure, decides what to do, acts on its own, and documents everything afterward — no human in the loop. The first time I tested it against a real failure, it got things wrong. Not once. Three times.
This isn't a confession of failure. It's the most valuable part of the whole project, and that's what I want to talk about.
Quick context
I built a portfolio platform simulating a simple e-commerce system — three microservices with a real dependency chain (orders → inventory → notifications), running on Kubernetes. On top of that, I layered the stack any serious SRE team runs: Prometheus and Grafana for observability, formal SLOs with error budgets, Chaos Mesh to inject failure on purpose, and — the most ambitious part — a Python controller that reacts on its own when something breaks.
The core idea: close the full loop. Detect → decide → act → learn. Not just monitor and wait for a human to wake up.
I documented every test as a "Game Day" — a structured exercise with a hypothesis stated beforehand, a real execution, and an honest write-up afterward, without dressing up what went wrong.
Game Day 005: the first time the controller actually acted
After weeks building the infrastructure, it was time to test the auto-remediation controller against a real failure — not a synthetic test, actual chaos breaking an actual service.
The plan: sustain an outage in the estoque-service (inventory), let Alertmanager fire the alert, and watch the controller react on its own.
I ran the test. Waited. And found three problems, one after another.
Problem 1 — the alert never reached the controller. Alertmanager was configured correctly, Prometheus was firing the right alerts, but nothing happened downstream. After digging, I found the cause: the Prometheus Operator automatically injects a namespace filter into every alert route — and my rules didn't carry that label. The alert was silently falling through into an empty receiver. No error anywhere in the logs. Fixed by adding the missing label.
Problem 2 — the controller acted on the wrong service. With routing fixed, the alert arrived. But the controller only knew how to look at the service that fired the alert (pedidos-service, the orders service) — it had no concept that the real root cause lived in a dependency (estoque-service). It restarted the orders service, which was perfectly healthy, while the actual problem kept running untouched.
Problem 3 — it lied about recovering. The kicker: even though nothing had actually been fixed, the controller declared "recovered" in 48 seconds. The cause: when the Prometheus query returned no data yet (which is normal right after any action, given the scrape interval), the code fell back to just checking whether the pod was "Ready" via the Kubernetes API. A ready pod isn't the same thing as a healthy service — but that's what the system was treating as success.
Three real bugs, found only because I tested against genuine failure instead of a scenario tame enough to succeed by default.
Game Day 006: fix it, then prove the fix works
The most interesting fix was the cross-service diagnosis one, because it required a real design decision, not just a bug patch.
I considered three approaches: a static dependency map baked into the code, an evidence-based approach (instrument the calls between services and query which downstream actually failed), and a broad sweep across every pod in the namespace. I went with the static map — with only three services, the topology is already known and simple, and an explicit map is auditable: anyone reading the code can see exactly what the controller knows. The evidence-based approach is the right call for when the service graph grows, but it would be unnecessary complexity at this stage.
I also changed the default behavior: previously, when the controller found no clear signal, it would blindly restart the whole Deployment. I removed that fallback. Now, if neither the alerting service nor any known dependency shows a signal, the controller takes no action — it logs the event and defers to a human. I'd rather have a system that admits "I don't know" than one that guesses.
Recovery confirmation now requires real data from Prometheus, always. No sample yet no longer means "success" — it means "keep trying."
Before burning more real error budget testing this (Game Day 005 had already consumed roughly 5% of the service's monthly budget in a single 16-minute test), I validated the new logic with a synthetic alert — no chaos at all, just simulating the payload Alertmanager would send. It confirmed the diagnosis correctly targeted the right service. Only then did I run the real test again, shorter than the previous one.
It worked — the controller correctly identified estoque-service as the root cause, even though the alert had come from pedidos-service.
But a new, honest, still-unresolved limit showed up: the chaos I'd configured killed a fresh pod every 15 seconds, repeatedly. In some cases, when the controller tried to recycle the specific pod it had identified, Chaos Mesh had already killed that pod again before the action completed — a 404, handled gracefully without crashing anything, but revealing that reactive per-pod remediation can't outrun a root cause that regenerates faster than the controller's reaction loop. I logged this as a known limitation, not something to bury.
Game Day 007: closing the learning loop
The last piece of the project was the automated post-mortem generator — every incident the controller processed should turn into a real report, enriched with Prometheus data, stored durably.
In the validation test with real chaos, everything worked as expected: alerts fired, cross-service diagnosis stayed correct, and the post-mortem was generated automatically with the real availability curve, the exact burn-rate expression that triggered, and even the 404 from Game Day 006 captured in the incident narrative.
But reviewing the numbers carefully, I found one more issue: the metric counting confirmed versus unconfirmed recoveries didn't add up to the right total. Two of the four incidents — specifically the ones where the action itself had errored — were invisible in that particular metric, because the code only had two buckets (confirmed / not confirmed), and "the action itself failed" was neither. It was a small inconsistency I only caught because I checked the numbers by hand instead of assuming "ran without errors" meant "correct."
What this taught me
Testing against real failure isn't optional — it's the whole point. Every one of these bugs would have sailed through any well-behaved synthetic test. They only surfaced because I simulated conditions that refused to cooperate.
A system that never failed hasn't been tested enough. If your chaos engineering only confirms what you already expected, you're testing your own hypothesis, not discovering real fragility.
Documenting what went wrong is more valuable than documenting what went right. A README that only shows success tells half the story. The Game Days that document "I expected X, Y happened instead, here's why, and here's what I changed" are what actually proves engineering ability.
Not every limit needs to be fixed immediately. The race between delete_pod and sustained chaos is still open in the project. That's fine — recognizing a limit, understanding why it exists, and consciously deciding it's not the priority right now is also a mature engineering decision.
Full code, architecture, and documentation for all 7 Game Days are public in the repository:
https://github.com/lucasferreiradealmeida/sre-self-healing-platform
If you work with reliability, observability, or chaos engineering, I'd love to hear your take — in the comments or a DM.
Top comments (0)