An incident bot caught the CrashLoopBackOff at 3:12 a.m., proposed delete_pod, and the on-call approved it half asleep at 3:14. The new pod went Running a few seconds later. The ticket closed itself as remediated, and the graph went quiet.
At 3:52 the same alert fired again. Only now the incident channel already said resolved, the context from the first firing was forty minutes stale, and the on-call had spent their adrenaline once.
Nothing in that loop malfunctioned. Detection worked. The proposed fix was reasonable. A human approved the action, exactly like the architecture diagram promised. The only step that never happened is the one nobody built. Deciding whether the incident was actually over.
We automated the middle of the loop
Incident tooling has been eating the loop from both ends for a while. Detection went first. Alerts, anomaly detection, burn rates. Diagnosis is going right now, every incident product grew an LLM layer this year that reads the events and drafts a root cause. Remediation is increasingly a runbook behind a button. Delete the pod, roll back the deploy, scale the node group, with a human approving instead of typing.
Then the loop just ends.
The step after remediation, confirming the system is healthy again and not just momentarily quiet, mostly doesn't exist as a step. It exists as a person. Someone keeps a dashboard open for a while afterward. How long is a while? Whatever their attention span, their meeting schedule, or the hour of the night decides. That informal vigil is the verification loop at most companies. It appears in no diagram, which is exactly why it gets skipped on the nights it matters most.
Quiet is not resolved
The reason an experienced operator lingers on the dashboard is that they know something the bot doesn't model. A fresh pod is quiet mostly because it's fresh.
Restart a crashing service and you get a grace period for free. Connections are new, caches are cold but clean, and the request that triggered the crash hasn't arrived yet. If the incident came from a bad config value, a poison message in a queue, or one customer's traffic shape, the failure isn't gone. It's waiting for the next cron tick, the retry storm, or that customer's next request.
So sixty quiet seconds after a restart verify very little. Zero errors under zero requests proves nothing at all. The thing you actually want to know is narrower and harder. Did the original failure mode fail to reproduce under the conditions that produced it?
What the missing half would have to do
It has a shape, if you decide to build it.
Watch the signal that opened the incident, not a generic health check. If the incident was a 5xx spike on one route, the question is whether that route stopped 5xxing under real traffic. The liveness probe was never the thing that broke, so it can't be the thing that clears it.
Size the window by traffic, not by wall clock. Sixty seconds after a 3 a.m. restart might contain four requests. The window that means something is enough traffic to have re-triggered the failure, which on a quiet service might be twenty minutes and on a busy one ninety seconds. Fixed windows exist because they're easy to implement, not because they verify anything.
Treat idle as unknown. A service that has seen no traffic since the fix is not verified, it's untested. Those deserve different labels, because a human reads resolved and stops thinking about it.
And write the outcome down. Verified, still failing, or not enough traffic to tell, decided by a defined rule, recorded where the next responder will find it. Half the value of verification is that the 3:52 page arrives with the 3:14 context attached instead of a closed ticket that says everything is fine.
Why this is the expensive half
A false resolved is about the most expensive outcome an incident can have. The second firing lands with less urgency, because didn't we already fix that. It lands with stale context, because the ticket closed and took the thread with it. And it lands on a responder who already spent their focus once tonight. Same bug, higher cost, worse position.
The manual version also consumes exactly the wrong people. The engineer who knows what to watch after a risky fix is usually the most senior person in the room, and the work looks like nothing. Sitting there with a graph open. It doesn't demo well, and nobody writes a launch post about staring. So the effort keeps flowing to the halves of the loop that demo well, detection and remediation, while the half that decides whether any of it worked stays a vigil.
Automating remediation without verification doesn't remove the human from the loop. It promotes them from typing the fix to judging the fix, and judging was always the harder job.
The same hole, without the incident
Strip the incident away and the shape is still there.
Every deploy is a change to production followed by the same unowned question. Did it actually work? The pipeline's last observable event says succeeded, which describes the delivery, not the outcome. The first minutes after the change carry most of the evidence either way, and in most teams the entity processing that evidence is whoever happens to have a dashboard open. The informal vigil again, just without a ticket attached.
That's the corner I've been building in. Relivio watches the first fifteen minutes after a production deploy. It takes raw signals teams already emit, error logs, stack traces, deploy markers, and returns one verdict per deploy. STABLE, WATCH, or RISK, with the affected API and a recommended action, written down where humans and the automation downstream can both read it. It's the missing half built for the deploy case instead of the incident case. An explicit answer to whether it's actually fine now, produced by a defined rule, instead of by whoever stayed awake.
Incident bots are going to keep getting better at the middle of the loop. The half that decides whether any of it worked is buildable too. It's just not the half anyone demos.
I am building Relivio, a small verdict layer for the first 15 minutes after a production deploy. If you are working on something similar, or running into the same problem, happy to talk. relivio.dev
Top comments (0)