Uptime read 99.9%. Latency was fine. PagerDuty never made a sound. And for hours, an autonomous agent kept making bad decisions anyway.
That's not a hypothetical it's the failure mode I built a talk around this week for IEEE Region 3's Women in Engineering series, and it's one I think more SRE teams are about to run into as agentic tooling moves from "chatbot bolted onto Slack" to "thing that actually writes to production." The problem is that our entire observability stack was built to catch infrastructural failure. It has almost nothing to say about semantic failure the agent that's up, responsive, and confidently wrong.
Here's how I've been thinking about closing that gap.
The old model breaks at the causal layer, not the alerting layer
Traditional on-call is alert driven: a threshold fires, a human opens a laptop, and from there it's detective work. That model was already straining before agents entered the picture. In a microservice banking stack, a single payment-path failure can light up twenty services at once. Correlation says everything is related. It takes a human or increasingly, a causal model to say which one service actually started it.
This is where I've spent most of my research time this year, under something I call TraceCausalNet: build a live service graph from distributed traces (not a static architecture diagram), and when an anomaly fires, run Granger-style causality tests which service's past behavior actually predicts another's failure before ranking candidates by interventional impact: if I healed this node, what fraction of the symptoms would actually disappear? Then hand the on-call engineer a ranked top-3 with reasons, not forty red tiles.
In practice, across production incident data I've studied, that cut mean time to root cause by roughly 83%, with top 3 accuracy around 91%. The number that matters more than the percentage, though, is what it changes at 3 AM: the engineer isn't staring at a wall of red anymore. They're evaluating three ranked hypotheses with evidence attached.
Three gates before an agent is allowed to act
Diagnosis is the easy half. The harder question is: once you know what's wrong, should an agent be allowed to fix it itself?
I use three tests, and none of them are about model confidence:
Reversibility can we cleanly undo this if the agent is wrong? A certificate renewal with a health check rollback: yes. An irreversible ledger mutation: no, a human decides, full stop.
Blast radius what's the actual worst case? Bounded to one service, one change window: automation is thinkable. Account-wide, or spanning multiple tenants: never silent auto.
Novelty — have we actually seen this pattern before? Inside the historical distribution, auto is reasonable. Out of distribution, the agent's job is to gather evidence and escalate not guess with confidence.
If a proposed action doesn't clear all three, it doesn't execute unattended. This sounds obvious written down. It's shockingly easy to skip once an agent's diagnosis has been right nine times in a row.
Measure the agent, not just the server
This is the part most teams are missing entirely, and it's the part I actually open-sourced. HTTP 200 tells you the server responded. It tells you nothing about whether the decision was correct. So I track four signals instead, published as agentsre:
DQR (Decision Quality Rate) did the agent's choice stay near a healthy behavioral baseline? (Leading indicator.)
TIE (Tool Invocation Efficiency) is the agent compensating with extra tool calls because its context is quietly rotting? (Early warning.)
HER (Human Escalation Rate) are people having to take the wheel more often than last week? (Lagging cost.)
AQDD (Approval Queue Depth Drift) is work piling up unapproved while every dashboard stays green? (The silent one and the one that catches you off guard if you're not watching it.)
These feed a progressive autonomy ladder: when the signals degrade, the agent doesn't get shut off abruptly it climbs back down, from full autonomy to human approval on writes to read-only to fully suspended. Autonomy, in this model, is a privilege the system can lose. Not a default state it holds by inertia.
bash
pip install agentsre
Repo: github.com/Ajay150313/agentsre · LangChain adapter: agentsre-langchain
Turning post-mortems into something that compounds
The last piece is memory. Most incident knowledge lives in one senior engineer's head, or in a post mortem doc nobody rereads until the next nearly-identical outage. I've been experimenting with having an LLM extract structured knowledge from unstructured incident write ups services, failure modes, contributing factors, remediations and storing it as a graph (service failed-because→ cause fixed-by→ action) so that when a new alert fires, the system retrieves analogous incidents in seconds instead of forty seven minutes of tribal memory. Across the corpus I've studied, this surfaced 14 recurring anti patterns fixing those is estimated to prevent roughly six high severity incidents a year.
What I'd tell a team starting this today
Don't start with the agent. Start with the measurement layer. If you can't tell me your current Decision Quality baseline, you're not ready to hand an agent write access you just don't have a way to know when it's drifted.
Curious what other teams are using to catch "technically up, actually wrong." It feels like the most underdiscussed failure mode in the agentic-SRE conversation right now.
Top comments (0)