The Arson We Applaud

It's cool to be the engineer who fixes the outage at 3 AM. It's cooler to build the system that never paged anyone in the first place.

I've spent eleven years in banking and infrastructure, and if there's one pattern I keep running into, it's this: we've built an entire culture around celebrating the fix and almost none around celebrating the absence of the fire. The allnighter gets a shoutout in standup. The frantic Slack thread becomes a story people tell in interviews. The "great save" gets a line in the postmortem doc. Meanwhile, the quarter where nothing broke the actual goal of the job gets nothing. No one writes that post.

I don't think this is a people problem. I think it's a tooling and incentive problem, and it's one I've been trying to chip away at with a set of governance primitives I've been opensourcing under a project called agentsre.

Firefighting vs. arson

Here's the uncomfortable framing I keep coming back to: most of the incidents we praise ourselves for resolving heroically were, somewhere upstream, avoidable. Someone shipped a change without enough signal about current system health. Someone deployed into a service that was already sitting on a thin error budget. Someone (or increasingly, some agent) took an irreversible action against a wide blast radius without anyone in the loop asking "should we, right now, given everything else in flight?"

We reward the firefighting. We rarely examine the arson.

The engineers I trust most in this industry are the boring ones. Their systems selfheal. Their runbooks are short because the failure modes they'd need are rare. They automated the toil away instead of building an identity around absorbing it. That's not a flashy skill set. It doesn't generate a satisfying story arc. But it's the actual craft.

What "engineering the absence of heroics" looks like in practice

Talk is easy here, so I want to point at something concrete instead of just restating the philosophy. The newest piece in agentsre is a PreAction SRE Gate a small, composable guardrail that evaluates whether a statechanging action (a deploy, a config push, an autoscale event, a tool call made by an autonomous agent) should be allowed to proceed before it runs, based on the system's current ability to absorb that risk.

The shape of it is simple. You describe the action:

action = ActionRequest(
    name="rolloutpaymentsapiv42",
    blast_radius="region",
    reversible=True,
    initiated_by="agent:deploybot",
    metadata={"service": "paymentsapi"},
)

And a set of evaluators error budget health, blast radius, concurrent inflight changes, and how much track record the initiating agent has each contribute a bounded risk score with a plainEnglish reason attached. The gate combines those into ALLOW, HOLD, or BLOCK, weighted so a single severe risk factor can't get diluted by several calm ones.

The part I care about most is the agentorigin evaluator. As more actions in production systems start being initiated by autonomous agents rather than humans, the implicit risk judgment a human operator makes ("is now a good time to ship this?") doesn't happen automatically anymore it has to be engineered back in, explicitly, or it just disappears. An agent with no established track record gets treated more conservatively by default than a human operator would be. Trust gets earned from observed behavior over time, not assumed because the caller is automated.

Why this matters more than it used to

For most of my career, "reliability engineering" meant reacting well: good alerting, fast rollback, clean incident response. That's still necessary. But it's built around the assumption that a human is the one taking the risky action, and that some judgment is happening before the action fires. That assumption is eroding fast as agentic systems take on more autonomous execution in production environments. If we don't build the preaction judgment back into the systems themselves, we're going to get a lot more 3 AM pages, except now nobody will be able to tell you which agent, executing which plan, decided the timing was fine.

99.99% uptime was never about heroics. It's the absence of heroics, engineered on purpose, on the days nobody's watching and nothing gets written up.

The full implementation, along with the rest of the SLI and governance primitives I've been building for agentic infrastructure, is open source: github.com/Ajay150313/agentsre.

I'd like to hear from other people building or operating agentic systems what does your team's pre action risk story look like right now? Is there one, or is it still "the agent has good judgment, probably"?