DEV Community: Ajay Devineni

Your AI Agent Has a 95% Success Rate. Your Workflow Has a 36% Success Rate. Here's the SRE Fix.Tags

Ajay Devineni — Fri, 17 Jul 2026 01:36:50 +0000

Your AI Agent Has a 95% Success Rate. Your Workflow Has a 36% Success Rate. Here's the SRE Fix. number published this week has been sitting with me since I read it.At 95% per-step accuracy — which is optimistic for current LLMs — a 10-step agent workflow succeeds roughly 60% of the time. At 20 steps, you're at 36%.That math comes from Lusser's Law, which reliability engineers have known since the 1950s. The reliability of a series of components equals the product of their individual reliabilities. We apply it to hardware systems, distributed architectures, deployment pipelines. We have not been applying it to AI agent workflows — and the industry is starting to feel the consequences.Microsoft's Azure SRE Agent team found that problems requiring more than four handoffs almost always failed. That's not a benchmark result. That's a production observation from the team that built one of the most sophisticated AI SRE systems in existence.Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 because of costs, unclear value, or inadequate risk controls.The inadequate risk controls part is what I want to address. Because the risk is calculable. And if the risk is calculable, it can be governed.Why the Demo Number LiesDemos usually hide compounding failure because they only show two or three steps. Production environments are usually five or more steps over messy inputs and edge cases.A demo that shows an agent complete a task in three steps with clean data is showing you 0.95^3 = 0.857 — 85.7% reliability. That looks good. Your stakeholders are impressed.The same agent deployed to a production incident investigation workflow with 10 steps is giving you 0.95^10 = 0.599 — just under 60% reliability. In SRE terms: your agent is failing roughly four out of ten production incident investigations, silently, without throwing an error.This mathematical reality makes autonomous multi-step workflows fundamentally challenging at production scale, requiring teams to rethink how they architect agent systems.Rethinking the architecture is exactly right. But it's not enough to know the math. You need a framework for governing the math — which is what SRE discipline gives you.The Missing SLO LayerWe have SLOs for services. We have error budgets for APIs. We have reliability targets for databases.We have almost none of that for agent workflows.A microservice either returns a 200 or it doesn't. AI agents return responses on a spectrum from correct to confidently wrong. Everyone's shipping agents. Almost nobody has a framework for deciding how much failure is acceptable.That's the gap. And it's the same gap SRE filled for microservices fifteen years ago.The answer isn't "make every step more reliable." At scale that's impossible — you can't prompt-engineer your way to 99.9% per-step accuracy across a complex incident investigation workflow. The answer is what SRE has always done: define an acceptable failure rate, measure against it, and design the system to fail gracefully when it breaches.Introducing the Workflow Reliability BudgetA Workflow Reliability Budget (WRB) applies error budget thinking to multi-step agent workflows rather than to individual services.The calculation:WRB = R_target / R_actual

where R_actual = (P_step)^n (Lusser's Law for the full workflow)
and R_target = your acceptable success rate for this workflow classFor an incident investigation workflow with 8 steps and 95% per-step accuracy:pythonfrom agentsre.slo_burn import AgentSLOBurnTracker, SLOTarget
import math

def calculate_workflow_reliability(
per_step_accuracy: float,
num_steps: int
) -> float:
"""
Apply Lusser's Law to calculate end-to-end workflow reliability.
This is the number your demo hides and your production exposes.
"""
return per_step_accuracy ** num_steps

The math your stakeholders need to see

per_step = 0.95
steps = 8
workflow_reliability = calculate_workflow_reliability(per_step, steps)

print(f"Per-step accuracy: {per_step:.0%}")
print(f"Workflow steps: {steps}")
print(f"End-to-end reliability: {workflow_reliability:.1%}")

Output: End-to-end reliability: 66.3%

Now wire this to your SLO framework

tracker = AgentSLOBurnTracker(
agent_id="incident-investigation-agent-v2",
task_class="multi-step-incident-investigation"
)

Workflow-level SLO — not per-step accuracy

tracker.add_slo(SLOTarget(
metric_name="WorkflowCompletionRate",
target_pct=90.0, # 90% end-to-end success
window_days=30,
good_threshold=1.0, # 1.0 = workflow completed successfully
higher_is_better=True
))

DQR at workflow level — not just per tool call

tracker.add_slo(SLOTarget(
metric_name="WorkflowDQR",
target_pct=85.0, # 85% of completed workflows with correct output
window_days=30,
good_threshold=0.8,
higher_is_better=True
))The key insight: WorkflowCompletionRate is not the same as per-step accuracy. A workflow that completed but produced a wrong final answer counts as a failure against DQR even if every intermediate step appeared to succeed. That's the distinction most teams miss.The Three Architecture FixesThe math points to three specific changes that actually move the reliability number:Fix 1: Shorten the chainAchieving high workflow reliability requires either very high per-step accuracy (above 99%), short workflows (fewer steps), or resilience architecture that detects and corrects errors before they compound.Every step you remove from an agent workflow is a multiplicative reliability improvement. An 8-step workflow at 95% per-step accuracy is 66% reliable. A 5-step workflow at the same per-step accuracy is 77% reliable. That's an 11-point improvement from design alone, not from model improvement.In practice: ruthlessly question every step in your agent workflow. If a step retrieves context the agent has already retrieved in a previous step, remove it. If a step validates something that was already validated, combine them. Shorter chains are not lazy design — they are reliability engineering.Fix 2: Add intermediate checkpointsError budgets force you to confront this compounding math instead of pretending each step is independent.The error doesn't compound uniformly — it compounds through dependency. An incorrect output at step 2 feeds into step 3, which builds on wrong context, which makes step 4 worse. By step 6, the agent is reasoning on top of accumulated wrongness.Intermediate checkpoints break this chain. After every two or three steps, the workflow validates intermediate output before continuing. Not a human checkpoint — a programmatic DQR check on the intermediate result.pythonfrom agentsre.reasoning_trace import AgentDecisionTrace
from agentsre.eval_pipeline import EvalRun, EvalCase

def checkpoint_intermediate_output(
agent_id: str,
step_number: int,
intermediate_output: str,
expected_properties: list,
trace: AgentDecisionTrace
) -> bool:
"""
Validate intermediate output before the next step consumes it.

If this checkpoint fails, the workflow stops and replans
rather than compounding the error through subsequent steps.

Returns True if output passes validation. False triggers replan.
"""
# Check each required property of the intermediate output
failed_properties = []
for prop in expected_properties:
    validator_fn = prop.get('validator')
    if validator_fn and not validator_fn(intermediate_output):
        failed_properties.append(prop.get('name'))

if failed_properties:
    # This is a replan event — RTD increments
    trace.record_replan(
        reason=(
            f"Step {step_number} intermediate output failed "
            f"properties: {failed_properties}. "
            "Replanning rather than compounding."
        ),
        tool_that_failed=f"step_{step_number}_output",
        new_plan=f"Retry step {step_number} with corrected inputs"
    )
    return False

return TrueFix 3: RTD as your workflow health signalRTD (Reasoning Trace Depth) already tells you when compounding is happening. When an agent re-plans three times in a 10-step workflow, it's because earlier steps produced outputs the agent couldn't work with cleanly. RTD rising is your leading indicator of compounding failure before the final output reveals it.The signal chain for workflow reliability:RTD rising → intermediate outputs degrading → DQR declining
 ↑                    ↑                        ↑

earliest signal mid-workflow signal lagging signalIf you're watching only DQR at the workflow level, you're watching the lagging signal. Wire RTD monitoring to your intermediate checkpoint layer and you catch compounding before it propagates through the full chain.The Workflow Reliability Dashboard Query# CloudWatch Insights — workflow reliability trend
fields @timestamp, agent_id, reasoning.rtd, outcome.task_completed,
quality.confidence_proxy, outcome.latency_ms
| filter trace_type = 'agent_decision_trace'
| filter agent_id = 'incident-investigation-agent-v2'
| stats
avg(reasoning.rtd) as avg_rtd,
sum(outcome.task_completed) / count() * 100 as completion_rate_pct,
avg(quality.confidence_proxy) as avg_confidence
by bin(1d)
| sort @timestamp ascIf completion_rate_pct is tracking below your WRB target, look at avg_rtd in the same window. Rising RTD before declining completion rate means compounding is starting — you have a window to intervene before the workflow-level SLO breaches.What This Means for Your Agent DesignThe teams that ship design short, checkpointed chains. The teams that stall keep adding steps and hoping.Lusser's Law isn't telling you agents can't work. It's telling you the design constraints for making them work reliably at scale. Fewer steps. Checkpoints between steps. RTD monitoring to catch compounding early. Workflow-level SLOs rather than per-step accuracy benchmarks.The math is the same math reliability engineers have been applying to hardware and distributed systems for decades. The discipline already exists. The application to AI agent workflows is new — but the principles are not.All frameworks referenced in this post are in the agentsre library at github.com/Ajay150313/agentsre. MIT licensed.Ajay Devineni | AWS Community Builder | Senior SRE/Platform EngineerDEV.to Tags:

sre #agenticai #devops #aws

Your AI Agent Passed the Gate. Now It's in Production. Here's Why That's When the Real SRE Work Starts

Ajay Devineni — Sun, 12 Jul 2026 10:32:56 +0000

AWS expanded their DevOps Agent four days ago. It now validates code autonomously before production — assessing changes, running tests, surfacing risks without waiting for a human to trigger it.
I want to write about what that means for SRE practitioners, because I think the conversation around this capability is missing something important.
The narrative is usually framed around speed. How much faster does code get validated? How much MTTR improves? Those numbers are real and they matter. But they measure the easy part.
The hard part isn't getting an AI agent to validate a release. The hard part is knowing whether to trust what it found and what to do when it missed something.
I've been building SLI frameworks for AI agents in production for seven months. Every framework I've built has been motivated by the same underlying question: how do you govern an actor that moves faster than your ability to verify it?
This post is the answer I've landed on for the release validation case specifically.
The Gap Between Observable and Interpretable
MIT Sloan published research showing people are 2.8 times more likely to trust AI systems they can interpret — not just observe, but interpret. The difference matters enormously in practice.
Observable means you can see what happened. The agent ran. It approved the release. A log entry exists.
Interpretable means you understand why. What signals did the agent evaluate? What did it decide to weight heavily? What did it see but discount? What would have caused it to reject instead of approve?
Most AI release validation tools give you the first. Very few give you the second. And in a postmortem after a release caused a production incident, the audit log alone doesn't answer the question you actually need answered: what did the agent see, and why did it say yes?
What Happens When It Gets It Wrong
Last month, a team shared a scenario I've heard variations of a dozen times. Their AI release validation tool approved a change. The change caused a latency regression in a payment processing path that only appeared under specific traffic conditions. Staging didn't catch it. The agent didn't catch it either.
The postmortem question was: what did the agent evaluate? The answer was: we have an approval timestamp and a summary. We don't have a reasoning trace. That's the gap. And it's not a vendor failure — it's a design gap that the SRE team needs to close, because the vendor's job is to ship a capable agent. The SRE team's job is to govern it.
The Three Things You Need Before Trusting AI Release Validation
I've been building toward this over seven months of framework work. Here's what I've concluded about the minimum viable governance layer for AI release validation in a regulated production environment.

A reasoning trace, not just an audit log An audit log tells you what the agent did. A reasoning trace tells you how it got there. One structured record per validation run — not per tool call capturing the initial hypothesis, what signals the agent evaluated, how it weighted conflicting evidence, and what would have caused a different outcome. This is directly related to RTD (Reasoning Trace Depth). A validation agent that evaluates twelve signals in parallel and returns a summary has high RTD and low interpretability. A validation agent that forms a hypothesis, follows the causal chain, and surfaces its reasoning has low RTD and high interpretability. Those two agents might produce the same approval decision. They produce very different postmortem data when they get it wrong.
A pre-validation state check, not just a post-validation summary Before an AI agent validates any release, it should check three things about the current state of the system it's validating into: Error budget remaining if the target service is already burning error budget faster than expected, a release that introduces even a 0.1% error rate increase is a different risk than the same release on a healthy system. Recent change velocity how many changes have been deployed to this service in the last 24 hours? High velocity plus AI-validated release is a compound risk that deserves human review regardless of what the agent found. Blast radius of what's being validated — which downstream services could be affected if the validation missed something? An agent validating a change to a payment processing path needs different scrutiny than one validating a change to an internal dashboard. None of these checks are about the agent's capability. They're about whether the conditions are right for autonomous action.
A clear escalation path when confidence is below threshold AI release validation agents don't always produce binary confident decisions. Sometimes the evidence is ambiguous. Sometimes there's a conflict between what the static analysis found and what the load test showed. Sometimes the agent finds something it can't classify. The question is what happens then. Does the agent approve anyway? Does it block? Does it escalate to a human with a specific question? The answer should always be the third option when confidence is below a defined threshold — and that threshold should be set by the SRE team, not by the vendor's default configuration. The Pre-Action SRE Gate from the agentsre library handles this for remediation agents. The same pattern applies here: if the agent's confidence is below threshold, the deployment gates pending human review of the specific conflicting signals. Not a generic "human approved" checkbox a specific review of what the agent couldn't resolve. The Production Readiness Checklist for AI Release Validation Before you trust an AI agent to validate releases autonomously in a regulated environment, here's what needs to be in place: The agent produces one reasoning trace per validation run not a summary, a trace you can replay in a postmortem. Your error budget is monitored per service, and the validation agent reads it before approving any release. A confidence threshold is defined and tested below threshold, the agent escalates rather than decides. A rollback procedure exists that doesn't depend on the same agent that validated the original release. A revert drill has been practiced in the last 30 days you've confirmed your team can disable the AI validation layer and revert to human-only release management within a defined time window. The last point is the one most teams skip. When the AI validation layer produces a bad approval and causes an incident, your team needs to be able to operate without it immediately — not after they've spent an hour figuring out how to disable it. Where This Sits in the Arc If you've been following this series: Post 1 established that AI agents need on-call rotation discipline. Post 4 introduced DQR, TIE, HER, AQDD as the measurement layer. Post 11 introduced RTD as reasoning observability. Post 13 introduced the Pre-Action SRE Gate. This post applies all four to the release validation case the specific scenario AWS just made mainstream with their DevOps Agent expansion. The governance layer doesn't come from the vendor. It comes from the SRE discipline your team already has, applied deliberately to a new class of actor. All the tools for building it are in the agentsre library at github.com/Ajay150313/agentsre. MIT licensed. Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer

Training a Model Is a Research Skill. Operating One Is an Infrastructure Discipline.

Ajay Devineni — Thu, 09 Jul 2026 01:36:40 +0000

Why the AI infrastructure hiring crisis is hiding in plain sight — and what production-grade ML ops actually requires

I've watched a GPU bill hit five figures in a single day because nobody had a cost circuit breaker on the inference endpoint.

The model was good. The research team had done real work. The fine-tuning was solid. Nobody had thought about what happens when you put a latency-sensitive LLM inference service behind real production traffic without auto-scaling bounds, without request rate limits, without a cost anomaly alert, and without a rollback path for the model weights.

The GPU kept serving. The bill kept climbing. The on-call engineer hired for their PyTorch experience, not their production infrastructure instincts — had no runbook for "inference endpoint is costing $800 per hour and accelerating."

This is the most predictable failure mode in AI infrastructure right now, and it's happening at scale.

The Hiring Category Error

The résumé filter that's creating this problem is specific: companies are screening for ML credentials PyTorch, Hugging Face, "fine-tuned a model on custom dataset," Transformers library familiarity — and treating those as sufficient qualification for production infrastructure ownership.

They are not the same job.

Training a model is a research skill. It requires deep understanding of loss functions, architecture choices, data pipeline quality, evaluation methodology, and the patience to run experiments that don't converge. It is hard, skilled, important work.

Operating a model in production is an infrastructure discipline. It requires capacity planning, GPU memory profiling under concurrent load, observability instrumentation that captures what matters for inference (token throughput, first-token latency, KV cache hit rate, queue depth), cost control systems with hard limits, blast radius analysis for every deployment, and a rollback path that actually works at 2:00 AM.

These overlap at the edges but they are not the same job. Hiring for one and expecting the other is how you get a five-figure GPU bill with no circuit breaker.

What Changes When You Operate at Scale

Here is what production ML infrastructure actually demands that doesn't appear on a research-oriented résumé:

GPU memory is not forgiving. A batch size that runs cleanly on a development P3 instance OOMs at 2x concurrent load on a P5 because nobody modeled the KV cache growth under real traffic patterns. The error is silent on the surface — the request fails, the client retries, the retry amplifies the load. A platform engineer who has done capacity planning knows to model this. A researcher who has done fine-tuning often doesn't.

Model weights are large artifacts with deployment complexity. A 70B parameter model in BF16 is 140GB. Getting that artifact from your model registry to your inference fleet reliably, with version control, with the ability to roll back without a full redeploy, with pre-warming on the new instances before you shift traffic — this is a software deployment problem, not a machine learning problem. The skills are in the infrastructure domain.

Inference is a latency-sensitive service. Everything your organization learned about p99 latency, tail latency management, connection pooling, load balancing, circuit breaking, and graceful degradation applies here. The model being impressive doesn't exempt the service from those requirements. Users experiencing 8-second p99 latency on an LLM endpoint don't care that the model architecture is elegant.

Network egress between regions is expensive at GPU scale. If your training cluster is in us-east-1 and your inference fleet is in us-west-2 and your model artifacts are large and you're pulling them frequently, someone is going to get a bill that surprises them. This is a cost architecture decision. It requires the same thinking as any other distributed system where data movement has cost.

Cost anomaly detection is non-negotiable. GPU compute is expensive at a rate that makes traditional cost anomaly thresholds meaningless. A virtual machine getting into a runaway state costs you maybe $50/hour. A GPU cluster in a bad state costs you $2,000/hour. Your cost alerting needs to reflect this. Most teams deploy their ML infrastructure with the same cost alerting thresholds they use for their web services, and discover the gap during an incident.

The "Boring in Production" Standard

The teams that will build durable AI infrastructure over the next few years aren't the ones with the fanciest models. They're the ones who took the research output seriously enough to subject it to the same operational rigor they apply to everything else.

Boring in production means:

The inference endpoint has been load-tested at 3x expected traffic and the degradation behavior is documented and understood
GPU utilization, memory pressure, queue depth, token throughput, and cost per request are all in the same observability dashboard
Model weight rollback takes less than ten minutes and has been practiced in a non-production environment
There is a hard cost ceiling per hour that triggers an automatic circuit breaker and pages the on-call engineer before the bill becomes a conversation with finance
The deployment pipeline is identical to every other service deployment pipeline — version controlled, reviewed, deployed through environments, observable from day one

None of this is specific to ML. All of it is infrastructure discipline applied to a new category of workload.

The GPU is expensive compute. The model weights are large artifacts. The inference service is a latency-sensitive API. The fundamentals didn't change. The job of the infrastructure engineer is to make sure the people who believe the fundamentals changed are not the ones with production access.

What Production-Grade MLOps Actually Requires

I've been building out the agentsre library with primitives that operationalize these patterns for AI/ML workloads — GPU cost circuit breakers, inference SLI definitions, model artifact governance gates. The design philosophy: every ML workload gets the same reliability treatment as every other production service. No exceptions for the model being impressive.

The primitives that have been most useful in production:

A GPU cost rate monitor that tracks cost-per-hour against a session budget and severs inference authority when the rate exceeds the threshold — before the bill, not after.

An inference SLI framework that defines first-token latency, token throughput, and queue depth as first-class SLIs with error budgets, so reliability of the inference service is measured the same way reliability of any other critical service is measured.

A model artifact governance gate that validates weight checksums, lineage provenance, and deployment window compliance before any model version reaches production traffic.

These aren't ML tools. They're SRE tools applied to ML workloads. That's the distinction that matters.

The Bottom Line for Hiring Teams

If you're building an AI infrastructure team and your entire interview loop is testing PyTorch knowledge and model architecture understanding, you are hiring for the research half of the job and leaving the operations half to chance.

The operations half is where the five-figure GPU bills live. It's where the 2:00 AM incidents live. It's where the systems either become boring and reliable or stay exciting and expensive.

Pair your ML talent with platform engineers who have lived through production incidents, who have oncall rotations, who have been paged because of something they shipped, who have a visceral understanding of blast radius and rollback and "what does this look like at 10x load."

Make them fight it out until the system is boring.

Boring in production is the highest compliment you can pay an AI platform.

What's the most expensive production lesson your team learned from ML infrastructure? Drop it in the comments the more specific, the more useful for everyone trying to avoid the same mistake.

Tags: #MLOps #AIInfrastructure #SRE #PlatformEngineering #CloudNative #AIOps #DevOps #GPU #LLMOps

Your AI Agent Has an IAM Role. That's Not a Feature It's an Uncontrolled Blast Radius.

Ajay Devineni — Tue, 07 Jul 2026 01:53:59 +0000

Your AI Agent Has an IAM Role. That's Not a Feature It's an Uncontrolled Blast Radius.

Why traditional identity models fail non-deterministic systems, and what enforcement actually looks like at the platform layer

At 2:14 AM on a Friday, an agentic remediation system I was responsible for made fourteen sequential API calls to AWS in eleven seconds.

The agent had detected an anomalous CloudWatch alarm. It retrieved a relevant runbook. It synthesized a remediation plan. Then it executed that plan — and each execution step produced a new signal the agent interpreted as requiring another action. Fourteen calls. Eleven seconds. No circuit breaker.

The IAM role didn't care. Every call was authenticated. Every call was authorized. The cloud provider executes what it receives. It has no concept of whether the call was generated by deterministic application code or a language model in a hallucinated reasoning loop.

We got lucky that night. The blast radius was contained to a single ECS cluster not serving production traffic. But the architectural failure was real, and it had nothing to do with the quality of the agent's reasoning.

The Category Error We're Building On

There's a widely-shared mental model in teams deploying infrastructure agents right now: if the model reasons well, the system is safe. Improve the prompt. Add a system message. Constrain the reasoning with better instructions.

This is the wrong level of enforcement.

Traditional IAM was designed for deterministic code. When you write an application that calls ec2:TerminateInstances, you know exactly when that call will be made, with what parameters, under what conditions. The call path is static. The IAM policy is a guard on a known, bounded behavior.

A language model doesn't have a call path. It has a reasoning trace that produces a call path at runtime one that changes based on the prompt, the context window, the retrieved documents, the tool call history, and whatever the model decides is the next logical step. The IAM role the agent holds doesn't know any of that. It knows the API call that arrived. It executes.

This means that when a prompt injection payload appears in a retrieved document a log file, a ticket description, a runbook that's been tampered with the execution authority the agent holds becomes the blast radius of that injection. The same IAM role that handles legitimate remediation handles the injected command. The cloud doesn't distinguish.

Prompt engineering is not a security control at this layer. It is a probabilistic influence on model behavior. Probabilistic is not deterministic. Infrastructure security requires deterministic enforcement.

The Three Failure Modes in Practice

Reasoning loops that become API storms

A model gets into a state where each tool call result is interpreted as evidence that another tool call is needed. Without a runtime circuit breaker, this generates hundreds of API calls before a human notices rate limit exhaustion on critical APIs, unbudgeted cost spikes, and potentially self-inflicted service degradation on the infrastructure the agent was supposed to protect.

Prompt injection through context retrieval

An agent retrieving context from a knowledge base, ticket system, or log aggregator is reading content that may have been authored by an adversary. A log line containing an instruction to scale all ASGs to maximum capacity is not a theoretical threat. Agents that pass retrieved content directly into their reasoning context without sanitization are vulnerable. The IAM role has no "was this call generated by an injection payload?" check.

Authority conflation

The agent finds a valid path to resolve an alert and executes it. The path has side effects — deleting a deployment that was intentional, terminating an instance running a scheduled batch job, modifying a security group that breaks a dependency. The agent had authority. The action was within scope. Nobody had established that "synthesizing a remediation plan" and "executing against live infrastructure" are two separate authority levels that should be enforced separately.

What Deterministic Enforcement Actually Looks Like

The principle: an AI agent should be allowed to reason, synthesize, and recommend. The decision to execute passes through a deterministic policy engine with no LLM in the critical path.

This isn't new. It's how safety-critical systems have always been designed. The novel problem is that most teams deploying infrastructure agents haven't made this separation explicit in their architecture.

Separation of Intent and Action

The agent produces a proposed action: a structured object describing what it wants to do, to what resource, with what parameters. That object passes to a policy engine — a deterministic validator with no LLM involvement — that checks: Is this resource tagged production? Is this action class permitted at the current error budget level? Has this resource been modified in the last N minutes?

Approved: execution proceeds. Rejected: the agent receives a structured refusal, the attempt is logged, and the agent has no path to self-authorize beyond what the policy engine allows.

Runtime anomaly detection with automatic authority severance

Tool-calling behavior is tracked at runtime: calls per minute, cost per session, resource types touched, action class frequency. When a metric crosses a threshold — more than twenty API calls in sixty seconds, more than ten distinct resource modifications in a session, cost exceeding a per-session budget the circuit breaker severs tool calling authority for the session and pages the on-call engineer.

The agent doesn't get a warning. Its tool access is revoked at the infrastructure layer. It can continue reasoning and producing recommendations. It cannot execute.

Isolated runtime containment

Any agent executing commands against infrastructure should run in a short-lived sandbox with scoped credentials, network egress restrictions, and a maximum session lifetime. When the session ends, the credentials expire. The blast radius of any single session is structurally bounded by the sandbox's resource scope not by the agent's self-restraint.

The Architecture Change That Mattered

After the fourteen-call incident, we made three changes. None of them touched the model.

A runtime session monitor tracked API calls per sixty-second window per agent session. Above fifteen calls, the session's IAM role was suspended via a resource-based policy update and an alert fired. The agent could still reason. It could not execute.

A pre-execution policy gate sat between the agent's proposed action and any live API call. The gate checked resource tags, action class, and a cooldown registry preventing the same resource from being modified more than once per ten minutes. Every gate decision approved or rejected went to CloudWatch with full context.

Agent execution moved into short-lived ECS tasks with session scoped IAM roles assumed at task start and automatically expired at task end. No persistent credentials. No carry-over between sessions.

The agent's reasoning didn't change. The architecture around it changed. That distinction is the entire point: the safety guarantees needed for autonomous infrastructure agents cannot be delegated to the model. They have to be built into the system.

The Question for Platform Teams

If you're deploying autonomous agents that touch live infrastructure, the design review question isn't "does the model reason well enough to be safe?"

The question is: if the model reasons incorrectly — bad prompt, stale context, prompt injection payload, hallucinated tool-calling loop — does your platform architecture bound the blast radius, or does the agent's IAM role determine it?

If the answer is "the IAM role determines it," you have an uncontrolled blast radius.

The controls exist. The architecture patterns are established. The question is whether we build them before the incident, or after.

How is your team decoupling the agent's reasoning layer from your cloud infrastructure permission boundaries? The more specific the implementation detail, the more useful for everyone building in this space.

RAG vs. MCP Is the Wrong Question Here's the Right One

Ajay Devineni — Sun, 05 Jul 2026 20:35:02 +0000

Why treating this as an architectural tradeoff is setting your production agents up to fail

I watched a team's agentic remediation system act on a runbook that was fourteen months stale.

The RAG retrieval worked correctly. The document ranked highest by cosine similarity. The agent interpreted that retrieval as authorization to execute. The live environment was meaningfully different from what the document described. The remediation made things worse.

Nobody had explicitly defined where knowing ends and doing begins.

That is the actual production risk hiding inside the "RAG vs. MCP" framing. Not which technology you pick. Whether you have drawn a hard line between your agent's knowledge layer and its execution layer — and whether that line is enforced in code rather than trusted in assumptions.

The Category Error

RAG and MCP don't compete. They answer completely different questions.

RAG answers: what does the agent know?

Runbook context. Historical incident patterns. Service documentation. Architecture decision records. Compliance policy text. Everything the agent needs to reason about a situation before it touches anything.

MCP answers: what can the agent do?

Live metric fetching. System state queries. Remediation execution. API calls against real infrastructure. All the actions that have a blast radius.

These two layers have fundamentally different risk profiles. A RAG query that returns a stale document is a reasoning error. An MCP call that executes against a live system based on a stale document is an incident.

The conflation of these two layers treating "I retrieved something relevant" as equivalent to "I have permission to execute it" removes the safety layer that should sit between them.

What the Conflation Looks Like in Practice

Pattern 1: RAG-to-execution without a gate

An agent retrieves documentation describing how to handle high CPU utilization on a service. The document is from before a major infrastructure migration. The agent, having retrieved relevant context, proceeds directly to execution — restarting pods that no longer exist in the topology described, or modifying configurations that have since moved to a different management plane.

The retrieval confidence was high. The execution authority was never scoped. There was no check between "I know something relevant" and "I will now act on it."

Pattern 2: Knowledge queries routed through execution tooling

The inverse failure. Teams that don't design the boundary explicitly tend to route everything through the heaviest available tool. A query that only needs to retrieve context ends up going through MCP tooling with live-system access — adding latency, consuming rate-limited API quota, and introducing non-determinism into what should be a deterministic knowledge lookup.

The agent has more execution authority than it needs for the task. This is a blast-radius problem waiting for the right failure condition.

The Boundary as a Reliability Control

In SRE practice, we talk about blast radius as a design constraint. You scope the potential damage of any failure before it happens, not after. Circuit breakers, canary deployments, feature flags, staged rollouts — all exist to keep the failure surface bounded.

The RAG/MCP boundary is the same class of control applied to agentic systems.

The design question isn't "RAG or MCP?" It's: at what point in this agent's reasoning flow does read-only context become a candidate for execution authority, and what explicit gate exists at that transition?

That gate needs to be deterministic. It needs to be auditable. And it needs to exist in code, not in the assumption that the model will reason correctly about the distinction every time.

What an Explicit Boundary Looks Like

A well-designed agentic system for infrastructure operations has three distinct layers with explicit handoff points.

Layer 1 — Situational awareness (RAG)

The agent retrieves runbook context, historical incident data, service topology, and policy constraints. This layer is read-only with no execution authority. The output is a structured context package: what the agent knows about the situation, what precedents exist, what the operational constraints are.

Layer 2 — Decision gate

An explicit check before any execution authority is granted. This is where you validate: Is the retrieved context fresh enough to act on? Does the proposed action fall within the pre-approved blast radius? Has a human review been configured for this action class? Is the error budget healthy enough to absorb a remediation attempt?

This layer is where most agentic systems have nothing. The model reasons through it implicitly. That implicit reasoning is not a reliability control.

Layer 3 — Execution (MCP)

Scoped tool calls with explicit permission boundaries. The agent can fetch live metrics, read current system state, and execute approved remediation actions. The scope of what's available in this layer is determined by what passed through Layer 2 — not by what the model decides is appropriate at runtime.

The War Story That Sharpened This for Me

A service I was responsible for had an agentic component handling routine disk pressure events. The pattern: detect high disk utilization → retrieve cleanup runbook → execute cleanup steps.

For eight months it worked correctly. Then we rotated to a new storage backend. The cleanup runbook in the knowledge base hadn't been updated.

The agent retrieved the old runbook with high confidence — service name matched, symptom description matched, top result by similarity score. It executed against paths that no longer existed as described. The cleanup didn't make things catastrophically worse, but it also didn't fix anything, and it consumed enough API quota that the legitimate remediation attempt that followed hit rate limits.

The failure wasn't in the retrieval. The failure was that there was no freshness check between retrieval and execution. No gate asking: is this document current enough to act on?

After that incident we added three architectural controls:

A document freshness gate on any retrieved context informing an execution decision. If document age exceeded a configured threshold for the action class, the agent escalated to a human.

A scope limiter on every MCP tool call, tying the available action set to the specific remediation class that was approved — not to everything the agent had tool access to.

A dry-run mode for remediation actions above a configured blast-radius threshold, staging planned changes as a human-reviewable diff before executing.

None of these are model improvements. All of them are architectural controls. The model's reasoning didn't change. What changed was the system around it.

The Question Worth Asking Your Team

If you're building or operating infrastructure agents right now, the design review question isn't "are we using RAG or MCP?" Both answers are probably yes.

The question is: is the boundary between your knowledge retrieval layer and your execution layer explicitly enforced in code, or is it implicitly trusted in the model's reasoning?

If the answer is "the model handles that," you don't have a boundary. You have an assumption.

Assumptions are not reliability controls.

Drop your approach in the comments — how are you enforcing the knowing/doing boundary in your agentic systems? The more specific the implementation detail, the more useful it is for everyone building in this space.

Tags: #AgenticAI #MCP #RAG #SRE #PlatformEngineering #CloudNative #AIOps #Observability

The Production Tax: What SRE Taught Me That Code Reviews Never Could

Ajay Devineni — Wed, 01 Jul 2026 00:24:14 +0000

A practitioner response to the "SREs are glorified sysadmins" conversation

I've been paged at 3:00 AM for an unrotated certificate that silently expired and took down an entire checkout flow.

I've watched a Kubernetes cluster pass every chaos test we threw at it, then surprise the entire team the first time a real Availability Zone went dark — on a Friday night, naturally.

I've reviewed IAM policies that looked perfectly clean in a pull request, then watched that same policy lock out a critical service account during a regional failover window because nobody had tested the assumed-role chain under actual failover conditions.

None of those failures were algorithmic. Every single one was operational.

This is the part of the job nobody explains in a system design interview.

The Staging Lie

There is a specific flavor of overconfidence that comes from watching a database migration succeed in staging. The data volumes are smaller. The read replicas aren't under live traffic. The connection pool isn't being competed for by seventeen other services simultaneously. Nobody is retrying failed writes on top of your migration window.

Staging tells you the migration logic is probably correct. It tells you almost nothing about whether the migration will survive contact with production.

The senior SREs I've worked with have a name for this gap: the staging lie. Not because anyone is being dishonest. Because the environment itself lies. It is structurally incapable of simulating the one thing that actually breaks things — real, concurrent, bursty, inconsistent human behavior at scale.

What changes when you've internalized this?

You start writing migrations differently. You design them to be pausable. You add row-count checkpoints. You instrument the migration itself, not just the application on top of it. You test rollback, not as an afterthought, but as part of the success criteria for the migration plan.

You stop treating staging success as a green light. You treat it as a starting point.

What IAM Failures Actually Teach You

The IAM lockout I mentioned above wasn't a permissions misconfiguration in the obvious sense. The policy was correct for the steady state. The problem was that nobody had modeled what the assumed-role chain looked like when the primary region was degraded and traffic was failing over.

The service account had permission to assume a role in the primary region. The role in the primary region had permission to access the resource. In failover, the service account was suddenly trying to assume a role in a region it had never touched, in an account where the trust policy hadn't been updated to reflect the failover path.

Clean in the PR review. Broken in a real incident.

The lesson isn't "audit your IAM policies more carefully." The lesson is: test your failure paths with the same rigor you test your happy paths. The failure path IS a feature. It has its own requirements. It needs its own coverage.

This is the mental shift that separates SRE work from application development in the way it actually matters. A developer's job is to make the system do the right thing. An SRE's job is to make the system do the least wrong thing when conditions are not right — and to know in advance exactly how wrong things can get.

Error Budgets Are Not a Reporting Tool

I want to push back on something I see regularly in teams that have adopted SRE vocabulary without adopting SRE discipline.

Error budgets get treated as a reporting metric. Something that lives in a dashboard, gets reviewed in a monthly reliability review, and occasionally generates a Slack message when it's burning down too fast.

That is not what an error budget is for.

An error budget is a decision-making tool. It answers a specific question in real time: do we have headroom to deploy this change right now, or are we burning reliability faster than our users have agreed to tolerate?

When the budget is healthy, you move fast. You ship features. You run experiments. You take calculated risks because you have margin to absorb them.

When the budget is tight, you slow down. You freeze non-critical deploys. You prioritize reliability work. You explicitly negotiate with the product team about what it means to ship into a degraded error budget.

The budget makes that conversation objective. It takes it out of the territory of "the SRE team is being conservative again" and puts it into the territory of "we made a commitment to our users and here is where we stand against that commitment."

Teams that get this right stop having the "ops vs. product" tension. The budget is the arbiter. It's not personal.

The War Story That Changed How I Write Code

Here is mine.

A service I was responsible for was making synchronous calls to a downstream dependency as part of a critical user-facing flow. The downstream service had a p99 latency of around 200ms under normal conditions. Fine.

During a regional brownout — not a full outage, a brownout — that downstream service's p99 climbed to 8 seconds. My service had no circuit breaker. No timeout that was actually enforced at the HTTP client level (there was a timeout configured; it wasn't working because of how the client library handled connection reuse). No fallback behavior.

The result: every request to my service held a thread for up to 8 seconds. The thread pool exhausted. My service started timing out from the perspective of the services calling it. The incident scope grew from "downstream service brownout" to "three-layer cascade across the critical path."

The brownout was 11 minutes. The recovery from my service's cascading failure took 47 minutes after the downstream came back, because the connection pool was in a state that required a rolling restart to clear.

What did I change after that?

Every external call now has an enforced timeout at the network layer, not just the application layer. Every integration has a circuit breaker. Every integration has a tested fallback path — not a "we'll figure it out" note in the runbook, but a codified fallback that is exercised in pre-production.

And I stopped thinking about retries as "something we add if we need it." Retries with exponential backoff and jitter are now a default expectation for any service I build that calls anything external. They go in on day one.

What the Best Developers Figure Out

The developers who become genuine strategic partners for platform teams are the ones who start asking the right questions before they write a line of feature code.

Not "will this work?" but "how will this fail?"

Not "what's the happy path latency?" but "what's my blast radius if this dependency degrades?"

Not "does this pass CI?" but "do I have a rollback path that doesn't require a hotfix cycle?"

These questions are not SRE questions. They are engineering questions. SRE teams have just been asking them longer, under more pressure, with more immediate feedback when the answers are wrong.

The discipline is learnable. The war stories are the curriculum.

What's yours? Drop it in the comments — the specific incident that changed how you think about reliability. The more specific, the more useful it is for everyone reading.

Tags: #SRE #DevOps #PlatformEngineering #SoftwareEngineering #CloudArchitecture #Reliability #IncidentManagement #ProductionEngineering

SOC2 Is a Report, Not a Security Program published: true tags: security, devops, compliance, cloud

Ajay Devineni — Sat, 27 Jun 2026 01:02:50 +0000

description: SOC2 measures whether you have a process, not whether the process works. Here's what real security looks like and why the audit doesn't capture it.

SOC2 has done more to harm security than help it.

Not the concept. The theater around it.

I've watched companies pass SOC2 with MFA "enforced" through a policy document nobody enforces. Access reviews where managers approve 200 entitlements in three minutes. Encryption "in transit and at rest" that stops at the load balancer.

The auditor signs off. The Type II report goes in the sales deck. Everyone moves on.

Meanwhile the shared admin credential is still in a Slack DM from 2021.

The actual problem with the framework

SOC2 measures whether you have a process, not whether the process works.

You can document a terrible control consistently and pass. You can run an excellent informal practice and fail.

The framework has no opinion on outcomes — only on documentation. That distinction matters enormously when you're trying to decide how much weight to give a vendor's compliance report.

What real security looks like

Real security looks like:

Blast radius limits on IAM — not policies that exist, but policies that are scoped, reviewed, and enforced at the boundary
Short-lived credentials everywhere — assume-role with time bounds, not long-lived keys sitting in CI secrets
Peer-reviewed infrastructure changes — the same code review culture you apply to application code
Alerting on identity anomalies — not just "did login succeed" but "is this login pattern normal for this principal at this time"
On-call engineers who can actually contain an incident at 3am — not runbooks that assume the reader has six hours and a working Slack

None of that is uniquely a SOC2 control. Most of it isn't measured by the audit at all.

The diagnostic question

If your security program would collapse the day after the auditor leaves, you don't have a security program. You have a report.

Compliance is the floor. We keep treating it like the ceiling.

Four questions worth pressure-testing

When did someone last actually test the incident response runbook end-to-end — live, with a timer?
How long does it take to rotate every secret in production if one leaks today?
How many engineers have production IAM permissions they haven't used in 90 days?
Can you enumerate every service account and what it can do?

If any of those answers are "unclear" — the SOC2 Type II report won't change that. The report just means you documented the gap consistently.

What controls look good on paper but fail in practice in your environment? Genuinely curious what patterns others are seeing.

If You're a Great Developer, You're Probably a Terrible SRE Here's How to Encode SRE Paranoia Into Your Pipeline So Engineers Don't Have to Think Like One

Ajay Devineni — Wed, 24 Jun 2026 00:42:34 +0000

The LinkedIn post I published three days ago triggered something I didn't expect.
Not disagreement — validation. Senior engineers from payments companies, healthcare platforms, and financial infrastructure all said the same thing in different words: they've watched brilliant developers ship changes that worked perfectly in staging and detonated in production, not because the engineers were careless, but because their entire training rewards forward motion while reliability work rewards the opposite.
This post is the practical follow-up. The opinion was: developers and SREs think differently, and you can't fix that by handing developers a Terraform module and calling it ownership. The practice is: here's how you encode SRE paranoia into automated gates so the system enforces what culture can't.
The Core Problem With "Shift Left"
A shift-left mindset means SREs can embed reliability principles from Dev to Ops, baking reliability and resiliency into each process, app, and code change. That's the theory. The practice is that "baking reliability in" almost always means asking developers to think like SREs — to internalize failure mode thinking as a natural habit. Dynatrace
That doesn't work at scale. It works for the senior engineer who has been paged at 3 AM enough times. It doesn't work for the engineer who has never been on-call for a service they built.
In a traditional DevOps environment, the developer who wrote the code is often the one paged, focusing on a hotfix to restore the pipeline. In an SRE-driven environment, the SRE team manages the incident using a pre-defined playbook, focusing on automated remediation to bring the system back within its SLO parameters while the developers continue their sprint. Full-Stack Techies
The SRE-driven model works because the SRE's paranoia is encoded into the playbook, the SLO, and the error budget — not because the developer has learned to think differently. The developer doesn't need to internalize the mindset. The system has already internalized it for them.
That's the design principle. Encode the paranoia. Don't outsource it to culture.
What SRE Paranoia Actually Looks Like in Code
An SRE reviewing a production change asks five questions that a developer typically doesn't:
What's the rollback path, and have we tested it recently? Not "does a rollback exist" — does the team have a recent drill showing it takes under 5 minutes?
What happens during deployment, not just after? Connections in flight. Transactions mid-way through. Cache state that doesn't match the new schema.
What does this change look like at 10x load with one dependency degraded? Not at nominal load with everything healthy.
What's the blast radius if this goes wrong at 2 AM with one engineer on-call? Can one person contain it, or does it need three?
What's the SLO burn rate impact of a 1% error rate for 10 minutes? Does the team have that calculation, or will they be running it during the incident?
None of these questions require the developer to become an SRE. They require the CI/CD pipeline to refuse to proceed until these questions are answered — in code, not in conversation.
Introducing SRE Paranoia Gates
An SRE Paranoia Gate is an automated production readiness check that encodes one specific failure-mode question as a machine-enforceable constraint. The gate runs in CI/CD. It produces a pass/fail signal. It has a named owner — an SRE who wrote it and is accountable for its accuracy.
Each gate maps to one of the five questions above:
python# agentsre/sre_paranoia_gates.py

from dataclasses import dataclass, field
from typing import List, Optional, Callable
from datetime import datetime, timezone, timedelta
from enum import Enum
import json

class GateResult(Enum):
PASS = "pass"
FAIL = "fail"
WARN = "warn" # Won't block but logs for SRE review

@dataclass
class ParanoiaGateCheck:
"""
Result of one SRE Paranoia Gate evaluation.

Every gate check is logged — pass or fail.
Passes build confidence. Failures block deployment.
Warns surface to SRE review without blocking.
"""
gate_id: str
gate_name: str
result: GateResult
reason: str
sre_owner: str
evidence: Optional[dict] = None
checked_at: str = field(
    default_factory=lambda: datetime.now(timezone.utc).isoformat()
)

def to_dict(self) -> dict:
    return {
        "gate_id": self.gate_id,
        "gate_name": self.gate_name,
        "result": self.result.value,
        "reason": self.reason,
        "sre_owner": self.sre_owner,
        "evidence": self.evidence,
        "checked_at": self.checked_at,
    }

class SREParanoiaGateRunner:
"""
Run all registered SRE Paranoia Gates before a production deployment.

Gates encode the five questions a seasoned SRE asks before any change:
    Gate 1: Is the rollback path tested and under 5 minutes?
    Gate 2: Is in-flight traffic handled during deployment?
    Gate 3: Has this been tested at degraded-dependency load?
    Gate 4: Is the blast radius survivable by one on-call engineer?
    Gate 5: Is the SLO error budget healthy enough to absorb a 1% error rate?

All gates must pass for deployment to proceed.
WARN gates surface to SRE review but don't block.
One FAIL blocks the entire deployment.
"""

def __init__(self, service_name: str, sre_owner: str):
    self.service_name = service_name
    self.sre_owner = sre_owner
    self._gates: List[Callable] = []

def register(self, gate_fn: Callable) -> None:
    """Register a gate function. Each gate returns ParanoiaGateCheck."""
    self._gates.append(gate_fn)

def run_all(self, context: dict) -> dict:
    """
    Run all registered gates against deployment context.

    Args:
        context: Deployment metadata including service config,
                 rollback info, blast radius, SLO state, load test results.

    Returns:
        Summary with all gate results and deployment decision.
    """
    results = []
    blocked = False

    for gate_fn in self._gates:
        try:
            check = gate_fn(context, self.sre_owner)
            results.append(check)
            if check.result == GateResult.FAIL:
                blocked = True
        except Exception as e:
            # Gate evaluation failure = FAIL, not skip
            results.append(ParanoiaGateCheck(
                gate_id="gate_error",
                gate_name=gate_fn.__name__,
                result=GateResult.FAIL,
                reason=f"Gate evaluation raised exception: {str(e)}",
                sre_owner=self.sre_owner,
                evidence={"exception": str(e)}
            ))
            blocked = True

    return {
        "service": self.service_name,
        "deployment_approved": not blocked,
        "gates_run": len(results),
        "gates_passed": sum(1 for r in results if r.result == GateResult.PASS),
        "gates_failed": sum(1 for r in results if r.result == GateResult.FAIL),
        "gates_warned": sum(1 for r in results if r.result == GateResult.WARN),
        "results": [r.to_dict() for r in results],
        "evaluated_at": datetime.now(timezone.utc).isoformat(),
        "decision": (
            "APPROVED — all SRE paranoia gates passed."
            if not blocked
            else "BLOCKED — one or more SRE paranoia gates failed. "
                 "Fix the flagged conditions before proceeding."
        )
    }

The Five Gates — Implemented
python# Gate 1: Rollback tested and under 5 minutes
def gate_rollback_readiness(context: dict, owner: str) -> ParanoiaGateCheck:
"""
Developers assume rollback exists. SREs verify it works in under 5 minutes.
The question isn't 'do we have a rollback' — it's 'did we test it recently?'
"""
rollback = context.get("rollback", {})
last_drill_date = rollback.get("last_tested_at")
p95_minutes = rollback.get("p95_duration_minutes")

if not last_drill_date:
    return ParanoiaGateCheck(
        gate_id="SPG-001",
        gate_name="Rollback Readiness",
        result=GateResult.FAIL,
        reason="No rollback drill date recorded. Rollbacks untested are rollbacks that fail at 3 AM.",
        sre_owner=owner
    )

days_since = (datetime.now(timezone.utc) 
              - datetime.fromisoformat(last_drill_date)).days

if days_since > 30:
    return ParanoiaGateCheck(
        gate_id="SPG-001",
        gate_name="Rollback Readiness",
        result=GateResult.FAIL,
        reason=f"Rollback last tested {days_since} days ago. Require < 30 days.",
        sre_owner=owner,
        evidence={"last_tested_at": last_drill_date, "days_since": days_since}
    )

if p95_minutes and p95_minutes > 5:
    return ParanoiaGateCheck(
        gate_id="SPG-001",
        gate_name="Rollback Readiness",
        result=GateResult.FAIL,
        reason=f"Rollback p95 is {p95_minutes}m. SRE target is < 5 minutes.",
        sre_owner=owner,
        evidence={"p95_minutes": p95_minutes}
    )

return ParanoiaGateCheck(
    gate_id="SPG-001",
    gate_name="Rollback Readiness",
    result=GateResult.PASS,
    reason=f"Rollback tested {days_since} days ago, p95 {p95_minutes}m.",
    sre_owner=owner
)

Gate 2: In-flight traffic handling

def gate_in_flight_traffic(context: dict, owner: str) -> ParanoiaGateCheck:
"""
Developers test what happens after deployment. SREs test what happens during.
Connections in flight, transactions mid-way, cache state mismatches.
"""
deploy = context.get("deployment", {})
graceful_shutdown = deploy.get("graceful_shutdown_seconds", 0)
drain_configured = deploy.get("connection_draining_enabled", False)
schema_backward_compat = deploy.get("schema_backward_compatible", None)

failures = []

if graceful_shutdown < 30:
    failures.append(
        f"Graceful shutdown is {graceful_shutdown}s — "
        "recommend minimum 30s for in-flight requests to complete."
    )

if not drain_configured:
    failures.append(
        "Connection draining not enabled — "
        "load balancer will drop in-flight requests on pod termination."
    )

if schema_backward_compat is False:
    failures.append(
        "Schema change is NOT backward compatible — "
        "both old and new code will be running simultaneously during rollout. "
        "Migration plan required."
    )

if failures:
    return ParanoiaGateCheck(
        gate_id="SPG-002",
        gate_name="In-Flight Traffic Safety",
        result=GateResult.FAIL,
        reason=" | ".join(failures),
        sre_owner=owner,
        evidence={"deployment_config": deploy}
    )

return ParanoiaGateCheck(
    gate_id="SPG-002",
    gate_name="In-Flight Traffic Safety",
    result=GateResult.PASS,
    reason="Graceful shutdown, connection draining, schema compatibility verified.",
    sre_owner=owner
)

Gate 3: Degraded-dependency load testing

def gate_degraded_load_test(context: dict, owner: str) -> ParanoiaGateCheck:
"""
Developers test at nominal load with all dependencies healthy.
SREs test at peak load with one critical dependency degraded.
The real question: what happens when the upstream rate-limits you
during a retry storm you caused?
"""
load_test = context.get("load_testing", {})
tested_at_peak = load_test.get("tested_at_peak_load", False)
tested_with_degraded_dep = load_test.get("tested_with_degraded_dependency", False)
retry_storm_tested = load_test.get("retry_storm_simulation", False)

if not tested_at_peak:
    return ParanoiaGateCheck(
        gate_id="SPG-003",
        gate_name="Degraded Load Testing",
        result=GateResult.FAIL,
        reason="No peak load test recorded. Staging at 10% traffic ≠ production at 100%.",
        sre_owner=owner
    )

if not tested_with_degraded_dep:
    return ParanoiaGateCheck(
        gate_id="SPG-003",
        gate_name="Degraded Load Testing",
        result=GateResult.WARN,
        reason=(
            "Peak load tested but not with a degraded dependency. "
            "Add fault injection to your load test suite. "
            "This is a warning — but it's where most production incidents live."
        ),
        sre_owner=owner
    )

return ParanoiaGateCheck(
    gate_id="SPG-003",
    gate_name="Degraded Load Testing",
    result=GateResult.PASS,
    reason="Peak load and degraded-dependency scenarios both tested.",
    sre_owner=owner,
    evidence={"load_test_config": load_test}
)

Gate 4: Blast radius survivability

def gate_blast_radius_survivability(context: dict, owner: str) -> ParanoiaGateCheck:
"""
The SRE question: if this goes wrong at 2 AM with one engineer on-call,
can they contain it alone? Or does it need three engineers and a war room?
"""
blast = context.get("blast_radius", {})
downstream_count = blast.get("downstream_service_count", 0)
contains_payment_path = blast.get("contains_payment_path", False)
single_engineer_containable = blast.get("single_engineer_containable", None)
has_circuit_breaker = blast.get("circuit_breaker_configured", False)

if not has_circuit_breaker and downstream_count > 3:
    return ParanoiaGateCheck(
        gate_id="SPG-004",
        gate_name="Blast Radius Survivability",
        result=GateResult.FAIL,
        reason=(
            f"Service has {downstream_count} downstream dependencies "
            "and no circuit breaker. "
            "Failure will cascade. One engineer cannot contain this at 2 AM."
        ),
        sre_owner=owner,
        evidence={"downstream_count": downstream_count}
    )

if contains_payment_path and single_engineer_containable is False:
    return ParanoiaGateCheck(
        gate_id="SPG-004",
        gate_name="Blast Radius Survivability",
        result=GateResult.FAIL,
        reason=(
            "Change touches payment path and requires multiple engineers to contain. "
            "Require change window with full team available."
        ),
        sre_owner=owner
    )

return ParanoiaGateCheck(
    gate_id="SPG-004",
    gate_name="Blast Radius Survivability",
    result=GateResult.PASS,
    reason="Blast radius survivable by on-call rotation. Circuit breakers configured.",
    sre_owner=owner
)

Gate 5: SLO error budget check

def gate_error_budget(context: dict, owner: str) -> ParanoiaGateCheck:
"""
The question developers don't ask: how much error budget remains?
If budget is < 20%, shipping anything non-trivial is a reliability bet
the team hasn't explicitly made.
"""
slo = context.get("slo", {})
budget_remaining_pct = slo.get("error_budget_remaining_pct", 100.0)
is_critical_change = context.get("is_critical_change", False)

if budget_remaining_pct < 5.0:
    return ParanoiaGateCheck(
        gate_id="SPG-005",
        gate_name="SLO Error Budget",
        result=GateResult.FAIL,
        reason=(
            f"Error budget at {budget_remaining_pct:.1f}% — critically low. "
            "No changes permitted until budget recovers. "
            "This is not optional. The SLO contract with your users says so."
        ),
        sre_owner=owner,
        evidence={"budget_remaining_pct": budget_remaining_pct}
    )

if budget_remaining_pct < 20.0 and is_critical_change:
    return ParanoiaGateCheck(
        gate_id="SPG-005",
        gate_name="SLO Error Budget",
        result=GateResult.FAIL,
        reason=(
            f"Error budget at {budget_remaining_pct:.1f}% and change is marked critical. "
            "Require SRE lead approval before proceeding."
        ),
        sre_owner=owner
    )

if budget_remaining_pct < 20.0:
    return ParanoiaGateCheck(
        gate_id="SPG-005",
        gate_name="SLO Error Budget",
        result=GateResult.WARN,
        reason=(
            f"Error budget at {budget_remaining_pct:.1f}%. "
            "Below 20% — proceed with caution. "
            "SRE team should be aware this change is consuming headroom."
        ),
        sre_owner=owner
    )

return ParanoiaGateCheck(
    gate_id="SPG-005",
    gate_name="SLO Error Budget",
    result=GateResult.PASS,
    reason=f"Error budget at {budget_remaining_pct:.1f}% — sufficient headroom.",
    sre_owner=owner
)

Running the Full Gate Suite
python# Usage in your CI/CD pipeline

runner = SREParanoiaGateRunner(
service_name="payments-service",
sre_owner="platform-sre-team"
)

Register all five gates

runner.register(gate_rollback_readiness)
runner.register(gate_in_flight_traffic)
runner.register(gate_degraded_load_test)
runner.register(gate_blast_radius_survivability)
runner.register(gate_error_budget)

Build context from deployment metadata

context = {
"rollback": {
"last_tested_at": "2026-06-01T09:00:00Z",
"p95_duration_minutes": 3.5
},
"deployment": {
"graceful_shutdown_seconds": 60,
"connection_draining_enabled": True,
"schema_backward_compatible": True
},
"load_testing": {
"tested_at_peak_load": True,
"tested_with_degraded_dependency": False, # WARN
"retry_storm_simulation": False
},
"blast_radius": {
"downstream_service_count": 4,
"contains_payment_path": True,
"single_engineer_containable": True,
"circuit_breaker_configured": True
},
"slo": {
"error_budget_remaining_pct": 42.0
},
"is_critical_change": False
}

result = runner.run_all(context)
print(json.dumps(result, indent=2))

In CI/CD: fail the pipeline if not approved

if not result["deployment_approved"]:
raise SystemExit("SRE Paranoia Gates blocked deployment. See gate results above.")
Why This Works Better Than Culture
The teams winning on reliability in 2026 are not the ones with the most sophisticated AI stack. They are the ones that paired intelligent tooling with genuine engineering culture and did the hard work of changing how ownership flows, not just how alerts fire. Sherlocks AI
SRE Paranoia Gates are how ownership flows change in practice. The SRE doesn't have to be in every deployment review meeting. The SRE's questions are already in the pipeline. A developer shipping a change either answers them — by filling in the deployment context — or the gate blocks the deployment and the developer learns why those questions matter.
That's shift left done correctly. Not "teach developers to think like SREs." Encode what SREs think into the system itself.
The code is in agentsre/sre_paranoia_gates.py. MIT licensed.
Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer

github.com/Ajay150313/agentsre | dev.to/ajaydevineni

Your 99.99% Uptime SLO Is Probably a Lie — Here's How to Fix It

Ajay Devineni — Fri, 19 Jun 2026 03:09:38 +0000

I've been in enough postmortems to know the pattern.

The executive slide says 99.99% uptime. The on-call engineer who
lived through the last three months knows it's closer to 99.7%.
Neither number is wrong exactly. They're measuring completely
different things — and that gap is where trust goes to die.

This is not a theoretical problem. Let me show you the math,
the code that catches it, and the cultural shift that fixes it.

The math nobody does in public

Four nines sounds impressive until you look at what it actually allows:

99.99% uptime over 365 days =
Total minutes in a year: 525,600
Allowed downtime (0.01%): 52.6 minutes per year
Per month: 4.4 minutes
Per week: 1.0 minute

Now count what your team actually experienced last year:

The "elevated error rates" incident that lasted 40 minutes
but got logged as "partial degradation, not full outage"
The auth provider hiccup that dropped regional logins for 22 minutes
but was excluded because "that's a third-party dependency"
The Friday night patch that broke a non-critical service
that everyone actually depends on — 90 minutes, excluded as
"planned maintenance window"

That's 152 minutes. You've already spent 2.9× your entire annual budget
by March, and your dashboard still shows four nines.

This is not fraud exactly. It's a measurement convention that happens
to always favor the measurer.

The three ways SLOs get gamed (usually unintentionally)

Excluding third-party dependencies

"The CDN was down, not us" is technically true and practically useless
to your customer who couldn't load your application.

User-experienced availability includes everything in the request path.
If your SLO excludes your auth provider, your DNS, and your CDN,
you are measuring something your users have never experienced.

Averaging across regions

"We maintained 99.99% globally" while US-East was down for 35 minutes
is one of the most common forms of SLO theater in distributed systems.

A customer in us-east-1 experienced a complete outage.
Your global aggregate made it disappear.

The planned maintenance carve-out

This one is the most honest-sounding and the most problematic.

If you need a maintenance window to deploy safely,
your deployment process is part of your reliability problem.
Excluding it from your SLO means you never fix it.
You just keep scheduling windows.

What honest SLO tracking looks like in code

Here's the implementation I use. It measures what users experience,
not what the infrastructure team finds convenient to measure.

python# honest_slo.py

MIT License — Ajay Devineni (github.com/Ajay150313)

from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Optional
import json

class ExclusionPolicy(Enum):
"""
Controls whether incidents can be excluded from SLO calculation.

STRICT:      Nothing excluded. User experience only.
STANDARD:    Excludes pre-announced maintenance with customer notification.
PERMISSIVE:  Excludes third-party, maintenance, and regional partial impact.
             (This is how most teams get to four nines on paper.)
"""
STRICT = "strict"
STANDARD = "standard"
PERMISSIVE = "permissive"

@dataclass
class Incident:
"""
A single reliability incident, recorded at the moment it is detected.

The 'excluded_reason' field is the audit trail.
Every exclusion must be justified in writing at the time it happens,
not retroactively cleaned up before the monthly review.
"""
incident_id: str
started_at: datetime
resolved_at: Optional[datetime]
severity: str                    # P1, P2, P3
impact_regions: list[str]        # actual affected regions, not "global"
impact_scope: str                # "full_outage" | "partial_degradation" | "elevated_errors"
root_cause_category: str         # "internal" | "third_party" | "infrastructure"
was_planned: bool = False
customer_notified_before: bool = False  # for planned maintenance
excluded_reason: Optional[str] = None  # must be filled at time of exclusion
excluded_by: Optional[str] = None      # who approved the exclusion

@property
def duration_minutes(self) -> float:
    if self.resolved_at is None:
        return (datetime.now(timezone.utc) - self.started_at).total_seconds() / 60
    return (self.resolved_at - self.started_at).total_seconds() / 60

@property
def is_open(self) -> bool:
    return self.resolved_at is None

@dataclass
class SLOCalculator:
"""
Calculates SLO compliance from a list of incidents.

The key design decision: the exclusion policy is set at calculator
construction and applied consistently. You cannot change the policy
after the fact to make your numbers look better.

Usage:
    calc = SLOCalculator(
        target_pct=99.9,
        window_days=30,
        policy=ExclusionPolicy.STANDARD,
    )
    result = calc.calculate(incidents)
"""
target_pct: float           # e.g. 99.9
window_days: int            # rolling window
policy: ExclusionPolicy = ExclusionPolicy.STANDARD
service_regions: list[str] = field(default_factory=lambda: ["global"])

@property
def window_minutes(self) -> float:
    return self.window_days * 24 * 60

@property
def error_budget_minutes(self) -> float:
    return self.window_minutes * (1 - self.target_pct / 100)

def _should_exclude(self, incident: Incident) -> tuple[bool, str]:
    """
    Returns (should_exclude, reason).

    The reason is logged regardless — so you can audit what
    would have been excluded under a stricter policy.
    """
    if incident.excluded_reason:
        # Explicit manual exclusion — always honored if policy allows it
        if self.policy == ExclusionPolicy.STRICT:
            return False, "strict policy: no exclusions allowed"
        return True, incident.excluded_reason

    if self.policy == ExclusionPolicy.STRICT:
        return False, ""

    # STANDARD: only exclude pre-announced maintenance
    # with documented customer notification
    if (self.policy == ExclusionPolicy.STANDARD
            and incident.was_planned
            and incident.customer_notified_before):
        return True, "pre-announced maintenance with customer notification"

    if self.policy == ExclusionPolicy.PERMISSIVE:
        if incident.root_cause_category == "third_party":
            return True, "third-party dependency (permissive policy)"
        if incident.was_planned:
            return True, "planned maintenance (permissive policy)"
        # Partial regional impact excluded under permissive
        if (incident.impact_scope == "partial_degradation"
                and len(incident.impact_regions) < len(self.service_regions)):
            return True, "partial regional impact (permissive policy)"

    return False, ""

def calculate(self, incidents: list[Incident]) -> dict:
    """
    Returns a complete SLO report including:
    - Compliance under the configured policy
    - What it would be under STRICT policy (no exclusions)
    - The gap between them (the "honesty gap")
    - Full audit trail of all exclusions
    """
    cutoff = datetime.now(timezone.utc) - timedelta(days=self.window_days)
    window_incidents = [
        i for i in incidents
        if i.started_at >= cutoff
    ]

    included_minutes = 0.0
    excluded_minutes = 0.0
    exclusion_log = []
    strict_downtime = 0.0

    for incident in window_incidents:
        duration = incident.duration_minutes
        strict_downtime += duration  # always count for strict calculation

        should_exclude, reason = self._should_exclude(incident)
        if should_exclude:
            excluded_minutes += duration
            exclusion_log.append({
                "incident_id": incident.incident_id,
                "duration_minutes": round(duration, 1),
                "reason": reason,
                "excluded_by": incident.excluded_by,
            })
        else:
            included_minutes += duration

    # Compliance under configured policy
    downtime_ratio = included_minutes / self.window_minutes
    achieved_pct = (1 - downtime_ratio) * 100
    budget_consumed_pct = (included_minutes / self.error_budget_minutes) * 100

    # What it would look like with zero exclusions (honest number)
    strict_ratio = strict_downtime / self.window_minutes
    strict_pct = (1 - strict_ratio) * 100
    honesty_gap = achieved_pct - strict_pct

    # Budget status
    budget_remaining = max(0.0, self.error_budget_minutes - included_minutes)
    is_breached = included_minutes > self.error_budget_minutes

    return {
        "slo_target_pct": self.target_pct,
        "window_days": self.window_days,
        "policy": self.policy.value,
        "achieved_pct": round(achieved_pct, 4),
        "strict_achieved_pct": round(strict_pct, 4),
        "honesty_gap_pct": round(honesty_gap, 4),
        "error_budget_minutes_total": round(self.error_budget_minutes, 1),
        "error_budget_minutes_consumed": round(included_minutes, 1),
        "error_budget_minutes_remaining": round(budget_remaining, 1),
        "error_budget_consumed_pct": round(budget_consumed_pct, 1),
        "is_breached": is_breached,
        "total_incidents": len(window_incidents),
        "excluded_incidents": len(exclusion_log),
        "excluded_minutes": round(excluded_minutes, 1),
        "exclusion_log": exclusion_log,
    }

@dataclass

class ErrorBudgetBurnTracker:
"""
Tracks error budget consumption rate over time.

The key insight: it's not just how much budget you've used,
it's how fast you're burning it. A team that burns 80% of their
budget in the first week of the month has a very different problem
than a team that burns it steadily.

Burn rate > 1.0 means you will exhaust the budget before
the window closes at the current rate.
"""
slo_target_pct: float
window_days: int

@property
def _budget_minutes(self) -> float:
    return self.window_days * 24 * 60 * (1 - self.slo_target_pct / 100)

def current_burn_rate(
    self,
    consumed_minutes: float,
    elapsed_days: float,
) -> float:
    """
    Burn rate of 1.0 = consuming budget at exactly the sustainable pace.
    Burn rate of 3.0 = will exhaust budget in 1/3 the remaining time.
    Burn rate of 14.4 = exhausts 30-day budget in ~50 hours (page-worthy).
    """
    if elapsed_days <= 0:
        return 0.0
    elapsed_minutes = elapsed_days * 24 * 60
    actual_rate = consumed_minutes / elapsed_minutes
    sustainable_rate = self._budget_minutes / (self.window_days * 24 * 60)
    if sustainable_rate <= 0:
        return 0.0
    return actual_rate / sustainable_rate

def time_to_exhaustion_hours(
    self,
    remaining_budget_minutes: float,
    current_burn_rate: float,
) -> Optional[float]:
    """
    At the current burn rate, how many hours until the budget is gone?
    Returns None if burn rate <= 1.0 (on track or ahead of pace).
    """
    if current_burn_rate <= 1.0 or remaining_budget_minutes <= 0:
        return None
    sustainable_minutes_per_hour = (
        self._budget_minutes / (self.window_days * 24)
    )
    actual_minutes_per_hour = sustainable_minutes_per_hour * current_burn_rate
    if actual_minutes_per_hour <= 0:
        return None
    return remaining_budget_minutes / actual_minutes_per_hour

def format_report(report: dict) -> str:
"""Print a human-readable SLO compliance report."""
breach_str = "🔴 BREACHED" if report["is_breached"] else "🟢 Within budget"
gap_str = (
f"⚠️ +{report['honesty_gap_pct']:.3f}% vs strict calculation"
if report["honesty_gap_pct"] > 0.001
else "✅ No exclusions applied"
)

lines = [
    f"\n{'═'*56}",
    f"  SLO COMPLIANCE REPORT ({report['window_days']}-day window)",
    f"{'═'*56}",
    f"  Policy:              {report['policy'].upper()}",
    f"  Target:              {report['slo_target_pct']}%",
    f"  Achieved:            {report['achieved_pct']}%  {breach_str}",
    f"  Strict (no excl.):   {report['strict_achieved_pct']}%  {gap_str}",
    f"{'─'*56}",
    f"  Error budget total:  {report['error_budget_minutes_total']} min",
    f"  Consumed:            {report['error_budget_minutes_consumed']} min  "
    f"({report['error_budget_consumed_pct']}%)",
    f"  Remaining:           {report['error_budget_minutes_remaining']} min",
    f"{'─'*56}",
    f"  Incidents (window):  {report['total_incidents']}",
    f"  Excluded:            {report['excluded_incidents']} "
    f"({report['excluded_minutes']} min)",
]

if report["exclusion_log"]:
    lines.append(f"\n  Exclusion audit trail:")
    for excl in report["exclusion_log"]:
        lines.append(
            f"    • {excl['incident_id']}: {excl['duration_minutes']} min — "
            f"{excl['reason']}"
        )
lines.append(f"{'═'*56}\n")
return "\n".join(lines)

── Example usage ─────────────────────────────────────────────────────────────

if name == "main":
from datetime import timezone

now = datetime.now(timezone.utc)

incidents = [
    Incident(
        incident_id="INC-2026-001",
        started_at=now - timedelta(days=25, hours=2),
        resolved_at=now - timedelta(days=25, hours=1, minutes=20),
        severity="P1",
        impact_regions=["us-east-1"],
        impact_scope="full_outage",
        root_cause_category="internal",
    ),
    Incident(
        incident_id="INC-2026-002",
        started_at=now - timedelta(days=18),
        resolved_at=now - timedelta(days=17, hours=23, minutes=37),
        severity="P2",
        impact_regions=["us-east-1", "eu-west-1"],
        impact_scope="elevated_errors",
        root_cause_category="third_party",
        excluded_reason="Auth provider outage — not our infrastructure",
        excluded_by="oncall-lead@company.com",
    ),
    Incident(
        incident_id="INC-2026-003",
        started_at=now - timedelta(days=5, hours=22),
        resolved_at=now - timedelta(days=5, hours=20, minutes=30),
        severity="P2",
        impact_regions=["us-east-1"],
        impact_scope="partial_degradation",
        root_cause_category="internal",
        was_planned=True,
        customer_notified_before=True,
    ),
]

print("\n--- PERMISSIVE (how most teams report) ---")
calc_permissive = SLOCalculator(
    target_pct=99.9,
    window_days=30,
    policy=ExclusionPolicy.PERMISSIVE,
    service_regions=["us-east-1", "eu-west-1"],
)
print(format_report(calc_permissive.calculate(incidents)))

print("--- STANDARD (honest, defensible) ---")
calc_standard = SLOCalculator(
    target_pct=99.9,
    window_days=30,
    policy=ExclusionPolicy.STANDARD,
    service_regions=["us-east-1", "eu-west-1"],
)
print(format_report(calc_standard.calculate(incidents)))

print("--- STRICT (pure user experience) ---")
calc_strict = SLOCalculator(
    target_pct=99.9,
    window_days=30,
    policy=ExclusionPolicy.STRICT,
    service_regions=["us-east-1", "eu-west-1"],
)
print(format_report(calc_strict.calculate(incidents)))

# Burn rate check
tracker = ErrorBudgetBurnTracker(slo_target_pct=99.9, window_days=30)
burn = tracker.current_burn_rate(consumed_minutes=85.0, elapsed_days=15)
hours_left = tracker.time_to_exhaustion_hours(
    remaining_budget_minutes=max(0, 43.2 - 85.0), current_burn_rate=burn
)
print(f"  Current burn rate: {burn:.2f}×")
if hours_left:
    print(f"  Budget exhaustion: {hours_left:.1f} hours at current rate")
else:
    print(f"  Budget already exhausted (consumed 85.0 of 43.2 min budget)"

AWS DevOps Agent Is GA And the Hardest Problem Isn't the Agent. It's What Happens to Your Team Six Months Later.

Ajay Devineni — Tue, 16 Jun 2026 03:20:51 +0000

AWS DevOps Agent went generally available this week. It's a frontier agent — autonomous, massively scalable, designed to work for hours or days without constant intervention. It analyzes data across monitoring tools, reviews recent deployments, and coordinates incident response. incident.io
I've been building the SRE governance framework for exactly this class of agent for five months. Eighteen posts, an open-source library, an arXiv paper, a DynamoDB-backed ARO registry, a Pre-Action SRE Gate, an EvalPipeline that runs nightly DQR checks.
All of that is the technical governance layer. This post is about the human governance layer — the one that fails quietly and shows up in your postmortem eighteen months after you deployed the agent.
The Pattern
The agent failed confidently, without signaling uncertainty, and the humans around it had gradually stopped watching. We have decades of research on automation complacency in aviation and industrial control systems. The failure mode is well understood: when automation performs reliably for an extended period, human operators reduce active monitoring. When the automation eventually fails — and it always does — the operators have lost the situational awareness to catch the failure quickly. Yahoo Finance
This is not a hypothetical for AI SRE agents. It's the natural trajectory of any reliable automation deployed into an on-call rotation.
Week one: engineers review every agent decision. They read the reasoning. They validate the RCA. They check whether the remediation matched what they would have done.
Week four: engineers review decisions that look unusual. The routine ones get a glance and a thumbs up.
Week twelve: engineers check whether the incident resolved. If it did, they move on. The agent's reasoning is no longer being read.
The agent's HER looks healthy. DQR is above threshold. Incidents are resolving faster. Every metric in the governance stack is green.
What isn't being measured: whether any human has verified that the agent's reasoning was correct, or whether the team has simply started accepting the outcome as a proxy for correctness.
Introducing ACR — Automation Complacency Rate
ACR is the fraction of agent decisions accepted by the team without active human verification of the reasoning, measured on a rolling window.
Formally:
ACR(A, t) = decisions_accepted_without_verification(A, t)
/ total_agent_decisions(A, t)
Where "accepted without verification" means the human noted the outcome (incident resolved, action taken) but did not read the decision trace, validate the RCA, or challenge the agent's reasoning.
This distinction matters because outcome acceptance and reasoning verification are two completely different signals. An agent can produce the right outcome for the wrong reason, and if your team only checks outcomes, you will not detect the reasoning drift until a novel incident exposes it.
Why ACR Rising Is Not Good News
The instinctive interpretation of rising ACR is that the agent has earned trust. The team reviewed everything early on, found the agent reliable, and appropriately reduced their verification overhead.
That interpretation is sometimes correct. It is also the interpretation that precedes most automation complacency incidents in aviation and industrial control systems.
The correct interpretation of rising ACR depends on what's happening to your other SLIs in the same window.
If ACR rises while DQR is stable and RTD is low — the agent may genuinely have earned reduced oversight for that task class. Narrow the blast radius, document the trust extension, and set a review date.
If ACR rises while DQR shows any downward trend — the team has reduced oversight while agent quality is slipping. This is the danger zone. The two signals moving in opposite directions is your automation complacency signature.
If ACR rises after HER drops — the agent is escalating less AND the team is reviewing less. This is the double-exposure pattern. Maximum undetected risk surface.
Three ACR Signals Worth Tracking
ACR trend — weekly rolling average, 30-day window. A rising trend over any 4-week period warrants a review meeting, not an alert. This is a cultural signal, not an infrastructure signal.
ACR by severity — P1 and P2 incidents should have near-zero ACR. If your team is accepting agent decisions on P1s without verification, that is your risk exposure quantified. Set a hard target: P1 ACR < 5%.
ACR after HER drop — when HER drops meaningfully (agent escalating less often), check ACR in the same window. If both move in the same direction — agent acts more, team reviews less — that combination needs governance intervention before the next novel failure mode.
What Verification Actually Means
Verification is not approval. It doesn't mean the engineer had to intervene or override the agent. It means a human read the agent's reasoning trace, validated that the RCA was correct, and confirmed that the remediation matched what an experienced engineer would have chosen.
This takes three to five minutes for a well-structured RTD trace. It's not a burden if the trace is readable. It is a burden if the trace requires excavating through unstructured logs — which is why Layer 3 observability (the RTD module from Post 11 of this series) exists. You cannot verify reasoning you cannot read.
pythonfrom agentsre.automation_complacency import ACRTracker, VerificationRecord

tracker = ACRTracker(agent_id="devops-agent-v1", task_class="incident-investigation")

After each agent decision, log whether a human verified the reasoning

tracker.record(VerificationRecord(
decision_id="inc-2026-0615-001",
outcome_accepted=True, # team noted incident resolved
reasoning_verified=False, # nobody read the RTD trace
severity="P2",
verifier=None
))

status = tracker.acr_status()
if status["p2_acr_pct"] > 20.0:
# P2 incidents being accepted without reasoning review
# This needs a team conversation, not an automated alert
notify_sre_lead(status)
The AWS DevOps Agent Specific Context
AWS DevOps Agent works for hours or days without constant intervention. When production incidents occur, it analyzes data across multiple monitoring tools, reviews recent deployments, and coordinates response teams. incident.io
An agent that works for hours without intervention is an agent where ACR will naturally rise. The longer the autonomous window, the more decisions accumulate before any human checkpoint.
This is not a criticism of the agent. It's a governance design requirement. Long-running autonomous agents need structured human verification checkpoints built into the workflow — not just at the end when the incident resolves, but at decision points during the investigation.
The Pre-Action Gate from Post 13 is one checkpoint. The ACR tracker is the measurement layer that tells you whether those checkpoints are actually being used as verification moments or just passed through as administrative steps.
The Postmortem Question
Add this field to every postmortem that involves an autonomous agent action:
"For each significant agent decision during this incident: was the reasoning verified by a human before the next action was taken, or was it accepted based on outcome?"
If the answer is consistently "accepted based on outcome" — your team's ACR is above zero and you don't have a measurement layer for it. That's the gap this post addresses.
The agentsre/automation_complacency.py module is now in the GitHub repo. MIT licensed, zero external dependencies for core logic.
Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer

github.com/Ajay150313/agentsre | dev.to/ajaydevineni

Google Published Their AI SRE Blueprint. Here's the Line-by-Line Mapping to What the Community Has Been Building

Ajay Devineni — Tue, 09 Jun 2026 01:26:43 +0000

Google published a white paper on May 28 that every SRE should read.
It details how they're architecting a new foundation for reliability with three core components: AI Operator (autonomous mitigation agents), Actus (strict execution guardrails), and IRM Analyzer (continuous evaluation pipelines grounded in human operational memory). The goal: safely govern high-velocity agentic software development at Google's scale. Rootly
I've been building toward the same architecture from the ground up for couple of months not inside Google, but as an independent practitioner trying to solve the same problem for teams who don't have Google's infrastructure or runway.
Reading the whitepaper, I found that every component Google named maps directly to something already in the agentsre library or this series. This post maps them side by side.
Google's Actus → Pre-Action SRE Gate
Actus is Google's physical execution control plane for safe autonomous mitigation — it bounds what an agent can do in production with strict policy enforcement before any action executes. Rootly
That's exactly what the Pre-Action SRE Gate does. Three checks before any autonomous action: error budget remaining (does the system have headroom?), AQDD state (can humans course-correct if this goes wrong?), and HER trend (is this agent already outside its reliable envelope?). If any check fails — agent escalates, does not act.
Google built Actus at the infrastructure level for internal systems. The Pre-Action Gate is the same pattern implemented as a Lambda + CloudWatch + DynamoDB pattern any AWS team can deploy this week.
Google's IRM Analyzer → DQR + RTD
IRM Analyzer is Google's continuous evaluation pipeline that captures human operational memory and runs nightly evaluations to prove agent readiness before deployment and during operation. Rootly
Two metrics from this series do the same work:
DQR (Decision Quality Rate) — is the agent's output correct? Measured continuously, not just at deployment.
RTD (Reasoning Trace Depth) — is the agent's reasoning stable? Re-planning cycles per task. Rises before DQR falls.
Google runs nightly evals against a corpus of human-validated incidents. For teams without that corpus, DQR and RTD measured in 30-day shadow mode are the approximation that's achievable without Google's internal incident database.
Google's AI Operator → The agent that needs ARO
Google SRE has AI agents that continuously monitor and improve playbooks and production documentation based on their usage during incidents. AI agents can also generate new playbooks from incidents. Nova AI Ops
This is AI Operator in action. And it's exactly the class of agent that needs Agent Reliability Ownership (ARO) registration — a named owner, a defined blast radius, and an escalation path — before it starts writing to production documentation.
An agent that can modify runbooks is an agent that can corrupt the guidance every human SRE relies on during an incident. Blast radius definition isn't optional for that class of agent. It's the most important governance artifact you have.
The gap Google doesn't address — fleet governance
Google's whitepaper covers individual agent governance well. What it doesn't cover — because at Google's scale it's a different problem — is fleet-level governance for teams where engineers are deploying their own agent workflows alongside platform-deployed agents.
That's the Agent Sprawl problem from Post 6. The Sprawl Registry and Postmortem Readiness Rate (PRR) from Post 12 address the fleet-level governance gap that Google's architecture assumes away.
What this means for your team
AI SRE technology is arriving faster than the trust frameworks needed to deploy it safely. Sherlocks AI
Google just published the trust framework for their environment. The agentsre library is the open-source implementation of the same framework for everyone else.
The three components that matter most to implement first, in order:
Start with Pre-Action Gate (Actus equivalent) — because an ungated agent is a liability before it's an asset.
Add DQR + RTD monitoring (IRM Analyzer equivalent) — because you can't evaluate what you don't measure.
Register every agent in ARO + Sprawl Registry (AI Operator governance) — because you can't own what you haven't named.
The whitepaper is at sre.google. The library is at github.com/Ajay150313/agentsre.
What component is your team missing most right now?
Ajay Devineni | AWS Community Builder | IEEE Senior Member Senior SRE/Platform Engineer | github.com/Ajay150313/agentsre

How to Evaluate Any AI SRE Tool A Practitioner's Framework Built From 15 Posts of Production SLIs

Ajay Devineni — Thu, 04 Jun 2026 01:17:39 +0000

Title: How to Evaluate Any AI SRE Tool — A Practitioner's Framework Built From 15 Posts of Production SLIs
Your manager just forwarded you a Gartner report. Analyst recognition of the AI SRE category, sustained on-call pressure, immature trust and governance frameworks, and the need for orchestration rather than disconnected agent experiments all arrived together in 2026. The question landing in every SRE team's backlog right now is: should we buy something, build something, or wait? Sherlocks
I've spent four months building the measurement layer for AI agents from scratch — DQR, TIE, HER, AQDD, RTD, CUR, Pre-Action Gate, Semantic Gap detection. Fifteen posts, an open-source library, and a growing arXiv paper. This post is where that work becomes a vendor evaluation framework.
Every claim in this framework maps to a metric I've already defined. You can verify these against any tool — commercial or open-source.
The Problem With Vendor Benchmarks
Datadog's Bits AI SRE decreases time to resolution by up to 95%. New Relic's users resolved incidents 25% faster than those without AI features. Both numbers are published. Both are real — in the environments they measured. Nova AI OpsInfoQ
The question is whether those environments match yours. A 95% MTTR improvement measured on a system with clean telemetry, well-structured runbooks, and narrow incident categories is a different number than what you'll see in a system with fragmented observability, complex dependency graphs, and novel failure modes.
Vendor benchmarks measure the tool in optimal conditions. Your evaluation needs to measure the tool in your conditions. These five questions give you the framework.
Question 1: Does it instrument the reasoning layer?
The semantic gap — the space between what an agent intended and what it executed — is invisible to infrastructure APM. I wrote about this last week using Sherlocks.ai's research: existing tools observe high-level intent or low-level actions, not the correlation between them.
Ask any vendor: do you track re-planning cycles per task? Can I see how many times the agent changed its approach before completing or escalating? Can I query that history after an incident?
If the answer is "we log prompts and tool calls," that's Layer 1 observability. Useful, necessary, insufficient. You need Layer 3 — one structured record per agent task showing the full decision sequence.
What to look for in a demo: ask them to show you a failed task trace. Does it show you the sequence of re-planning decisions, or just the final outcome and the spans?
Question 2: What is the Human Escalation Rate in their benchmark?
HER — the fraction of agent decisions that escalated to human judgment — is the most honest single metric for how autonomous a tool actually is. A low MTTR number paired with a high HER means humans were doing most of the resolution work, faster because the agent assembled context for them. That's valuable. It's not the same as autonomous remediation.
Ask: in your benchmark environment, what percentage of incidents did the agent resolve without human action? What percentage required human approval before execution? What triggered escalation most often?
These questions reveal whether the tool is an autonomous remediator or a very good assistant. Both are legitimate. Only one of them matches the vendor's headline claim.
Question 3: Does it check SLO state before acting?
An agent that remediates without checking your current error budget can compound a degraded situation. I formalized this in the Pre-Action SRE Gate (Post 13): three checks before any autonomous action — error budget remaining, AQDD state, and the agent's own HER trend.
Ask any vendor: does your agent check SLO error budget before executing a remediation? What happens if the error budget is critically low — does it act anyway or escalate? Can I configure the pre-action gate thresholds?
A tool that doesn't have an answer to this question is not safe for production systems where the error budget is already burning.
Question 4: What is the defined blast radius per agent?
Komodor's Klaudia is trained specifically on pod crashes, failed rollouts, autoscaler friction, misconfigurations, and cascading failures in Kubernetes environments. That specificity is its blast radius. 95% accuracy in that domain does not mean 95% accuracy outside it. Yisusvii
Every AI SRE tool has an implicit blast radius — the set of systems and failure modes it was trained and tested on. Good tools make this explicit. Ask: what systems can this agent modify autonomously? What systems are write-locked? What failure categories is the accuracy claim based on?
If the vendor can't give you a concrete blast radius definition, the accuracy number is a marketing claim. If they can, you can evaluate whether that blast radius covers your actual failure distribution.
Question 5: What is the ownership model when it's wrong?
This is the question vendors like least. When the agent makes a bad remediation decision and compounds the incident, who is accountable? The vendor's SLA covers service availability, not the operational consequences of an agent action.
In your environment, the answer should map to your ARO (Agent Reliability Ownership) registration — a named human owner, a defined escalation path, and an audit log of every gate check the agent ran before acting.
Ask any vendor: does your tool generate an audit log of agent decision reasoning before each action? Is that log queryable during incident review? Who owns the agent's behavior in my environment?
If the audit log doesn't exist, you cannot write a complete postmortem after an agent-involved incident. That's the accountability gap that makes autonomous agents unsafe in regulated production environments.
The Build vs Buy Decision Matrix
Given these five questions, here's how I'd frame the build-vs-buy decision:
Buy if: Your failure distribution maps closely to the tool's blast radius, you don't need custom SLIs beyond what the vendor provides, and the vendor can answer all five questions with specifics.
Build if: Your failure distribution is broad or novel, you need custom SLIs (DQR, RTD, HER, AQDD are all absent from commercial tools today), or you need to satisfy regulatory requirements that mandate audit trails the vendor doesn't generate.
Hybrid (most realistic): Buy the investigation layer — vendor tools are genuinely good at assembling incident context faster than humans. Build the governance layer — Pre-Action Gates, ARO registration, Semantic Gap detection, Sprawl Registry. The agentsre library is designed for exactly this hybrid.
The Evaluation Scorecard
python# agentsre/tool_evaluation.py

from dataclasses import dataclass, field
from typing import Dict, List, Optional
import json
from datetime import datetime, timezone

@dataclass
class ToolEvaluationScore:
"""
Five-question evaluation scorecard for AI SRE tooling.

Use this to evaluate commercial tools or internal builds
against the SLI framework from the agentsre series.

Score each question 0 (no), 1 (partial), 2 (yes).
Total score >= 8: consider for production.
Total score 5-7: pilot with governance layer built separately.
Total score < 5: not production-ready for autonomous operation.
"""
tool_name: str
evaluator: str
environment_context: str  # Brief description of your stack

# Q1: Reasoning layer instrumentation
tracks_replanning_cycles: int = 0    # 0/1/2
can_query_decision_sequence: int = 0
q1_notes: str = ""

# Q2: HER transparency
her_in_benchmark_disclosed: int = 0
autonomous_vs_assisted_split_disclosed: int = 0
q2_notes: str = ""

# Q3: Pre-action SLO gate
checks_error_budget_before_acting: int = 0
gate_thresholds_configurable: int = 0
q3_notes: str = ""

# Q4: Blast radius definition
blast_radius_explicit: int = 0
accuracy_claim_scoped_to_blast_radius: int = 0
q4_notes: str = ""

# Q5: Ownership and audit
generates_decision_audit_log: int = 0
audit_log_queryable_postmortem: int = 0
q5_notes: str = ""

evaluated_at: str = field(
    default_factory=lambda: datetime.now(timezone.utc).isoformat()
)

@property
def total_score(self) -> int:
    return (
        self.tracks_replanning_cycles +
        self.can_query_decision_sequence +
        self.her_in_benchmark_disclosed +
        self.autonomous_vs_assisted_split_disclosed +
        self.checks_error_budget_before_acting +
        self.gate_thresholds_configurable +
        self.blast_radius_explicit +
        self.accuracy_claim_scoped_to_blast_radius +
        self.generates_decision_audit_log +
        self.audit_log_queryable_postmortem
    )

@property
def recommendation(self) -> str:
    if self.total_score >= 8:
        return "CONSIDER: meets production governance bar"
    elif self.total_score >= 5:
        return "PILOT: build governance layer separately before production"
    else:
        return "NOT READY: missing critical governance capabilities"

def to_report(self) -> Dict:
    return {
        "tool": self.tool_name,
        "evaluator": self.evaluator,
        "environment": self.environment_context,
        "scores": {
            "q1_reasoning_layer": {
                "tracks_replanning": self.tracks_replanning_cycles,
                "queryable_decision_sequence": self.can_query_decision_sequence,
                "notes": self.q1_notes
            },
            "q2_her_transparency": {
                "her_disclosed": self.her_in_benchmark_disclosed,
                "autonomous_split_disclosed": self.autonomous_vs_assisted_split_disclosed,
                "notes": self.q2_notes
            },
            "q3_pre_action_gate": {
                "checks_error_budget": self.checks_error_budget_before_acting,
                "configurable_thresholds": self.gate_thresholds_configurable,
                "notes": self.q3_notes
            },
            "q4_blast_radius": {
                "explicit_definition": self.blast_radius_explicit,
                "accuracy_scoped": self.accuracy_claim_scoped_to_blast_radius,
                "notes": self.q4_notes
            },
            "q5_audit_ownership": {
                "audit_log_generated": self.generates_decision_audit_log,
                "queryable_in_postmortem": self.audit_log_queryable_postmortem,
                "notes": self.q5_notes
            }
        },
        "total_score": f"{self.total_score}/20",
        "recommendation": self.recommendation,
        "evaluated_at": self.evaluated_at
    }

def to_json(self) -> str:
    return json.dumps(self.to_report(), indent=2)

Where This Fits in the Arc
Posts 1–14 built the measurement framework: SLIs for agent output quality, control plane reliability, reasoning observability, context management, ownership governance, semantic gap detection.
Post 15 is the practical payoff — you now have a five-question framework, grounded in production SLIs, to evaluate any AI SRE tool your manager asks you to assess. Whether the answer is buy, build, or hybrid, the framework gives you a defensible, technically grounded recommendation.
The ToolEvaluationScore dataclass is in agentsre/tool_evaluation.py. Use it to document your evaluation and generate a report you can share with your team.
Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer
github.com/Ajay150313/agentsre | dev.to/ajaydevineni