DEV Community: Ajay Devineni

The Arson We Applaud

Ajay Devineni — Mon, 10 Aug 2026 02:07:53 +0000

It's cool to be the engineer who fixes the outage at 3 AM. It's cooler to build the system that never paged anyone in the first place.

I've spent eleven years in banking and infrastructure, and if there's one pattern I keep running into, it's this: we've built an entire culture around celebrating the fix and almost none around celebrating the absence of the fire. The allnighter gets a shoutout in standup. The frantic Slack thread becomes a story people tell in interviews. The "great save" gets a line in the postmortem doc. Meanwhile, the quarter where nothing broke the actual goal of the job gets nothing. No one writes that post.

I don't think this is a people problem. I think it's a tooling and incentive problem, and it's one I've been trying to chip away at with a set of governance primitives I've been opensourcing under a project called agentsre.

Firefighting vs. arson

Here's the uncomfortable framing I keep coming back to: most of the incidents we praise ourselves for resolving heroically were, somewhere upstream, avoidable. Someone shipped a change without enough signal about current system health. Someone deployed into a service that was already sitting on a thin error budget. Someone (or increasingly, some agent) took an irreversible action against a wide blast radius without anyone in the loop asking "should we, right now, given everything else in flight?"

We reward the firefighting. We rarely examine the arson.

The engineers I trust most in this industry are the boring ones. Their systems selfheal. Their runbooks are short because the failure modes they'd need are rare. They automated the toil away instead of building an identity around absorbing it. That's not a flashy skill set. It doesn't generate a satisfying story arc. But it's the actual craft.

What "engineering the absence of heroics" looks like in practice

Talk is easy here, so I want to point at something concrete instead of just restating the philosophy. The newest piece in agentsre is a PreAction SRE Gate a small, composable guardrail that evaluates whether a statechanging action (a deploy, a config push, an autoscale event, a tool call made by an autonomous agent) should be allowed to proceed before it runs, based on the system's current ability to absorb that risk.

The shape of it is simple. You describe the action:

action = ActionRequest(
    name="rolloutpaymentsapiv42",
    blast_radius="region",
    reversible=True,
    initiated_by="agent:deploybot",
    metadata={"service": "paymentsapi"},
)

And a set of evaluators error budget health, blast radius, concurrent inflight changes, and how much track record the initiating agent has each contribute a bounded risk score with a plainEnglish reason attached. The gate combines those into ALLOW, HOLD, or BLOCK, weighted so a single severe risk factor can't get diluted by several calm ones.

The part I care about most is the agentorigin evaluator. As more actions in production systems start being initiated by autonomous agents rather than humans, the implicit risk judgment a human operator makes ("is now a good time to ship this?") doesn't happen automatically anymore it has to be engineered back in, explicitly, or it just disappears. An agent with no established track record gets treated more conservatively by default than a human operator would be. Trust gets earned from observed behavior over time, not assumed because the caller is automated.

Why this matters more than it used to

For most of my career, "reliability engineering" meant reacting well: good alerting, fast rollback, clean incident response. That's still necessary. But it's built around the assumption that a human is the one taking the risky action, and that some judgment is happening before the action fires. That assumption is eroding fast as agentic systems take on more autonomous execution in production environments. If we don't build the preaction judgment back into the systems themselves, we're going to get a lot more 3 AM pages, except now nobody will be able to tell you which agent, executing which plan, decided the timing was fine.

99.99% uptime was never about heroics. It's the absence of heroics, engineered on purpose, on the days nobody's watching and nothing gets written up.

The full implementation, along with the rest of the SLI and governance primitives I've been building for agentic infrastructure, is open source: github.com/Ajay150313/agentsre.

I'd like to hear from other people building or operating agentic systems what does your team's pre action risk story look like right now? Is there one, or is it still "the agent has good judgment, probably"?

I Built an LLM That Reads Compliance Regulations and Blocks Non-Compliant Code Deployments IEEE Published the Results

Ajay Devineni — Fri, 07 Aug 2026 22:48:17 +0000

My paper on LLM-based regulatory compliance automation for financial DevOps was published at IEEE ICCBI 2026. The core idea sounds almost too good: an LLM reads regulatory documents (Basel III, GDPR, PCI-DSS), extracts compliance rules, and blocks non-compliant code from deploying. Here's how we actually built it, and where the hard problems are.

Why Manual Compliance Is Broken

Financial institutions face regulatory requirements that are:

Written in legal prose, not machine executable rules
Updated frequently — GDPR guidance evolves, Basel capital requirements change, new national AI regulations appear
Jurisdictionally contradictory a system compliant with GDPR may violate a local data sovereignty law in a different market

The standard approach: a compliance team manually translates regulations into a static checklist, developers manually verify their changes against the checklist, and an audit team manually verifies everything before regulators arrive.

This doesn't work at DevOps velocity. Code ships 10–50 times per day. The compliance team can't review every commit.

The LLM Solution — And Its Problems

The framework uses an LLM to do what compliance officers do: read regulatory text and extract actionable rules. But naively prompting an LLM with "here is GDPR article 25, what does it require?" creates a dangerous new problem: hallucination in compliance contexts.

A compliance system that confidently generates incorrect requirements is worse than no compliance system.

Here's how the framework addresses this:

RAG-Backed Rule Generation

from agentsre.compliance import RegulatoryRuleExtractor

extractor = RegulatoryRuleExtractor(
    llm_backend='bedrock',          # Amazon Bedrock for financial services
    model_id='anthropic.claude-3-5-sonnet',
    rag_source_bucket='s3://compliance-docs/regulations/',
    citation_required=True,         # EVERY rule must cite source text
    confidence_threshold=0.85       # Low-confidence rules go to human review
)

# Extract rules from regulatory document
rules = extractor.extract(
    document='gdpr_article_25_data_protection_by_design.pdf',
    scope=['data_storage', 'encryption', 'access_control', 'retention']
)

# Returns structured rules WITH citations
# {
#   'rule_id': 'GDPR-25-1',
#   'requirement': 'Personal data must be encrypted at rest using AES-256 or equivalent',
#   'source_citation': 'GDPR Article 25(1): "...appropriate technical measures..."',
#   'confidence': 0.94,
#   'infrastructure_targets': ['S3', 'RDS', 'EBS', 'DynamoDB']
# }

Key design decision: the LLM never operates from memory. It always re-reads the source regulatory document through RAG before generating a rule. This eliminates the most dangerous hallucination vector — stale or fabricated regulatory requirements.

CI/CD Gate Integration

# .github/workflows/compliance-gate.yml (conceptual)
name: Compliance Gate

on: [pull_request]

jobs:
  compliance-check:
    runs-on: ubuntu-latest
    steps:
      - name: Evaluate Compliance Impact
        run: |
          agentsre-compliance evaluate \
            --diff ${{ github.event.pull_request.diff_url }} \
            --rule-registry $COMPLIANCE_REGISTRY_URL \
            --jurisdictions "US,EU,UK" \
            --failure-mode strict \
            --output compliance-report.json

      - name: Block on Non-Compliance
        if: steps.compliance-check.outputs.compliant == 'false'
        run: |
          echo "::error::Compliance violations detected. See compliance-report.json"
          cat compliance-report.json
          exit 1

The gate evaluates the code diff — not just the final state — because compliance violations are often introduced in specific changes and obscured in the final artifact.

Multi-Jurisdictional Conflict Resolution

from agentsre.compliance import JurisdictionResolver

resolver = JurisdictionResolver(
    jurisdictions=['GDPR', 'CCPA', 'PDPA_SG', 'LGPD_BR'],
    conflict_strategy='strictest_wins',
    human_escalation_threshold='direct_conflict'
)

# When GDPR requires deletion within 30 days
# and local law requires retention for 7 years:
# → Escalate to human policy decision, don't resolve autonomously
resolution = resolver.evaluate(rule_set)

strictest_wins for ambiguity, human_escalation for direct conflict. This is the right failure mode for regulated systems.

Hard Problems We Haven't Fully Solved

Regulatory update propagation: When a regulatory body issues guidance that amends an existing regulation without changing the source document, the RAG system won't catch it. We handle this with a human-curated "regulatory update feed" that triggers rule re-extraction when new guidance is published.

Legacy system integration: Some financial systems expose compliance-relevant configurations only through mainframe JCL or COBOL report outputs. Parsing these into a format the framework can evaluate required custom adapters — not generalizable.

False positive rate in blocking mode: Early versions of the framework blocked 30% of legitimate deployments with false positive compliance violations. After tuning confidence thresholds and adding a "remediation suggestion" layer (not just FAIL, but "here's how to fix it"), false positive rates dropped to acceptable levels.

Results

Benchmarked against manual compliance processes and rule-based automated tools:

Compliance coverage: LLM framework covers regulatory requirements that static rule engines miss — particularly ambiguous or cross-referenced clauses
Audit preparation time: Reduced from weeks to hours through continuous audit trail generation
Regulatory update lag: Near-zero — framework re-extracts rules from updated source documents automatically
CI/CD throughput impact: Minimal — async validation for low-risk changes, synchronous blocking only for high-risk patterns

Try It

pip install agentsre

The agentsre.compliance subpackage (in development) implements the rule extraction and CI/CD gate components. Contributions welcome.

GitHub: github.com/Ajay150313/agentsre

Paper: IEEE ICCBI 2026, Paper ID ICCBI-870 → https://ieeexplore.ieee.org/abstract/document/11619889

Google Scholar: scholar.google.com/citations?user=AyVSzecAAAAJ

If you're implementing compliance automation in a regulated industry — financial services, healthcare, government I'd genuinely like to hear what approaches you've tried. Particularly interested in how teams handle LLM confidence calibration for high-stakes decisions.

I Published IEEE Research on ML-Driven Incident Prediction for Banking Cloud Here's What Actually Works in Production

Ajay Devineni — Fri, 07 Aug 2026 22:32:40 +0000

My paper on machine learning-driven incident prediction for banking cloud operations was just published at IEEE ICCBI 2026. I want to share the actual technical substance not just the abstract because the engineering details are what SRE practitioners can actually use.

The Problem in Plain Terms

You're on-call for a banking platform. 3 AM. CloudWatch fires 47 alerts in 90 seconds. Half are noise. One is the early signal of a cascade that, in 20 minutes, will take down payment processing for 300,000 customers.

You can't tell which one.

This is the state of most SRE operations today: reactive, threshold-based, and drowning in noise.

The paper presents a framework that inverts this. Instead of reacting to symptoms, you predict incidents 15–45 minutes before they materialize, with enough confidence to trigger automated remediation.

What We Actually Built

The framework has four stages. I'll describe them as code-level concepts, not academic abstractions.

1. Telemetry Normalization

Raw cloud metrics are useless for ML they have wildly different scales, units, and sampling frequencies. The first stage normalizes everything:

from agentsre.proactive import TelemetryNormalizer

normalizer = TelemetryNormalizer(
    metrics=['cpu_utilization', 'memory_pressure', 'network_rx_bytes', 
             'disk_iops', 'app_error_rate', 'db_connection_pool_usage'],
    window_minutes=60,
    fill_strategy='forward_fill'  # critical for banking metrics with gaps
)

normalized_df = normalizer.fit_transform(raw_metrics_df)

Without normalization, a CPU spike from 40% to 80% and a memory leak from 85% to 87% look incomparable. After normalization, you can see that the memory trajectory is more dangerous despite smaller absolute change.

2. Feature Engineering The Differentiator

Most practitioners skip this. It's where most of the predictive signal lives.

from agentsre.proactive import TemporalFeatureExtractor

extractor = TemporalFeatureExtractor(
    lag_windows=[5, 15, 30, 60],   # minutes before current timestamp
    rolling_stats=['mean', 'std', 'p95', 'p99'],
    seasonality_encoding=True       # banking has strong daily/weekly patterns
)

# This creates ~200 features from 6 raw metrics
feature_matrix = extractor.transform(normalized_df)

The lag features are key. Incidents in banking cloud don't appear instantaneously — they have precursor signatures 30–60 minutes earlier. Standard monitoring tools never see these because they only look at current values.

3. LSTM Prediction Model

import torch
import torch.nn as nn

class IncidentPredictionLSTM(nn.Module):
    def __init__(self, input_dim=200, hidden_dim=128, num_layers=3, output_dim=3):
        # output_dim=3: probability of [LOW, MEDIUM, HIGH] severity incident
        super().__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, 
                           batch_first=True, dropout=0.2)
        self.classifier = nn.Sequential(
            nn.Linear(hidden_dim, 64),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(64, output_dim),
            nn.Softmax(dim=-1)
        )

    def forward(self, x):
        lstm_out, _ = self.lstm(x)
        return self.classifier(lstm_out[:, -1, :])

Why LSTM specifically? Banking cloud has strong temporal dependencies overnight batch processing, market-open transaction spikes, end-of-month settlement windows. LSTM networks capture these patterns; standard ML models (random forests, XGBoost) do not.

4. Response Orchestration

Predictions without action are dashboards, not engineering. The framework closes the loop:

from agentsre.proactive import ProactiveResponseOrchestrator

orchestrator = ProactiveResponseOrchestrator(
    high_severity_threshold=0.75,
    medium_severity_threshold=0.55,
    actions={
        'HIGH': ['page_oncall', 'scale_out_asg', 'enable_circuit_breaker'],
        'MEDIUM': ['notify_slack', 'pre_warm_cache', 'alert_dbas'],
        'LOW': ['log_to_dashboard', 'queue_for_review']
    }
)

Results

In simulated banking cloud environments, the framework demonstrated:

Excellent prediction accuracy for both high-severity and low severity incidents
15–45 minute predictive window enough time for automated OR human response
Significant reduction in alert fatigue through confidence-scored predictions vs. binary threshold alerts
Measurable improvement in system uptime through proactive intervention

What You Can Steal For Your Team Right Now

Even without deploying the full LSTM stack, you can apply the core ideas:

Add temporal lag features to your existing ML monitoring — if you're already using any ML for anomaly detection, adding 15/30/60-minute lag features of your key metrics will improve accuracy without changing models
Implement confidence-scored alerting — replace binary threshold alerts with probability-based alerts. P(incident) > 0.8 is more actionable than cpu_utilization > 80%
Close the feedback loop — log every prediction with its eventual outcome. This labeled dataset is gold for retraining and is free to collect from day one

Try It

pip install agentsre

The agentsre.proactive subpackage implements all six components of this framework. The library is MIT licensed, production-tested, and actively maintained.

GitHub: github.com/Ajay150313/agentsre

Paper: IEEE ICCBI 2026, Paper ID ICCBI-874 → https://ieeexplore.ieee.org/abstract/document/11619688

Google Scholar: scholar.google.com/citations?user=AyVSzecAAAAJ

Drop a comment if you've tackled incident prediction in production I'm particularly interested in how teams handle concept drift when the incident distribution shifts after a major infrastructure change.

The Vendors Shipped Autonomous SRE Agents. Here's the Governance Layer They Didn't Ship

Ajay Devineni — Fri, 07 Aug 2026 01:51:10 +0000

Microsoft published a real production number this week: Azure SRE Agent has mitigated more than 35,000 incidents and saves over 20,000 engineering hours per month across Microsoft's own services. Dynatrace launched their Autonomous SRE Agent in August 2026 — their Chief Product Officer framing it directly: "Most observability platforms stop at data, leaving humans to find answers, determine what to do, and execute."

Both statements are true. The technology is working at scale. And the governance layer that determines whether you can trust those autonomous decisions is almost never part of the product announcement.

This article covers what the SRE community needs to build for itself five metrics that measure autonomous SRE agent behavior at the semantic layer, where vendor observability stops.

What the Press Releases Don't Cover

Microsoft's 20,000 engineering hours saved per month means tens of thousands of autonomous decisions made. Some percentage of those decisions are wrong. Not because the technology is bad — because all production systems fail at the tail, and autonomous systems that act on imperfect information will sometimes act incorrectly.

The responsible question is not whether to use autonomous SRE agents. The responsible question is: how do you know when the agent's decision was wrong, and what do you do about it?

Azure SRE Agent, Dynatrace Autonomous SRE Agent, AWS DevOps Agent, PagerDuty SRE Agent — all of them are shipping this year. None of them ship with the five measurement patterns below. That's not a criticism. It's a gap the SRE community is positioned to close.

The Five Governance Metrics

1. Root Cause Accuracy Rate (RCAR)

RCAR measures the percentage of agent root cause attributions confirmed correct in your postmortem record, tracked per incident category.

Why per category matters: an agent with 94% overall accuracy may have 70% accuracy for the specific incident class that represents your highest-stakes failures. Overall accuracy hides this. Per-category RCAR surfaces it.

from agentsre.ai_sre_governance import AISREAgentGovernor

governor = AISREAgentGovernor(
    agent_space_id="your-autonomous-sre-agent",
    rcar_threshold=85.0,
    arer_threshold=5.0,
    hor_warning_threshold=30.0,
)

# After each postmortem — wire to your incident management webhook
governor.record_investigation(
    incident_id="INC-2026-0805-001",
    category="database-failure",
    root_cause_confirmed=True,       # postmortem verdict
    remediation_executed=True,
    remediation_rolled_back=False,
    human_overrode=False,
)

Alert threshold: RCAR < 85% for any category over 7 days → suspend autonomous remediation for that category. Not all categories — just the one that's degrading.

2. Autonomous Remediation Error Rate (ARER)

ARER measures the percentage of autonomous remediations that required rollback or correction. This is your blast radius signal — the metric that tells you whether wrong investigations are being acted upon.

The risk profile is asymmetric: a wrong investigation that was caught before action is recoverable. A wrong investigation followed by an autonomous remediation applied to the wrong component is a different category of failure entirely.

Alert threshold: ARER > 5% → suspend all autonomous remediations immediately, page the named SRO.

results = governor.collect("database-failure")
for r in results:
    print(r)
    if r.breached:
        # Suspend autonomy for this category
        disable_autonomous_remediation(r.metric, r.alert_message)

3. Human Override Rate (HOR)

HOR measures the percentage of agent recommendations that on-call engineers override before execution. This is the leading indicator engineers begin overriding before RCAR measurably falls.

When an engineer overrides the agent's recommendation, they're telling you the agent's model of the system no longer matches their understanding of reality. That's the earliest possible signal of drift — earlier than any metric that requires waiting for a postmortem.

Alert threshold: HOR > 30% for any incident category → review the agent's topology model and context freshness before RCAR falls.

4. Decision Quality Rate (DQR)

DQR measures the percentage of agent decisions falling within expected behavioral bounds against a rolling baseline. It is not an accuracy metric — it doesn't require ground truth labels. It's a drift detection signal that fires before user-visible failures.

from agentsre import AgentSLICollector, TaskRecord

collector = AgentSLICollector()

collector.record(TaskRecord(
    task_id="t-001",
    task_class="incident-triage",
    tool_calls=4,
    decision_confidence=0.71,    # drifting from 0.91 baseline
    required_escalation=False,
    completed=True,
))

for r in collector.collect("incident-triage"):
    if r.breached:
        alert_oncall(r.name, r.alert_message)

Alert threshold: DQR < 85% for 15 minutes → reduce autonomy level before the incident escalates.

5. Approval Queue Depth Drift (AQDD)

AQDD is the metric that standard SLO burn-rate alerts structurally cannot detect. When tasks are submitted for human approval and humans fall behind, the queue grows silently — no completion event, no failure event, no burn. Standard SLO dashboards show healthy. Operations are degrading.

Alert threshold: AQDD > 2x baseline for 30 minutes → human review layer is saturated, page SRO immediately.

The Progressive Autonomy Model

These five metrics gate a progressive autonomy model for autonomous SRE agents:

Level 1 — Triage only: DQR and TIE instrumented. Agent surfaces signals. Humans decide everything. No RCAR or ARER needed yet.

Level 2 — Advised actions: HER tracked. Agent recommends remediation. Human approves every action. RCAR begins as measurement.

Level 3 — Bounded autonomy: RCAR > 85% sustained 30 days for a specific incident category → agent executes autonomously for that category. ARER tracked. Any breach suspends autonomy.

Level 4 — Governed autonomy: All five metrics active and within bounds. Named SRO. Quarterly review of all categories.

Microsoft's Azure SRE Agent is operating at Level 3-4 for specific incident classes across Microsoft's internal services. That's what 35,000 incidents and 20,000 hours saved means. The governance model that makes it possible is the same model described above — it's just not in the press release.

What To Do This Week

If your team is evaluating Dynatrace Autonomous SRE Agent, AWS DevOps Agent, or any autonomous SRE tooling launching this month:

Instrument DQR and TIE first — observation mode, no alerts yet
Run 30 days of behavioral baseline before setting any threshold
Register a named SRO before enabling any autonomous remediation
Start at Level 1 (triage only). Earn Level 2 with 30 days of data.
Write the postmortem process for the agent before the first wrong decision — not after

The vendors shipped the capability this week. The governance is yours to build.

Open-source implementation: https://github.com/Ajay150313/agentsre
LinkedIn discussion: https://www.linkedin.com/feed/update/urn:li:groupPost:6585254-7490483687269064704/?utm_source=share&utm_medium=member_desktop&rcm=ACoAACIp55QBRGVmAcEbf0D-1PaR5vEbm2yMcJU

Which of the five metrics does your team currently track for your autonomous SRE tooling?

MCP Final Spec Shipped. Now It Needs SRE Governance

Ajay Devineni — Wed, 29 Jul 2026 01:53:54 +0000

The Model Context Protocol shipped its final specification on July 28, 2026 — adding Tasks and MCP Apps extensions, with LangGraph 1.0 treating MCP tools as first-class graph nodes and Netzilo shipping cross-platform runtime kill switches for compromised agents.

MCP stops being an experimental protocol today. It is production infrastructure.

Production infrastructure needs SRE governance. Here is what that looks like for MCP.

What Changed in the Final Spec

The final MCP spec introduces two extensions that change the reliability picture significantly:

Tasks: Long-running operations are now a first-class primitive in MCP. An agent can initiate a task, the MCP server executes it asynchronously, and the result is available when complete. This is not a workaround — it is the specified behavior. Tasks can now run for minutes or hours autonomously without the agent polling.

MCP Apps: A standardized way for MCP servers to expose UI components alongside tool calls. Agents can now interact with structured interfaces, not just raw function returns.

The reliability implication of both: MCP is no longer a request-response protocol with human-timescale latency. It is an orchestration layer for long-running autonomous work.

The Four SRE Gaps That Arrive With the Final Spec

1. Every MCP Tool Needs an SLO

Your agent's reliability is bounded by the worst MCP server it calls. An agent that calls five MCP tools has five dependency SLOs. If any one fails silently — returning an unexpected format, timing out without an error, returning stale data — your agent's Decision Quality Rate degrades with no infrastructure-layer signal.

Instrument Tool Invocation Efficiency per MCP server, not just per task class:

from agentsre import AgentSLICollector, TaskRecord

collector = AgentSLICollector()

# Track per MCP server — not just per task
for mcp_server, tool_calls in mcp_trace.server_calls.items():
    collector.record(TaskRecord(
        task_id=task_id,
        task_class=f"mcp:{mcp_server}",   # per-server tracking
        tool_calls=tool_calls,
        decision_confidence=task_confidence,
        completed=task_completed,
    ))

# Each MCP server gets its own TIE baseline and breach alert
results = collector.collect(f"mcp:{mcp_server}")

This gives you per-MCP-server behavioral baselines. When a specific server starts generating elevated tool call counts — retry storms, format changes, latency degradation — TIE catches it before it surfaces as task failure.

2. MCP Circuit Breaker at the Boundary

When an MCP tool fails, what does your agent do? Most frameworks retry. Retry logic without a circuit breaker at the MCP boundary produces retry storms — the agent calls a failing tool repeatedly, TIE climbs silently, and the first observable signal is a cost spike.

Apply the circuit breaker at the MCP call layer:

from agentsre import AgentChainCircuitBreaker

# Circuit breaker per MCP server
mcp_breakers = {
    server_name: AgentChainCircuitBreaker(
        open_threshold=85.0,    # open if semantic success rate drops below 85%
        close_threshold=95.0,   # close after recovery
        on_state_change=lambda state: alert_mcp_server_owner(server_name, state),
    )
    for server_name in mcp_servers
}

def call_mcp_tool(server_name: str, tool: str, params: dict):
    breaker = mcp_breakers[server_name]

    if not breaker.allow_request(server_name, tool):
        # Circuit open — route to fallback or human escalation
        return fallback_handler(server_name, tool, params)

    result = mcp_client.call(server_name, tool, params)
    success = validate_mcp_result(result)
    breaker.record_result(server_name, tool, success=success)
    return result

3. Long-Running Tasks Need Approval Gates

The Tasks extension means an MCP operation can run for minutes or hours. During that time, the agent is making autonomous decisions with no human checkpoint. The governance question is not whether Tasks should exist — it's which tasks get an approval gate and which run fully autonomous.

Map your task classes to your blast radius tiers:

from agentsre.production_readiness import BlastRadiusTier

MCP_TASK_GATES = {
    "log-analysis":       BlastRadiusTier.LOW,     # fully autonomous
    "config-validation":  BlastRadiusTier.LOW,     # fully autonomous
    "service-restart":    BlastRadiusTier.MEDIUM,  # approve before execute
    "database-migration": BlastRadiusTier.HIGH,    # always human approval
    "credential-rotation":BlastRadiusTier.CRITICAL,# named approver + audit
}

def should_gate_task(task_type: str) -> bool:
    tier = MCP_TASK_GATES.get(task_type, BlastRadiusTier.HIGH)
    return tier in [BlastRadiusTier.HIGH, BlastRadiusTier.CRITICAL]

4. Kill Switches Need Runbooks

Netzilo ships kill switches for compromised agents. A kill switch without a runbook is a panic button — you know how to stop the agent, but not what to check, what to preserve for the postmortem, or how to safely restart.

Before enabling any MCP-connected agent in production, write three runbook sections:

Trigger conditions: What observable signal causes you to reach for the kill switch? TIE > 3x baseline? AQDD > 5x? An out-of-scope API call in your audit log?

Kill procedure: Kill switch activation → preserve agent state for postmortem → notify dependent systems → route pending tasks to human queue.

Recovery criteria: What must be true before restarting the agent? Root cause identified, blast radius assessed, SLO targets reviewed.

Connecting to the Existing Framework

The MCP final spec adds one new governance requirement to the existing agentsre framework:

Per-MCP-server TIE tracking — instrument task_class=f"mcp:{server_name}" for each server
MCP boundary circuit breaker — AgentChainCircuitBreaker at the MCP call layer
Task blast radius mapping — BlastRadiusTier per MCP task class
Kill switch runbook — added to RunbookDefinition detection/containment/recovery

Everything else in the governance framework — DQR, HER, AQDD, RCAR, named SRO, pre-go-live validator — applies unchanged.

Open-source implementation: https://github.com/Ajay150313/agentsre
LinkedIn discussion: https://www.linkedin.com/feed/update/urn:li:groupPost:6585254-7488048830827966464/?utm_source=share&utm_medium=member_desktop&rcm=ACoAACIp55QBRGVmAcEbf0D-1PaR5vEbm2yMcJU

What's your biggest reliability concern now that MCP Tasks are a production primitive?

Your AI Agent Framework Has a CVE. Here's the SRE Security Model That Fixes It.

Ajay Devineni — Sat, 25 Jul 2026 01:32:23 +0000

CISA added Langflow to its Known Exploited Vulnerabilities catalog in July 2026. CVE-2026-55255, CVSS 9.9 — an access control flaw in the /api/v1/responses endpoint that let any authenticated user execute flows belonging to another user, exposing embedded LLM provider keys and cloud credentials.

It is the first AI agent orchestration platform on the KEV list. The same list as Windows zero-days, Cisco IOS vulnerabilities, and OpenSSL critical patches. The list that US federal agencies are required to remediate on fixed timelines.

The operational implication is straightforward: AI agent frameworks are now treated as critical infrastructure by the cybersecurity community. The security model for experimental tooling is not adequate. The security model for critical infrastructure is required.

What the Exploit Actually Looked Like

The Langflow vulnerability was not a model failure. The LLM did nothing wrong. The exploit path was entirely in the orchestration layer — the software built around the model to give it tools, memory, and production access.

The attack chain, documented by Sysdig between June 22–25, 2026:

Attacker authenticates to a Langflow instance (any user account)
Supplies a victim's flow UUID to the /api/v1/responses endpoint
No ownership check — endpoint executes the victim's flow
LLM provider API keys, cloud credentials, and database secrets embedded in the flow are now accessible
Attacker chains with Langflow RCE CVE-2026-33017 to stage second-stage implants

The damage came from what the agent was allowed to do: read credentials, execute arbitrary flows, communicate outward. Three ordinary capabilities, chained by an attacker through an authorization gap in the framework.

This is the security shape of AI agents in 2026. The production question is no longer "did the model answer safely?" It is "what is the agent allowed to see, call, and change — and who can control that from outside?"

Agent Zero Trust — July 2026's Dominant Security Theme

Adversa AI's July 2026 security report documents the shift: the dominant theme across the month is "Agent Zero Trust." Google DeepMind and Anthropic have both published frameworks stating the same principle: treat AI agents as potential insider threats.

Not because agents are malicious. Because they have the access profile of a malicious insider: broad system access, autonomous execution, and the ability to read, write, and communicate across your production environment.

The Zero Trust model applied to AI agents means:

Verify every action — not just at authentication time
Grant minimum necessary permissions per task class — not broad system access
Monitor what the agent accesses, not just what it outputs
Treat context inputs as untrusted — prompt injection through data sources is an active exploit class

Cycode's 2026 report adds the operational baseline: 80% of IT workers have already observed AI agents performing tasks without authorization. The threat is not theoretical.

The SRE Security Model for AI Agents

The SRE community has been applying defense-in-depth to production infrastructure for two decades. The same principles apply to AI agent frameworks — with specific adaptations for the failure modes that make agents different from traditional software.

1. Treat AI Agent Frameworks as Infrastructure — Not Libraries

The Langflow CVE makes this concrete: an agent orchestration framework that has production database connections, cloud credentials, and LLM API keys embedded in it is infrastructure. It requires:

Patch schedules: same cadence as your OS and network hardware, not the same as a Python package update
Change control: framework version upgrades go through your deployment pipeline with behavioral canary testing (this is what agentsre.sprawl.FrameworkVersionGovernance implements)
Access reviews: who can reach the Langflow endpoint, who can create flows, who can execute other users' flows — reviewed quarterly
Vulnerability scanning: agent frameworks added to your CVE monitoring the same as any other infrastructure component

2. Least Privilege Per Task Class

The Langflow exploit was possible because flows had access to credentials they needed for legitimate use — but the authorization model didn't prevent cross-user execution. The fix at the architectural layer is least privilege scoped to task class:

from agentsre.production_readiness import BlastRadiusAssessment, BlastRadiusTier

# Scope permissions to the minimum required for each task class
# Same principle as IAM least privilege — applied to AI agents
assessment = BlastRadiusAssessment(
    agent_id="your-agent",
    tier=BlastRadiusTier.LOW,           # read-only until demonstrated reliable
    can_execute_writes=False,            # no write access at initial deployment
    can_modify_infrastructure=False,     # never at initial deployment
    worst_case_description="Read-only access — credential exposure limited to read scope",
    acceptable=True,
    signed_off_by="security-lead@team.com",
)

# Promote to MEDIUM only after:
# - 30 days observation at LOW with zero security events
# - RCAR > 85% — root cause accuracy demonstrated
# - Named security reviewer sign-off

Each task class gets its own permission scope. An agent that summarizes logs doesn't need the permissions required to restart services. An agent that correlates alerts doesn't need write access to your configuration store.

3. Context Is an Attack Surface — Treat External Inputs as Untrusted

The EchoLeak vulnerability (CVE-2025-32711, CVSS 9.3) established the attack pattern a year ago: a single crafted email caused Microsoft 365 Copilot to exfiltrate sensitive data by following instructions hidden in the email's content. The agent behaved helpfully — exactly as designed. The instructions happened to be malicious.

Every external data source your agent reads is a potential injection vector:

Emails, tickets, and support messages
Log files and error reports (Adversa AI documented 85% exploitation success rate via Sentry error reports)
Code comments and PR descriptions (CVE-2025-53773, CVSS 9.6 — GitHub Copilot RCE via PR descriptions)
RAG knowledge base documents

The operational requirement: sanitize and validate all external inputs before they reach the agent's context window. Treat every user-generated artifact as potentially adversarial.

4. Runtime Monitoring of Agent Actions — Not Just Outputs

Current AI agent observability watches what the agent produces. Agent Zero Trust requires watching what the agent does during execution:

Which data sources did it access?
Which APIs did it call?
What did it write or modify?
Did it attempt to access resources outside its permitted scope?

This is the runtime monitoring layer that sits above the semantic SLIs (DQR, TIE, HER) and below the infrastructure metrics. It's the agent equivalent of CloudTrail — an audit log of agent actions, not just agent results.

The Production Security Checklist

Before any AI agent framework touches production credentials, external APIs, or customer data:

□ Framework added to vulnerability management process (CVE monitoring + patch schedule)
□ Access control reviewed — who can execute flows, who can read credentials
□ Least privilege scoped per task class (BlastRadiusTier assessed and signed off)
□ External input sanitization in place for all data sources the agent reads
□ Runtime action logging enabled — audit trail of what the agent accesses
□ Credential isolation — LLM API keys and cloud credentials in secrets manager, not embedded in flows
□ Named security reviewer (separate from SRO) who reviews permission scope quarterly

Connecting to the Broader Framework

The AI-SRE governance framework now has a security layer:

agentsre.production_readiness — BlastRadiusAssessment + 5 pre-go-live checks
agentsre.sprawl.FrameworkVersionGovernance — behavioral canary before framework upgrades (catches both performance regressions and behavior changes from security patches)
agentsre.ai_sre_governance — RCAR, ARER, HOR for ongoing governance

The Langflow CVE adds one requirement to the pre-go-live checklist: framework added to vulnerability management process. The rest of the security model follows from SRE principles already in place.

Has your team added your AI agent framework to your vulnerability management process?

60% of SREs Don't Trust AI Agents Enough to Deploy Them. Here's How to Build the Trust Layer

Ajay Devineni — Wed, 22 Jul 2026 02:12:27 +0000

The Register published a survey this week of 696 SRE and DevOps experts. The headline number: 73% are not using AIOps in production. Only 8% have deployed it. 19% are in pilot.

The question worth sitting with isn't why adoption is low. It's why the 60% who cited lack of trust as their #1 barrier haven't been able to close that gap — even as the vendors selling AI SRE tooling multiply weekly.

The answer is straightforward once you name it: trust in production systems is not built through demonstrations or benchmarks. It's built through sustained measurement. You trust your database because you have 90 days of error rate, query latency, and replication lag data showing it behaves within expected bounds. You trust your Kubernetes cluster because liveness probes, resource limits, and rolling deployment gates have been running for months.

AI agents don't get a different deal. You trust them when you can measure them — and right now, most teams can't.

The Measurement Gap Underneath the Trust Gap

The same Futurum Group enterprise survey that found 71% of companies claim to deploy AI agents also found that only 11% of intended agentic use cases from the previous year actually reached production. The gap between "claiming to deploy" and "actually in production" isn't primarily a capability gap. It's a governance and measurement gap.

Teams that reach production with AI agents share one thing: they instrumented the agent's behavioral baseline before they tried to set SLO targets, and they set SLO targets before they enabled autonomous actions. The sequence matters. You cannot trust something you haven't measured. You cannot govern something you don't trust.

The measurement layer that builds trust for AI agents has five components, each addressing a failure mode that standard infrastructure observability misses entirely.

The Five Measurements That Build Trust

1. Decision Quality Rate (DQR) — Is the agent making valid decisions?

Standard observability: HTTP 200, latency 142ms, error rate 0.0%. All green.
DQR: decision confidence 0.91 → 0.61 over 4 hours. Something drifted.

Alert threshold: DQR < 85% for 15 minutes → investigate before users notice.

2. Tool Invocation Efficiency (TIE) — Is it working harder than it should?

TIE measures the ratio of tool calls per task completion against a rolling baseline. When an agent compensates for degraded tools, stale context, or framework overhead, it invokes more tools per task. TIE rising above 1.5x baseline is a signal that something in the agent's environment has degraded — even if every individual tool call succeeds.

This is the metric that catches retry loops before the billing cycle closes. It's also the signal that catches framework upgrade regressions before they surface as wrong outputs.

3. Human Escalation Rate (HER) — Is it handing off more than expected?

HER measures the percentage of tasks requiring human intervention. Set a budget — 5% is a reasonable starting point. When HER climbs above that budget, the agent is telling you it's operating outside its reliable envelope. The signal is direct: more escalation means less reliable autonomous operation.

HER also gates production readiness. An agent with HER > 5% in the 30-day observation window is not ready for production. This is the readiness probe equivalent for AI agents.

4. Approval Queue Depth Drift (AQDD) — Are humans falling behind?

AQDD tracks tasks submitted for human approval against a baseline. Standard SLO burn-rate alerts miss this failure mode entirely: the tasks are submitted, the agent thinks it's working, but humans aren't reviewing fast enough and the queue grows silently.

Alert threshold: AQDD > 2x baseline for 30 minutes → human review layer is saturated.

5. Root Cause Accuracy Rate (RCAR) — Is the diagnosis actually right?

RCAR is the trust metric for AI SRE agents specifically. It measures the percentage of root cause attributions confirmed correct in subsequent postmortems, per incident category.

An agent with 94% overall accuracy may have 70% accuracy for the specific incident class that represents your highest-stakes scenarios. RCAR per category surfaces this. Set a threshold: RCAR < 85% for any category over 7 days → suspend autonomous remediation for that category.

The Trust-Building Sequence

Trust in AI agents is not built by deploying them and watching. It's built through a specific sequence:

30 days of observation before any SLO target. Instrument all five metrics in read-only mode. Record what the agent actually does — DQR baseline, TIE baseline, HER baseline. You cannot commit to reliability you haven't measured. Skipping this step is why pilots fail to convert.

SLO targets before autonomous actions. Set your targets based on the observation window data. DQR target should be the 10th percentile of your observation window, not a vendor default. HER target should reflect what your team's review capacity can actually sustain.

Named SRO before production. One person — not a team. Their pager fires when the SLO breaches. This is the accountability layer that converts measurement into action. Without it, a breaching metric has no owner.

Runbook before the first incident. Write what you'll do when DQR drops, when TIE spikes, when RCAR falls below threshold. Two hours to write. Six hours saved on the first real incident.

from agentsre import AgentSLICollector, TaskRecord
from agentsre.production_readiness import AgentGoLiveValidator

# Step 1: 30 days of observation
collector = AgentSLICollector()
collector.record(TaskRecord(
    task_id="t-001",
    task_class="incident-triage",
    tool_calls=3,
    decision_confidence=0.91,
    completed=True,
))

# Step 2: Review baselines after 30 days, then set SLOs
# Step 3: Register SRO before enabling autonomous actions
validator = AgentGoLiveValidator(agent_id="your-agent")
# ... register SLO, SRO, runbook, blast radius
report = validator.validate(["incident-triage"])
print(report.summary())
# Fix any blocking failures before go-live

Why 89% of Planned Deployments Don't Reach Production

The Futurum Group data says 71% of businesses claim to be deploying AI agents but only 11% of intended use cases actually reached production last year. The gap is the governance sequence above, skipped.

Teams that skip the 30-day observation period don't have baselines. Without baselines, they can't set meaningful SLO targets. Without SLO targets, every anomaly is noise — there's no threshold to tell them whether the agent is within acceptable bounds. Without that signal, trust doesn't build. The pilot stalls.

The technology is ready. The measurement practice is the missing step.

The trust gap closing requires the same thing that closed the Kubernetes adoption gap in 2018: a governance layer that makes reliability measurable, owned, and governed. SLOs, named owners, runbooks. Applied to the AI agent layer.

Open-source implementation: https://github.com/Ajay150313/agentsre
LinkedIn discussion: https://www.linkedin.com/feed/update/urn:li:activity:7485516083882508288?utm_source=share&utm_medium=member_desktop&rcm=ACoAACIp55QBRGVmAcEbf0D-1PaR5vEbm2yMcJU

What's the one metric your team would need to trust an AI agent with autonomous remediation?

Your AI Agent Has a 95% Success Rate. Your Workflow Has a 36% Success Rate. Here's the SRE Fix.Tags

Ajay Devineni — Fri, 17 Jul 2026 01:36:50 +0000

Your AI Agent Has a 95% Success Rate. Your Workflow Has a 36% Success Rate. Here's the SRE Fix. number published this week has been sitting with me since I read it.At 95% per-step accuracy — which is optimistic for current LLMs — a 10-step agent workflow succeeds roughly 60% of the time. At 20 steps, you're at 36%.That math comes from Lusser's Law, which reliability engineers have known since the 1950s. The reliability of a series of components equals the product of their individual reliabilities. We apply it to hardware systems, distributed architectures, deployment pipelines. We have not been applying it to AI agent workflows — and the industry is starting to feel the consequences.Microsoft's Azure SRE Agent team found that problems requiring more than four handoffs almost always failed. That's not a benchmark result. That's a production observation from the team that built one of the most sophisticated AI SRE systems in existence.Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 because of costs, unclear value, or inadequate risk controls.The inadequate risk controls part is what I want to address. Because the risk is calculable. And if the risk is calculable, it can be governed.Why the Demo Number LiesDemos usually hide compounding failure because they only show two or three steps. Production environments are usually five or more steps over messy inputs and edge cases.A demo that shows an agent complete a task in three steps with clean data is showing you 0.95^3 = 0.857 — 85.7% reliability. That looks good. Your stakeholders are impressed.The same agent deployed to a production incident investigation workflow with 10 steps is giving you 0.95^10 = 0.599 — just under 60% reliability. In SRE terms: your agent is failing roughly four out of ten production incident investigations, silently, without throwing an error.This mathematical reality makes autonomous multi-step workflows fundamentally challenging at production scale, requiring teams to rethink how they architect agent systems.Rethinking the architecture is exactly right. But it's not enough to know the math. You need a framework for governing the math — which is what SRE discipline gives you.The Missing SLO LayerWe have SLOs for services. We have error budgets for APIs. We have reliability targets for databases.We have almost none of that for agent workflows.A microservice either returns a 200 or it doesn't. AI agents return responses on a spectrum from correct to confidently wrong. Everyone's shipping agents. Almost nobody has a framework for deciding how much failure is acceptable.That's the gap. And it's the same gap SRE filled for microservices fifteen years ago.The answer isn't "make every step more reliable." At scale that's impossible — you can't prompt-engineer your way to 99.9% per-step accuracy across a complex incident investigation workflow. The answer is what SRE has always done: define an acceptable failure rate, measure against it, and design the system to fail gracefully when it breaches.Introducing the Workflow Reliability BudgetA Workflow Reliability Budget (WRB) applies error budget thinking to multi-step agent workflows rather than to individual services.The calculation:WRB = R_target / R_actual

where R_actual = (P_step)^n (Lusser's Law for the full workflow)
and R_target = your acceptable success rate for this workflow classFor an incident investigation workflow with 8 steps and 95% per-step accuracy:pythonfrom agentsre.slo_burn import AgentSLOBurnTracker, SLOTarget
import math

def calculate_workflow_reliability(
per_step_accuracy: float,
num_steps: int
) -> float:
"""
Apply Lusser's Law to calculate end-to-end workflow reliability.
This is the number your demo hides and your production exposes.
"""
return per_step_accuracy ** num_steps

The math your stakeholders need to see

per_step = 0.95
steps = 8
workflow_reliability = calculate_workflow_reliability(per_step, steps)

print(f"Per-step accuracy: {per_step:.0%}")
print(f"Workflow steps: {steps}")
print(f"End-to-end reliability: {workflow_reliability:.1%}")

Output: End-to-end reliability: 66.3%

Now wire this to your SLO framework

tracker = AgentSLOBurnTracker(
agent_id="incident-investigation-agent-v2",
task_class="multi-step-incident-investigation"
)

Workflow-level SLO — not per-step accuracy

tracker.add_slo(SLOTarget(
metric_name="WorkflowCompletionRate",
target_pct=90.0, # 90% end-to-end success
window_days=30,
good_threshold=1.0, # 1.0 = workflow completed successfully
higher_is_better=True
))

DQR at workflow level — not just per tool call

tracker.add_slo(SLOTarget(
metric_name="WorkflowDQR",
target_pct=85.0, # 85% of completed workflows with correct output
window_days=30,
good_threshold=0.8,
higher_is_better=True
))The key insight: WorkflowCompletionRate is not the same as per-step accuracy. A workflow that completed but produced a wrong final answer counts as a failure against DQR even if every intermediate step appeared to succeed. That's the distinction most teams miss.The Three Architecture FixesThe math points to three specific changes that actually move the reliability number:Fix 1: Shorten the chainAchieving high workflow reliability requires either very high per-step accuracy (above 99%), short workflows (fewer steps), or resilience architecture that detects and corrects errors before they compound.Every step you remove from an agent workflow is a multiplicative reliability improvement. An 8-step workflow at 95% per-step accuracy is 66% reliable. A 5-step workflow at the same per-step accuracy is 77% reliable. That's an 11-point improvement from design alone, not from model improvement.In practice: ruthlessly question every step in your agent workflow. If a step retrieves context the agent has already retrieved in a previous step, remove it. If a step validates something that was already validated, combine them. Shorter chains are not lazy design — they are reliability engineering.Fix 2: Add intermediate checkpointsError budgets force you to confront this compounding math instead of pretending each step is independent.The error doesn't compound uniformly — it compounds through dependency. An incorrect output at step 2 feeds into step 3, which builds on wrong context, which makes step 4 worse. By step 6, the agent is reasoning on top of accumulated wrongness.Intermediate checkpoints break this chain. After every two or three steps, the workflow validates intermediate output before continuing. Not a human checkpoint — a programmatic DQR check on the intermediate result.pythonfrom agentsre.reasoning_trace import AgentDecisionTrace
from agentsre.eval_pipeline import EvalRun, EvalCase

def checkpoint_intermediate_output(
agent_id: str,
step_number: int,
intermediate_output: str,
expected_properties: list,
trace: AgentDecisionTrace
) -> bool:
"""
Validate intermediate output before the next step consumes it.

If this checkpoint fails, the workflow stops and replans
rather than compounding the error through subsequent steps.

Returns True if output passes validation. False triggers replan.
"""
# Check each required property of the intermediate output
failed_properties = []
for prop in expected_properties:
    validator_fn = prop.get('validator')
    if validator_fn and not validator_fn(intermediate_output):
        failed_properties.append(prop.get('name'))

if failed_properties:
    # This is a replan event — RTD increments
    trace.record_replan(
        reason=(
            f"Step {step_number} intermediate output failed "
            f"properties: {failed_properties}. "
            "Replanning rather than compounding."
        ),
        tool_that_failed=f"step_{step_number}_output",
        new_plan=f"Retry step {step_number} with corrected inputs"
    )
    return False

return TrueFix 3: RTD as your workflow health signalRTD (Reasoning Trace Depth) already tells you when compounding is happening. When an agent re-plans three times in a 10-step workflow, it's because earlier steps produced outputs the agent couldn't work with cleanly. RTD rising is your leading indicator of compounding failure before the final output reveals it.The signal chain for workflow reliability:RTD rising → intermediate outputs degrading → DQR declining
 ↑                    ↑                        ↑

earliest signal mid-workflow signal lagging signalIf you're watching only DQR at the workflow level, you're watching the lagging signal. Wire RTD monitoring to your intermediate checkpoint layer and you catch compounding before it propagates through the full chain.The Workflow Reliability Dashboard Query# CloudWatch Insights — workflow reliability trend
fields @timestamp, agent_id, reasoning.rtd, outcome.task_completed,
quality.confidence_proxy, outcome.latency_ms
| filter trace_type = 'agent_decision_trace'
| filter agent_id = 'incident-investigation-agent-v2'
| stats
avg(reasoning.rtd) as avg_rtd,
sum(outcome.task_completed) / count() * 100 as completion_rate_pct,
avg(quality.confidence_proxy) as avg_confidence
by bin(1d)
| sort @timestamp ascIf completion_rate_pct is tracking below your WRB target, look at avg_rtd in the same window. Rising RTD before declining completion rate means compounding is starting — you have a window to intervene before the workflow-level SLO breaches.What This Means for Your Agent DesignThe teams that ship design short, checkpointed chains. The teams that stall keep adding steps and hoping.Lusser's Law isn't telling you agents can't work. It's telling you the design constraints for making them work reliably at scale. Fewer steps. Checkpoints between steps. RTD monitoring to catch compounding early. Workflow-level SLOs rather than per-step accuracy benchmarks.The math is the same math reliability engineers have been applying to hardware and distributed systems for decades. The discipline already exists. The application to AI agent workflows is new — but the principles are not.All frameworks referenced in this post are in the agentsre library at github.com/Ajay150313/agentsre. MIT licensed.Ajay Devineni | AWS Community Builder | Senior SRE/Platform EngineerDEV.to Tags:

sre #agenticai #devops #aws

Your AI Agent Passed the Gate. Now It's in Production. Here's Why That's When the Real SRE Work Starts

Ajay Devineni — Sun, 12 Jul 2026 10:32:56 +0000

AWS expanded their DevOps Agent four days ago. It now validates code autonomously before production — assessing changes, running tests, surfacing risks without waiting for a human to trigger it.
I want to write about what that means for SRE practitioners, because I think the conversation around this capability is missing something important.
The narrative is usually framed around speed. How much faster does code get validated? How much MTTR improves? Those numbers are real and they matter. But they measure the easy part.
The hard part isn't getting an AI agent to validate a release. The hard part is knowing whether to trust what it found and what to do when it missed something.
I've been building SLI frameworks for AI agents in production for seven months. Every framework I've built has been motivated by the same underlying question: how do you govern an actor that moves faster than your ability to verify it?
This post is the answer I've landed on for the release validation case specifically.
The Gap Between Observable and Interpretable
MIT Sloan published research showing people are 2.8 times more likely to trust AI systems they can interpret — not just observe, but interpret. The difference matters enormously in practice.
Observable means you can see what happened. The agent ran. It approved the release. A log entry exists.
Interpretable means you understand why. What signals did the agent evaluate? What did it decide to weight heavily? What did it see but discount? What would have caused it to reject instead of approve?
Most AI release validation tools give you the first. Very few give you the second. And in a postmortem after a release caused a production incident, the audit log alone doesn't answer the question you actually need answered: what did the agent see, and why did it say yes?
What Happens When It Gets It Wrong
Last month, a team shared a scenario I've heard variations of a dozen times. Their AI release validation tool approved a change. The change caused a latency regression in a payment processing path that only appeared under specific traffic conditions. Staging didn't catch it. The agent didn't catch it either.
The postmortem question was: what did the agent evaluate? The answer was: we have an approval timestamp and a summary. We don't have a reasoning trace. That's the gap. And it's not a vendor failure — it's a design gap that the SRE team needs to close, because the vendor's job is to ship a capable agent. The SRE team's job is to govern it.
The Three Things You Need Before Trusting AI Release Validation
I've been building toward this over seven months of framework work. Here's what I've concluded about the minimum viable governance layer for AI release validation in a regulated production environment.

A reasoning trace, not just an audit log An audit log tells you what the agent did. A reasoning trace tells you how it got there. One structured record per validation run — not per tool call capturing the initial hypothesis, what signals the agent evaluated, how it weighted conflicting evidence, and what would have caused a different outcome. This is directly related to RTD (Reasoning Trace Depth). A validation agent that evaluates twelve signals in parallel and returns a summary has high RTD and low interpretability. A validation agent that forms a hypothesis, follows the causal chain, and surfaces its reasoning has low RTD and high interpretability. Those two agents might produce the same approval decision. They produce very different postmortem data when they get it wrong.
A pre-validation state check, not just a post-validation summary Before an AI agent validates any release, it should check three things about the current state of the system it's validating into: Error budget remaining if the target service is already burning error budget faster than expected, a release that introduces even a 0.1% error rate increase is a different risk than the same release on a healthy system. Recent change velocity how many changes have been deployed to this service in the last 24 hours? High velocity plus AI-validated release is a compound risk that deserves human review regardless of what the agent found. Blast radius of what's being validated — which downstream services could be affected if the validation missed something? An agent validating a change to a payment processing path needs different scrutiny than one validating a change to an internal dashboard. None of these checks are about the agent's capability. They're about whether the conditions are right for autonomous action.
A clear escalation path when confidence is below threshold AI release validation agents don't always produce binary confident decisions. Sometimes the evidence is ambiguous. Sometimes there's a conflict between what the static analysis found and what the load test showed. Sometimes the agent finds something it can't classify. The question is what happens then. Does the agent approve anyway? Does it block? Does it escalate to a human with a specific question? The answer should always be the third option when confidence is below a defined threshold — and that threshold should be set by the SRE team, not by the vendor's default configuration. The Pre-Action SRE Gate from the agentsre library handles this for remediation agents. The same pattern applies here: if the agent's confidence is below threshold, the deployment gates pending human review of the specific conflicting signals. Not a generic "human approved" checkbox a specific review of what the agent couldn't resolve. The Production Readiness Checklist for AI Release Validation Before you trust an AI agent to validate releases autonomously in a regulated environment, here's what needs to be in place: The agent produces one reasoning trace per validation run not a summary, a trace you can replay in a postmortem. Your error budget is monitored per service, and the validation agent reads it before approving any release. A confidence threshold is defined and tested below threshold, the agent escalates rather than decides. A rollback procedure exists that doesn't depend on the same agent that validated the original release. A revert drill has been practiced in the last 30 days you've confirmed your team can disable the AI validation layer and revert to human-only release management within a defined time window. The last point is the one most teams skip. When the AI validation layer produces a bad approval and causes an incident, your team needs to be able to operate without it immediately — not after they've spent an hour figuring out how to disable it. Where This Sits in the Arc If you've been following this series: Post 1 established that AI agents need on-call rotation discipline. Post 4 introduced DQR, TIE, HER, AQDD as the measurement layer. Post 11 introduced RTD as reasoning observability. Post 13 introduced the Pre-Action SRE Gate. This post applies all four to the release validation case the specific scenario AWS just made mainstream with their DevOps Agent expansion. The governance layer doesn't come from the vendor. It comes from the SRE discipline your team already has, applied deliberately to a new class of actor. All the tools for building it are in the agentsre library at github.com/Ajay150313/agentsre. MIT licensed. Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer

Training a Model Is a Research Skill. Operating One Is an Infrastructure Discipline.

Ajay Devineni — Thu, 09 Jul 2026 01:36:40 +0000

Why the AI infrastructure hiring crisis is hiding in plain sight — and what production-grade ML ops actually requires

I've watched a GPU bill hit five figures in a single day because nobody had a cost circuit breaker on the inference endpoint.

The model was good. The research team had done real work. The fine-tuning was solid. Nobody had thought about what happens when you put a latency-sensitive LLM inference service behind real production traffic without auto-scaling bounds, without request rate limits, without a cost anomaly alert, and without a rollback path for the model weights.

The GPU kept serving. The bill kept climbing. The on-call engineer hired for their PyTorch experience, not their production infrastructure instincts — had no runbook for "inference endpoint is costing $800 per hour and accelerating."

This is the most predictable failure mode in AI infrastructure right now, and it's happening at scale.

The Hiring Category Error

The résumé filter that's creating this problem is specific: companies are screening for ML credentials PyTorch, Hugging Face, "fine-tuned a model on custom dataset," Transformers library familiarity — and treating those as sufficient qualification for production infrastructure ownership.

They are not the same job.

Training a model is a research skill. It requires deep understanding of loss functions, architecture choices, data pipeline quality, evaluation methodology, and the patience to run experiments that don't converge. It is hard, skilled, important work.

Operating a model in production is an infrastructure discipline. It requires capacity planning, GPU memory profiling under concurrent load, observability instrumentation that captures what matters for inference (token throughput, first-token latency, KV cache hit rate, queue depth), cost control systems with hard limits, blast radius analysis for every deployment, and a rollback path that actually works at 2:00 AM.

These overlap at the edges but they are not the same job. Hiring for one and expecting the other is how you get a five-figure GPU bill with no circuit breaker.

What Changes When You Operate at Scale

Here is what production ML infrastructure actually demands that doesn't appear on a research-oriented résumé:

GPU memory is not forgiving. A batch size that runs cleanly on a development P3 instance OOMs at 2x concurrent load on a P5 because nobody modeled the KV cache growth under real traffic patterns. The error is silent on the surface — the request fails, the client retries, the retry amplifies the load. A platform engineer who has done capacity planning knows to model this. A researcher who has done fine-tuning often doesn't.

Model weights are large artifacts with deployment complexity. A 70B parameter model in BF16 is 140GB. Getting that artifact from your model registry to your inference fleet reliably, with version control, with the ability to roll back without a full redeploy, with pre-warming on the new instances before you shift traffic — this is a software deployment problem, not a machine learning problem. The skills are in the infrastructure domain.

Inference is a latency-sensitive service. Everything your organization learned about p99 latency, tail latency management, connection pooling, load balancing, circuit breaking, and graceful degradation applies here. The model being impressive doesn't exempt the service from those requirements. Users experiencing 8-second p99 latency on an LLM endpoint don't care that the model architecture is elegant.

Network egress between regions is expensive at GPU scale. If your training cluster is in us-east-1 and your inference fleet is in us-west-2 and your model artifacts are large and you're pulling them frequently, someone is going to get a bill that surprises them. This is a cost architecture decision. It requires the same thinking as any other distributed system where data movement has cost.

Cost anomaly detection is non-negotiable. GPU compute is expensive at a rate that makes traditional cost anomaly thresholds meaningless. A virtual machine getting into a runaway state costs you maybe $50/hour. A GPU cluster in a bad state costs you $2,000/hour. Your cost alerting needs to reflect this. Most teams deploy their ML infrastructure with the same cost alerting thresholds they use for their web services, and discover the gap during an incident.

The "Boring in Production" Standard

The teams that will build durable AI infrastructure over the next few years aren't the ones with the fanciest models. They're the ones who took the research output seriously enough to subject it to the same operational rigor they apply to everything else.

Boring in production means:

The inference endpoint has been load-tested at 3x expected traffic and the degradation behavior is documented and understood
GPU utilization, memory pressure, queue depth, token throughput, and cost per request are all in the same observability dashboard
Model weight rollback takes less than ten minutes and has been practiced in a non-production environment
There is a hard cost ceiling per hour that triggers an automatic circuit breaker and pages the on-call engineer before the bill becomes a conversation with finance
The deployment pipeline is identical to every other service deployment pipeline — version controlled, reviewed, deployed through environments, observable from day one

None of this is specific to ML. All of it is infrastructure discipline applied to a new category of workload.

The GPU is expensive compute. The model weights are large artifacts. The inference service is a latency-sensitive API. The fundamentals didn't change. The job of the infrastructure engineer is to make sure the people who believe the fundamentals changed are not the ones with production access.

What Production-Grade MLOps Actually Requires

I've been building out the agentsre library with primitives that operationalize these patterns for AI/ML workloads — GPU cost circuit breakers, inference SLI definitions, model artifact governance gates. The design philosophy: every ML workload gets the same reliability treatment as every other production service. No exceptions for the model being impressive.

The primitives that have been most useful in production:

A GPU cost rate monitor that tracks cost-per-hour against a session budget and severs inference authority when the rate exceeds the threshold — before the bill, not after.

An inference SLI framework that defines first-token latency, token throughput, and queue depth as first-class SLIs with error budgets, so reliability of the inference service is measured the same way reliability of any other critical service is measured.

A model artifact governance gate that validates weight checksums, lineage provenance, and deployment window compliance before any model version reaches production traffic.

These aren't ML tools. They're SRE tools applied to ML workloads. That's the distinction that matters.

The Bottom Line for Hiring Teams

If you're building an AI infrastructure team and your entire interview loop is testing PyTorch knowledge and model architecture understanding, you are hiring for the research half of the job and leaving the operations half to chance.

The operations half is where the five-figure GPU bills live. It's where the 2:00 AM incidents live. It's where the systems either become boring and reliable or stay exciting and expensive.

Pair your ML talent with platform engineers who have lived through production incidents, who have oncall rotations, who have been paged because of something they shipped, who have a visceral understanding of blast radius and rollback and "what does this look like at 10x load."

Make them fight it out until the system is boring.

Boring in production is the highest compliment you can pay an AI platform.

What's the most expensive production lesson your team learned from ML infrastructure? Drop it in the comments the more specific, the more useful for everyone trying to avoid the same mistake.

Tags: #MLOps #AIInfrastructure #SRE #PlatformEngineering #CloudNative #AIOps #DevOps #GPU #LLMOps

Your AI Agent Has an IAM Role. That's Not a Feature It's an Uncontrolled Blast Radius.

Ajay Devineni — Tue, 07 Jul 2026 01:53:59 +0000

Your AI Agent Has an IAM Role. That's Not a Feature It's an Uncontrolled Blast Radius.

Why traditional identity models fail non-deterministic systems, and what enforcement actually looks like at the platform layer

At 2:14 AM on a Friday, an agentic remediation system I was responsible for made fourteen sequential API calls to AWS in eleven seconds.

The agent had detected an anomalous CloudWatch alarm. It retrieved a relevant runbook. It synthesized a remediation plan. Then it executed that plan — and each execution step produced a new signal the agent interpreted as requiring another action. Fourteen calls. Eleven seconds. No circuit breaker.

The IAM role didn't care. Every call was authenticated. Every call was authorized. The cloud provider executes what it receives. It has no concept of whether the call was generated by deterministic application code or a language model in a hallucinated reasoning loop.

We got lucky that night. The blast radius was contained to a single ECS cluster not serving production traffic. But the architectural failure was real, and it had nothing to do with the quality of the agent's reasoning.

The Category Error We're Building On

There's a widely-shared mental model in teams deploying infrastructure agents right now: if the model reasons well, the system is safe. Improve the prompt. Add a system message. Constrain the reasoning with better instructions.

This is the wrong level of enforcement.

Traditional IAM was designed for deterministic code. When you write an application that calls ec2:TerminateInstances, you know exactly when that call will be made, with what parameters, under what conditions. The call path is static. The IAM policy is a guard on a known, bounded behavior.

A language model doesn't have a call path. It has a reasoning trace that produces a call path at runtime one that changes based on the prompt, the context window, the retrieved documents, the tool call history, and whatever the model decides is the next logical step. The IAM role the agent holds doesn't know any of that. It knows the API call that arrived. It executes.

This means that when a prompt injection payload appears in a retrieved document a log file, a ticket description, a runbook that's been tampered with the execution authority the agent holds becomes the blast radius of that injection. The same IAM role that handles legitimate remediation handles the injected command. The cloud doesn't distinguish.

Prompt engineering is not a security control at this layer. It is a probabilistic influence on model behavior. Probabilistic is not deterministic. Infrastructure security requires deterministic enforcement.

The Three Failure Modes in Practice

Reasoning loops that become API storms

A model gets into a state where each tool call result is interpreted as evidence that another tool call is needed. Without a runtime circuit breaker, this generates hundreds of API calls before a human notices rate limit exhaustion on critical APIs, unbudgeted cost spikes, and potentially self-inflicted service degradation on the infrastructure the agent was supposed to protect.

Prompt injection through context retrieval

An agent retrieving context from a knowledge base, ticket system, or log aggregator is reading content that may have been authored by an adversary. A log line containing an instruction to scale all ASGs to maximum capacity is not a theoretical threat. Agents that pass retrieved content directly into their reasoning context without sanitization are vulnerable. The IAM role has no "was this call generated by an injection payload?" check.

Authority conflation

The agent finds a valid path to resolve an alert and executes it. The path has side effects — deleting a deployment that was intentional, terminating an instance running a scheduled batch job, modifying a security group that breaks a dependency. The agent had authority. The action was within scope. Nobody had established that "synthesizing a remediation plan" and "executing against live infrastructure" are two separate authority levels that should be enforced separately.

What Deterministic Enforcement Actually Looks Like

The principle: an AI agent should be allowed to reason, synthesize, and recommend. The decision to execute passes through a deterministic policy engine with no LLM in the critical path.

This isn't new. It's how safety-critical systems have always been designed. The novel problem is that most teams deploying infrastructure agents haven't made this separation explicit in their architecture.

Separation of Intent and Action

The agent produces a proposed action: a structured object describing what it wants to do, to what resource, with what parameters. That object passes to a policy engine — a deterministic validator with no LLM involvement — that checks: Is this resource tagged production? Is this action class permitted at the current error budget level? Has this resource been modified in the last N minutes?

Approved: execution proceeds. Rejected: the agent receives a structured refusal, the attempt is logged, and the agent has no path to self-authorize beyond what the policy engine allows.

Runtime anomaly detection with automatic authority severance

Tool-calling behavior is tracked at runtime: calls per minute, cost per session, resource types touched, action class frequency. When a metric crosses a threshold — more than twenty API calls in sixty seconds, more than ten distinct resource modifications in a session, cost exceeding a per-session budget the circuit breaker severs tool calling authority for the session and pages the on-call engineer.

The agent doesn't get a warning. Its tool access is revoked at the infrastructure layer. It can continue reasoning and producing recommendations. It cannot execute.

Isolated runtime containment

Any agent executing commands against infrastructure should run in a short-lived sandbox with scoped credentials, network egress restrictions, and a maximum session lifetime. When the session ends, the credentials expire. The blast radius of any single session is structurally bounded by the sandbox's resource scope not by the agent's self-restraint.

The Architecture Change That Mattered

After the fourteen-call incident, we made three changes. None of them touched the model.

A runtime session monitor tracked API calls per sixty-second window per agent session. Above fifteen calls, the session's IAM role was suspended via a resource-based policy update and an alert fired. The agent could still reason. It could not execute.

A pre-execution policy gate sat between the agent's proposed action and any live API call. The gate checked resource tags, action class, and a cooldown registry preventing the same resource from being modified more than once per ten minutes. Every gate decision approved or rejected went to CloudWatch with full context.

Agent execution moved into short-lived ECS tasks with session scoped IAM roles assumed at task start and automatically expired at task end. No persistent credentials. No carry-over between sessions.

The agent's reasoning didn't change. The architecture around it changed. That distinction is the entire point: the safety guarantees needed for autonomous infrastructure agents cannot be delegated to the model. They have to be built into the system.

The Question for Platform Teams

If you're deploying autonomous agents that touch live infrastructure, the design review question isn't "does the model reason well enough to be safe?"

The question is: if the model reasons incorrectly — bad prompt, stale context, prompt injection payload, hallucinated tool-calling loop — does your platform architecture bound the blast radius, or does the agent's IAM role determine it?

If the answer is "the IAM role determines it," you have an uncontrolled blast radius.

The controls exist. The architecture patterns are established. The question is whether we build them before the incident, or after.

How is your team decoupling the agent's reasoning layer from your cloud infrastructure permission boundaries? The more specific the implementation detail, the more useful for everyone building in this space.

RAG vs. MCP Is the Wrong Question Here's the Right One

Ajay Devineni — Sun, 05 Jul 2026 20:35:02 +0000

Why treating this as an architectural tradeoff is setting your production agents up to fail

I watched a team's agentic remediation system act on a runbook that was fourteen months stale.

The RAG retrieval worked correctly. The document ranked highest by cosine similarity. The agent interpreted that retrieval as authorization to execute. The live environment was meaningfully different from what the document described. The remediation made things worse.

Nobody had explicitly defined where knowing ends and doing begins.

That is the actual production risk hiding inside the "RAG vs. MCP" framing. Not which technology you pick. Whether you have drawn a hard line between your agent's knowledge layer and its execution layer — and whether that line is enforced in code rather than trusted in assumptions.

The Category Error

RAG and MCP don't compete. They answer completely different questions.

RAG answers: what does the agent know?

Runbook context. Historical incident patterns. Service documentation. Architecture decision records. Compliance policy text. Everything the agent needs to reason about a situation before it touches anything.

MCP answers: what can the agent do?

Live metric fetching. System state queries. Remediation execution. API calls against real infrastructure. All the actions that have a blast radius.

These two layers have fundamentally different risk profiles. A RAG query that returns a stale document is a reasoning error. An MCP call that executes against a live system based on a stale document is an incident.

The conflation of these two layers treating "I retrieved something relevant" as equivalent to "I have permission to execute it" removes the safety layer that should sit between them.

What the Conflation Looks Like in Practice

Pattern 1: RAG-to-execution without a gate

An agent retrieves documentation describing how to handle high CPU utilization on a service. The document is from before a major infrastructure migration. The agent, having retrieved relevant context, proceeds directly to execution — restarting pods that no longer exist in the topology described, or modifying configurations that have since moved to a different management plane.

The retrieval confidence was high. The execution authority was never scoped. There was no check between "I know something relevant" and "I will now act on it."

Pattern 2: Knowledge queries routed through execution tooling

The inverse failure. Teams that don't design the boundary explicitly tend to route everything through the heaviest available tool. A query that only needs to retrieve context ends up going through MCP tooling with live-system access — adding latency, consuming rate-limited API quota, and introducing non-determinism into what should be a deterministic knowledge lookup.

The agent has more execution authority than it needs for the task. This is a blast-radius problem waiting for the right failure condition.

The Boundary as a Reliability Control

In SRE practice, we talk about blast radius as a design constraint. You scope the potential damage of any failure before it happens, not after. Circuit breakers, canary deployments, feature flags, staged rollouts — all exist to keep the failure surface bounded.

The RAG/MCP boundary is the same class of control applied to agentic systems.

The design question isn't "RAG or MCP?" It's: at what point in this agent's reasoning flow does read-only context become a candidate for execution authority, and what explicit gate exists at that transition?

That gate needs to be deterministic. It needs to be auditable. And it needs to exist in code, not in the assumption that the model will reason correctly about the distinction every time.

What an Explicit Boundary Looks Like

A well-designed agentic system for infrastructure operations has three distinct layers with explicit handoff points.

Layer 1 — Situational awareness (RAG)

The agent retrieves runbook context, historical incident data, service topology, and policy constraints. This layer is read-only with no execution authority. The output is a structured context package: what the agent knows about the situation, what precedents exist, what the operational constraints are.

Layer 2 — Decision gate

An explicit check before any execution authority is granted. This is where you validate: Is the retrieved context fresh enough to act on? Does the proposed action fall within the pre-approved blast radius? Has a human review been configured for this action class? Is the error budget healthy enough to absorb a remediation attempt?

This layer is where most agentic systems have nothing. The model reasons through it implicitly. That implicit reasoning is not a reliability control.

Layer 3 — Execution (MCP)

Scoped tool calls with explicit permission boundaries. The agent can fetch live metrics, read current system state, and execute approved remediation actions. The scope of what's available in this layer is determined by what passed through Layer 2 — not by what the model decides is appropriate at runtime.

The War Story That Sharpened This for Me

A service I was responsible for had an agentic component handling routine disk pressure events. The pattern: detect high disk utilization → retrieve cleanup runbook → execute cleanup steps.

For eight months it worked correctly. Then we rotated to a new storage backend. The cleanup runbook in the knowledge base hadn't been updated.

The agent retrieved the old runbook with high confidence — service name matched, symptom description matched, top result by similarity score. It executed against paths that no longer existed as described. The cleanup didn't make things catastrophically worse, but it also didn't fix anything, and it consumed enough API quota that the legitimate remediation attempt that followed hit rate limits.

The failure wasn't in the retrieval. The failure was that there was no freshness check between retrieval and execution. No gate asking: is this document current enough to act on?

After that incident we added three architectural controls:

A document freshness gate on any retrieved context informing an execution decision. If document age exceeded a configured threshold for the action class, the agent escalated to a human.

A scope limiter on every MCP tool call, tying the available action set to the specific remediation class that was approved — not to everything the agent had tool access to.

A dry-run mode for remediation actions above a configured blast-radius threshold, staging planned changes as a human-reviewable diff before executing.

None of these are model improvements. All of them are architectural controls. The model's reasoning didn't change. What changed was the system around it.

The Question Worth Asking Your Team

If you're building or operating infrastructure agents right now, the design review question isn't "are we using RAG or MCP?" Both answers are probably yes.

The question is: is the boundary between your knowledge retrieval layer and your execution layer explicitly enforced in code, or is it implicitly trusted in the model's reasoning?

If the answer is "the model handles that," you don't have a boundary. You have an assumption.

Assumptions are not reliability controls.

Drop your approach in the comments — how are you enforcing the knowing/doing boundary in your agentic systems? The more specific the implementation detail, the more useful it is for everyone building in this space.

Tags: #AgenticAI #MCP #RAG #SRE #PlatformEngineering #CloudNative #AIOps #Observability