Pradeep Kandepaneni

Posted on Jul 29

Building a Production-Safe AI Remediation Firewall for Amazon EKS

#aiops #devops #stepfunctions #cloudwatch

Originally published on the AWS Builder Center:

AWS Builder Center

Connect with builders who understand your journey. Share solutions, influence AWS product development, and access useful content that accelerates your growth. Your community starts here.

builder.aws.com

AI can reduce the time required to investigate incidents, correlate telemetry, and identify probable root causes. But giving an AI system unrestricted production access creates a new reliability risk: the remediation platform itself can become the cause of the next outage.

This article presents an AI Remediation Firewall for workloads running on Amazon Elastic Kubernetes Service (Amazon EKS).

The design follows one principle:

AI proposes. Deterministic policy authorises. Controlled automation executes. Independent telemetry verifies.

The architecture uses:

AWS DevOps Agent for investigation and root-cause analysis
A structured remediation proposal as the handoff contract
AWS Step Functions for workflow control
Amazon Verified Permissions for deterministic authorisation
AWS Systems Manager Automation for controlled execution
Amazon CloudWatch Application Signals and CloudWatch Synthetics for independent verification
Kubernetes RBAC and IAM to restrict blast radius
Progressive delivery and rollback to contain failed changes

The objective is not unrestricted autonomous operations. It is bounded autonomy.

The production incident

Consider a digital commerce platform running on Amazon EKS.

The application contains:

API gateway
Checkout service
Inventory service
Payment service
Pricing service
Amazon Aurora PostgreSQL
Amazon ElastiCache for Redis
Amazon SQS
Application Load Balancer
CloudWatch Application Signals
CloudWatch Synthetics

A deployment changes the checkout configuration:

database:
  poolSize: 80
  connectionTimeoutMilliseconds: 500

retry:
  maximumAttempts: 8
  initialBackoffMilliseconds: 25

The previous configuration was:

database:
  poolSize: 30
  connectionTimeoutMilliseconds: 2000

retry:
  maximumAttempts: 3
  initialBackoffMilliseconds: 250

Within minutes:

Checkout p99 latency increases from about 220 ms to more than four seconds
HTTP 5xx errors increase
Aurora connections approach the configured maximum
SQS queue depth grows
HPA activity increases
CPU usage rises across checkout and payment services
Pods remain Running
Liveness probes continue to pass

This is a difficult incident because infrastructure health looks acceptable at a superficial level.

Restarting pods provides only temporary relief. Increasing replicas creates more database connections and makes the problem worse.

AWS DevOps Agent investigates:

CloudWatch metrics and logs
Application Signals topology
Kubernetes events
Deployment history
CloudTrail configuration changes
Aurora metrics
SQS metrics
Approved runbooks and previous incidents

It identifies the latest configuration change as the probable trigger and proposes:

Roll back the checkout service to the previous known-good revision.

That recommendation may be correct, but it is not yet authorised.

Why an AI recommendation must be treated as untrusted input

An AI-generated remediation can be technically plausible and still be unsafe because:

The root-cause analysis may be incomplete
Multiple deployments may be active
The target may have changed after the investigation started
The rollback revision may contain a security defect
The proposed blast radius may be too large
The service may already be recovering
The action may affect a critical dependency
The rollback artefact may no longer exist
The recommendation may be based on stale telemetry

The AI system must not decide:

Its own permissions
Whether its evidence is sufficient
Whether the blast radius is acceptable
Whether human approval is required
Whether the remediation succeeded

Those decisions belong outside the AI reasoning boundary.

Reference architecture

CloudWatch SLO alarm or incident alert
                 |
                 v
          AWS DevOps Agent
                 |
       Investigation and RCA
                 |
                 v
      Remediation Proposal Adapter
                 |
                 v
          Amazon EventBridge
                 |
                 v
          AWS Step Functions
                 |
       +---------+----------+
       |                    |
       v                    v
Live-context collector   Schema validator
       |                    |
       +---------+----------+
                 |
                 v
       Deterministic risk engine
                 |
                 v
    Amazon Verified Permissions
                 |
     +-----------+-----------+
     |           |           |
     v           v           v
Auto-execute  Approval      Deny
     |           |
     +-----+-----+
           |
           v
AWS Systems Manager Automation
           |
           v
Restricted EKS executor
           |
           v
Canary or bounded rollout
           |
           v
Application Signals and Synthetics
           |
     +-----+------+
     |            |
     v            v
 Promote       Roll back

The trust boundaries are:

Investigation boundary
AI gathers evidence and proposes an action.
Authorisation boundary
Deterministic logic decides whether that action is allowed.
Execution boundary
A restricted runbook performs one allowlisted operation.
Verification boundary
Independent telemetry decides whether recovery occurred.

1. Keep diagnostic access read-only

The investigation layer should be able to:

Read CloudWatch metrics and logs
Read Kubernetes events
Read deployment metadata
Read CloudTrail changes
Read service topology
Read approved incident history
Invoke narrow diagnostic tools

It should not have broad write access to:

Kubernetes resources
IAM
Security groups
Databases
Secrets
CI/CD pipelines
DNS

Expose bounded tools such as:

get_kubernetes_events(namespace, workload)
get_deployment_revision(namespace, deployment)
collect_eks_node_diagnostics(instance_id)
query_application_errors(service, start_time, end_time)
compare_configuration(current_revision, previous_revision)

Do not expose:

execute_shell(command)

A generic shell endpoint turns a controlled diagnostic integration into remote code execution.

2. Require a structured remediation proposal

Do not send free-form AI text to an executor.

Use a versioned contract:

{
  "schemaVersion": "1.0",
  "proposalId": "rp-01JZ8A17R2K5",
  "incidentId": "inc-checkout-0017",
  "createdAt": "2026-07-28T20:12:31Z",
  "expiresAt": "2026-07-28T20:17:31Z",
  "target": {
    "environment": "production",
    "cluster": "commerce-prod",
    "region": "us-east-1",
    "namespace": "checkout",
    "resourceType": "ArgoRollout",
    "resourceName": "checkout-api"
  },
  "action": {
    "type": "ROLLBACK_DEPLOYMENT",
    "parameters": {
      "targetRevision": 41
    },
    "requestedBlastRadiusPercent": 5
  },
  "evidence": [
    {
      "source": "APPLICATION_SIGNALS",
      "type": "LATENCY_INCREASE"
    },
    {
      "source": "CLOUDTRAIL",
      "type": "RECENT_CHANGE"
    },
    {
      "source": "KUBERNETES",
      "type": "CONFIGURATION_CHANGE"
    },
    {
      "source": "AURORA",
      "type": "RESOURCE_SATURATION"
    }
  ],
  "rollback": {
    "available": true,
    "knownGoodRevision": 41
  },
  "agentAssessment": {
    "confidencePercent": 92,
    "summary": "Revision 42 introduced retry amplification"
  }
}

Reject proposals containing:

Unknown actions
Wildcard targets
Free-form commands
Missing rollback data
Expired timestamps
Unregistered resources
Unknown revisions
Excessive blast radius
Insufficient evidence

The proposal contains intent, not executable commands.

Accept:

{
  "action": {
    "type": "ROLLBACK_DEPLOYMENT",
    "parameters": {
      "targetRevision": 41
    }
  }
}

Reject:

{
  "command": "kubectl delete pods --all --all-namespaces"
}

3. Separate action risk from incident urgency

A critical incident does not make a dangerous remediation safe.

Calculate two independent values.

Action risk

Action Risk =
    Action Severity       × 0.25
  + Resource Criticality  × 0.20
  + Blast Radius          × 0.20
  + Evidence Uncertainty  × 0.15
  + Change Collision      × 0.10
  + Rollback Weakness     × 0.10

Example action severity:

Action	Severity
Restart one stateless pod	10
Scale within approved limits	20
Progressive rollback	30
Modify HPA policy	45
Change database parameters	70
Modify routing	85
Modify IAM	95
Delete data	100

Example resource criticality:

Resource	Criticality
Development service	10
Internal production service	35
Customer-facing API	70
Authentication service	90
Payment platform	100

Incident urgency

Incident Urgency =
    SLO Burn Rate      × 0.40
  + Customer Impact    × 0.35
  + Incident Duration  × 0.25

Urgency may shorten approval time or escalate responders. It must not lower the action-risk threshold.

4. Do not trust AI confidence directly

The control plane should calculate validated confidence from evidence quality.

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any

ALLOWLISTED_ACTIONS = {
    "RESTART_SINGLE_POD",
    "SCALE_WITHIN_LIMIT",
    "ROLLBACK_DEPLOYMENT",
}

@dataclass(frozen=True)
class RiskResult:
    risk_score: int
    validated_confidence: int
    route: str
    reasons: list[str]

def clamp(value: int) -> int:
    return max(0, min(100, value))

def validated_confidence(proposal: dict[str, Any]) -> int:
    evidence = proposal.get("evidence", [])
    sources = {item.get("source") for item in evidence if item.get("source")}
    types = {item.get("type") for item in evidence if item.get("type")}

    agent_confidence = clamp(
        int(proposal.get("agentAssessment", {}).get("confidencePercent", 0))
    )

    has_change = "RECENT_CHANGE" in types
    has_telemetry = bool(
        {"LATENCY_INCREASE", "ERROR_RATE_INCREASE", "RESOURCE_SATURATION"}
        & types
    )

    if len(sources) >= 3 and has_change and has_telemetry:
        cap = 95
    elif len(sources) >= 2 and has_telemetry:
        cap = 75
    else:
        cap = 50

    return min(agent_confidence, cap)

def calculate_action_risk(
    proposal: dict[str, Any],
    context: dict[str, Any],
) -> RiskResult:
    action = proposal["action"]["type"]

    if action not in ALLOWLISTED_ACTIONS:
        return RiskResult(100, 0, "DENY", ["Action is not allowlisted"])

    expires_at = datetime.fromisoformat(
        proposal["expiresAt"].replace("Z", "+00:00")
    )

    if datetime.now(timezone.utc) >= expires_at:
        return RiskResult(100, 0, "DENY", ["Proposal expired"])

    confidence = validated_confidence(proposal)
    uncertainty = 100 - confidence

    risk = round(
        clamp(context["actionSeverity"]) * 0.25
        + clamp(context["resourceCriticality"]) * 0.20
        + clamp(context["blastRadiusPercent"]) * 0.20
        + clamp(uncertainty) * 0.15
        + (100 if context["concurrentChange"] else 0) * 0.10
        + (0 if context["rollbackAvailable"] else 100) * 0.10
    )

    reasons = []

    if not context["rollbackAvailable"]:
        risk = max(risk, 70)
        reasons.append("No verified rollback point")

    if context["concurrentChange"]:
        risk = max(risk, 65)
        reasons.append("Concurrent change detected")

    if context["blastRadiusPercent"] > 25:
        risk = max(risk, 65)
        reasons.append("Blast radius exceeds 25 percent")

    if confidence < 70:
        risk = max(risk, 61)
        reasons.append("Evidence confidence below threshold")

    if risk <= 25 and confidence >= 90:
        route = "AUTO_EXECUTE"
    elif risk <= 60:
        route = "REQUIRE_APPROVAL"
    else:
        route = "DENY"

    return RiskResult(clamp(risk), confidence, route, reasons)

The weights are organisational policy, not universal values.

The critical properties are:

Inputs are bounded
Logic is deterministic
Algorithms are versioned
Decisions are reproducible
Every factor is recorded
AI cannot change thresholds
Failure defaults to denial

5. Refresh production context before authorisation

Do not authorise using only the data captured during investigation.

Immediately before execution, verify:

The incident is still active
The target revision is still deployed
No other production change is running
The previous revision still exists
The previous revision is security-compliant
The workload has not already recovered
The target resource still maps to the same owner and criticality
The SLO is still burning
The proposal has not expired

This prevents time-of-check/time-of-use failures.

A five-minute proposal expiration is a reasonable starting point for autonomous actions.

6. Authorise with Amazon Verified Permissions

Model the remediation controller as an application:

Principal: remediation orchestrator
Action: auto-execute or request-approval
Resource: production workload
Context: risk, confidence, blast radius, rollback status, action type

Evaluate AutoExecute first. If denied, evaluate RequestApproval. If both are denied, block the proposal.

Automatic execution policy:

permit (
    principal == RemediationControl::Service::"orchestrator",
    action == RemediationControl::Action::"AutoExecute",
    resource
)
when {
    context.riskScore <= 25 &&
    context.validatedConfidence >= 90 &&
    context.blastRadiusPercent <= 10 &&
    context.rollbackAvailable &&
    !context.concurrentChange &&
    (
        context.actionType == "RESTART_SINGLE_POD" ||
        context.actionType == "SCALE_WITHIN_LIMIT" ||
        context.actionType == "ROLLBACK_DEPLOYMENT"
    )
};

Approval policy:

permit (
    principal == RemediationControl::Service::"orchestrator",
    action == RemediationControl::Action::"RequestApproval",
    resource
)
when {
    context.riskScore <= 60 &&
    context.validatedConfidence >= 70 &&
    context.blastRadiusPercent <= 25 &&
    context.rollbackAvailable &&
    !context.concurrentChange
};

Explicitly forbidden actions:

forbid (
    principal,
    action,
    resource
)
when {
    context.actionType == "MODIFY_IAM" ||
    context.actionType == "DELETE_DATA" ||
    context.actionType == "DISABLE_AUDIT_LOGGING" ||
    context.actionType == "OPEN_PUBLIC_NETWORK_ACCESS"
};

The authorisation request can be submitted from Lambda:

import boto3

client = boto3.client("verifiedpermissions")

def is_authorized(
    policy_store_id: str,
    route_action: str,
    workload_id: str,
    context: dict,
) -> bool:
    response = client.is_authorized(
        policyStoreId=policy_store_id,
        principal={
            "entityType": "RemediationControl::Service",
            "entityId": "orchestrator",
        },
        action={
            "actionType": "RemediationControl::Action",
            "actionId": route_action,
        },
        resource={
            "entityType": "RemediationControl::Workload",
            "entityId": workload_id,
        },
        context={
            "contextMap": {
                "riskScore": {"long": context["riskScore"]},
                "validatedConfidence": {
                    "long": context["validatedConfidence"]
                },
                "blastRadiusPercent": {
                    "long": context["blastRadiusPercent"]
                },
                "rollbackAvailable": {
                    "boolean": context["rollbackAvailable"]
                },
                "concurrentChange": {
                    "boolean": context["concurrentChange"]
                },
                "actionType": {"string": context["actionType"]},
            }
        },
    )

    return response["decision"] == "ALLOW"

Store with every decision:

Risk-engine version
Policy-store ID
Policy version
Risk score
Validated confidence
Decision route
Determining policy IDs

7. Orchestrate with AWS Step Functions

The state machine should coordinate:

Proposal validation
Expiration check
Live-context collection
Risk calculation
Verified Permissions evaluation
Human approval where required
Runbook execution
Canary verification
Promotion or rollback
Audit completion

A simplified workflow:

{
  "StartAt": "ValidateProposal",
  "States": {
    "ValidateProposal": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "CollectLiveContext"
    },
    "CollectLiveContext": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "CalculateRisk"
    },
    "CalculateRisk": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "AuthoriseRoute"
    },
    "AuthoriseRoute": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "RouteDecision"
    },
    "RouteDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.decision.route",
          "StringEquals": "AUTO_EXECUTE",
          "Next": "ExecuteRunbook"
        },
        {
          "Variable": "$.decision.route",
          "StringEquals": "REQUIRE_APPROVAL",
          "Next": "ApprovalRunbook"
        }
      ],
      "Default": "Denied"
    },
    "ExecuteRunbook": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Next": "VerifyCanary"
    },
    "ApprovalRunbook": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Next": "VerifyCanary"
    },
    "VerifyCanary": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "VerificationDecision"
    },
    "VerificationDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.verification.passed",
          "BooleanEquals": true,
          "Next": "Promote"
        }
      ],
      "Default": "Rollback"
    },
    "Promote": {
      "Type": "Succeed"
    },
    "Rollback": {
      "Type": "Fail"
    },
    "Denied": {
      "Type": "Succeed"
    }
  }
}

Production implementations also need:

Retries with bounded backoff
Catch handlers
Idempotency
Dead-letter handling
Timeouts
Duplicate suppression
Per-workload locking
Compensating actions
Audit-write failure handling

8. Execute only allowlisted runbooks

Map each action to a versioned Systems Manager runbook.

RUNBOOK_CATALOG = {
    "RESTART_SINGLE_POD": {
        "document": "AIOps-RestartSinglePod-v3",
        "maximumBlastRadius": 10,
    },
    "SCALE_WITHIN_LIMIT": {
        "document": "AIOps-ScaleDeployment-v4",
        "maximumBlastRadius": 20,
    },
    "ROLLBACK_DEPLOYMENT": {
        "document": "AIOps-RollbackDeployment-v7",
        "maximumBlastRadius": 25,
    },
}

The AI selects intent from the catalogue. It does not generate commands.

Example runbook:

description: Roll back an approved EKS workload
schemaVersion: '0.3'

assumeRole: '{{ AutomationAssumeRole }}'

parameters:
  AutomationAssumeRole:
    type: AWS::IAM::Role::Arn

  ExecutorFunctionName:
    type: String

  ClusterName:
    type: String

  Namespace:
    type: String

  WorkloadName:
    type: String

  TargetRevision:
    type: Integer

  TrafficPercent:
    type: Integer
    allowedValues:
      - 5
      - 10
      - 25

  ProposalId:
    type: String

mainSteps:
  - name: ExecuteRestrictedRollback
    action: aws:invokeLambdaFunction
    timeoutSeconds: 120
    maxAttempts: 1
    onFailure: Abort
    inputs:
      FunctionName: '{{ ExecutorFunctionName }}'
      InvocationType: RequestResponse
      Payload: >
        {
          "operation": "ROLLBACK_DEPLOYMENT",
          "cluster": "{{ ClusterName }}",
          "namespace": "{{ Namespace }}",
          "workload": "{{ WorkloadName }}",
          "targetRevision": {{ TargetRevision }},
          "trafficPercent": {{ TrafficPercent }},
          "proposalId": "{{ ProposalId }}"
        }

For medium-risk actions, add an aws:approve step before invoking the child runbook.

High-risk actions such as IAM changes, data deletion, network exposure, or disabling audit controls should remain outside the autonomous remediation catalogue.

9. Restrict EKS access

Use a dedicated IAM role and EKS access entry for the executor.

Do not grant cluster-admin.

Map the role to a Kubernetes group:

aiops-remediation-executor

Namespace-scoped Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: aiops-remediation-executor
  namespace: checkout
rules:
  - apiGroups:
      - apps
      - argoproj.io
    resources:
      - deployments
      - rollouts
    verbs:
      - get
      - list
      - watch
      - patch
      - update

  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch

RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: aiops-remediation-executor
  namespace: checkout
subjects:
  - kind: Group
    name: aiops-remediation-executor
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: aiops-remediation-executor
  apiGroup: rbac.authorization.k8s.io

The executor must not be able to:

Read Secrets
Modify ClusterRoles
Create privileged pods
Execute commands in arbitrary containers
Modify other namespaces
Delete namespaces
Change its own IAM access
Modify EKS access entries

For stronger isolation, use separate executor roles for each action category.

10. Limit blast radius with progressive delivery

An approved action should not immediately affect every replica.

Start with:

Initial traffic: 5%
Observation window: 5 minutes
Required successful synthetic runs: 3
Maximum 5xx rate: 1%
Maximum p99 latency: 750 ms
Required business validation success: 100%

Stop promotion when:

Error rate rises
Latency exceeds the threshold
New Kubernetes errors appear
Database pressure worsens
SLO burn rate increases
Business validation fails
Another production change begins

Promote in stages:

5% → 25% → 50% → 100%

Each stage requires independent verification.

11. Verify business behaviour, not only infrastructure health

A successful rollout does not prove recovery.

Pods may be healthy and return HTTP 200 while producing an incorrect checkout total.

CloudWatch Synthetics should validate both technical and business invariants.

const https = require("https");
const log = require("SyntheticsLogger");

const API_HOST = process.env.API_HOST;

function sendRequest(options, body) {
  return new Promise((resolve, reject) => {
    const request = https.request(options, response => {
      let data = "";

      response.on("data", chunk => {
        data += chunk;
      });

      response.on("end", () => {
        resolve({
          statusCode: response.statusCode,
          body: data
        });
      });
    });

    request.on("error", reject);
    request.write(body);
    request.end();
  });
}

exports.handler = async () => {
  const body = JSON.stringify({
    customerId: "synthetic-customer",
    items: [
      {
        sku: "SYNTHETIC-SKU-001",
        quantity: 2
      }
    ]
  });

  const options = {
    hostname: API_HOST,
    method: "POST",
    path: "/api/v1/checkout/quote",
    headers: {
      "Content-Type": "application/json",
      "Content-Length": Buffer.byteLength(body),
      "X-Synthetic-Transaction": "true"
    },
    timeout: 5000
  };

  const started = Date.now();
  const response = await sendRequest(options, body);
  const duration = Date.now() - started;

  if (response.statusCode !== 200) {
    throw new Error(`Unexpected HTTP ${response.statusCode}`);
  }

  if (duration > 750) {
    throw new Error(`Latency exceeded threshold: ${duration} ms`);
  }

  const payload = JSON.parse(response.body);

  const subtotal = payload.lines.reduce(
    (total, line) => total + line.unitPrice * line.quantity,
    0
  );

  const expected = subtotal + payload.tax + payload.shipping;

  if (Math.abs(payload.total - expected) > 0.01) {
    throw new Error(
      `Business invariant failed. Expected ${expected}, received ${payload.total}`
    );
  }

  if (payload.currency !== "USD") {
    throw new Error(`Unexpected currency: ${payload.currency}`);
  }

  log.info(`Synthetic checkout passed in ${duration} ms`);

  return {
    passed: true,
    durationMilliseconds: duration
  };
};

Synthetic transactions must not create real payments, shipments, inventory consumption, or customer notifications.

Use:

Reserved synthetic identities
Test inventory
Payment-provider test mode
Cleanup workflows
Explicit synthetic headers
Analytics filters

12. Verification contract

The verification layer should query authoritative telemetry directly.

{
  "proposalId": "rp-01JZ8A17R2K5",
  "observationWindowSeconds": 300,
  "requirements": {
    "minimumSuccessfulSyntheticRuns": 3,
    "maximumErrorRatePercent": 1,
    "maximumP99LatencyMilliseconds": 750,
    "maximumDatabaseConnectionUtilisationPercent": 75,
    "maximumSloBurnRate": 2,
    "minimumHealthyReplicaPercent": 100
  }
}

Example result:

{
  "passed": true,
  "observations": {
    "successfulSyntheticRuns": 5,
    "errorRatePercent": 0.4,
    "p99LatencyMilliseconds": 420,
    "databaseConnectionUtilisationPercent": 61,
    "sloBurnRate": 0.8,
    "healthyReplicaPercent": 100
  }
}

Promotion requires every mandatory condition to pass.

Do not average away a critical business failure.

End-to-end execution

Step 1: Detect

Application Signals detects an elevated SLO burn rate and triggers an investigation.

Step 2: Investigate

AWS DevOps Agent correlates telemetry, configuration history, deployment metadata, and service dependencies.

Step 3: Propose

The proposal adapter creates:

Action: ROLLBACK_DEPLOYMENT
Target revision: 41
Initial traffic: 5%
Agent confidence: 92%
Rollback available: Yes

Step 4: Refresh context

The control plane confirms:

Revision 42 is still active
Revision 41 exists
No concurrent deployment is running
The incident is still active
The workload matches the inventory record

Step 5: Calculate risk

Factor	Value
Action severity	30
Resource criticality	70
Blast radius	5
Evidence uncertainty	8
Change collision	0
Rollback weakness	0

30 × 0.25 = 7.50
70 × 0.20 = 14.00
 5 × 0.20 = 1.00
 8 × 0.15 = 1.20
--------------------
Risk score = 23.70, rounded to 24

Validated confidence is 92%.

Step 6: Authorise

Verified Permissions allows automatic execution because:

Risk is at or below 25
Confidence is at least 90
Blast radius is at or below 10%
Rollback exists
No concurrent change is active
The action is allowlisted

Step 7: Execute

Systems Manager invokes the restricted rollback executor.

The executor can modify only:

commerce-prod
└── checkout namespace
    └── checkout-api

Step 8: Verify

Five percent of traffic moves to revision 41.

CloudWatch Synthetics validates the checkout path. Application Signals validates latency, errors, and SLO burn rate. Database and queue metrics confirm recovery.

Step 9: Promote or roll back

If every gate passes, traffic moves progressively to 100%.

If any gate fails, the workflow restores the prior state and marks the remediation unsuccessful.

Step 10: Prevent recurrence

The incident generates preventive work:

Add retry budgets
Enforce exponential backoff
Validate connection-pool settings
Add configuration contract tests
Add load tests
Add deployment policies
Improve SLO coverage

A platform that repeatedly fixes the same incident without creating preventive engineering work is only automating toil.

Failure scenarios to test

1. Retry amplification

Inject excessive retries, minimal backoff, and aggressive database pool settings.

Expected result:

AI correlates deployment and database saturation
Scaling is not selected as the first response
Rollback is proposed
Connection pressure decreases after remediation

2. Silent network failure

Inject an invalid route or packet-filtering rule.

Expected result:

Pods may remain running
Service-to-service calls fail
Node diagnostics provide evidence
The system does not restart every workload blindly

3. HPA thrashing

Configure an aggressive target with insufficient stabilisation.

Expected result:

Oscillation is detected
Unbounded scaling is denied
HPA changes require approval
Verification uses a longer window

4. Partial secret rotation

Rotate credentials for only part of the workload.

Expected result:

Pod-level error distribution reveals the split
Secret changes remain outside autonomous execution
A predefined security runbook is used

5. Phantom success

Return HTTP 200 with an incorrect checkout total.

Expected result:

Infrastructure checks pass
Business synthetic fails
Promotion stops
Automatic rollback begins

6. Concurrent deployment

Start another deployment after the proposal is generated.

Expected result:

Context refresh detects the collision
Automatic execution is denied
The proposal becomes stale

7. Missing rollback artefact

Delete the previous image or revision.

Expected result:

Rollback weakness increases
Risk exceeds the threshold
The proposal is denied

Security controls

IAM separation

Use separate roles for:

Investigation
Proposal ingestion
Context collection
Risk evaluation
Policy evaluation
Orchestration
Runbook execution
EKS modification
Verification
Audit writing

Do not allow one role to modify the proposal, policy, executor, and audit record.

Immutable evidence

Store decisions and evidence in encrypted Amazon S3.

Consider:

Versioning
Object Lock where required
AWS KMS customer-managed keys
Restricted deletion
CloudTrail data events
Payload checksums

Idempotency

Create an idempotency key:

incidentId + target + actionType + targetRevision

If an identical action is:

Running: reject the duplicate
Successful: return the previous result
Failed: require a new proposal
Rolled back: block automatic retry

Concurrency locking

Use DynamoDB conditional writes to lock a workload:

LOCK#commerce-prod#checkout#checkout-api

Only one remediation workflow should modify a workload at a time.

Input integrity

Hash the proposal before authorisation.

Before execution, verify:

hash(authorised proposal) == hash(execution proposal)

Network controls

Where possible:

Use private subnets
Use a private EKS endpoint
Use VPC endpoints
Restrict outbound access
Restrict security-group egress
Avoid general internet access for the executor

Secrets

The proposal must not contain:

Passwords
Tokens
Private keys
Database credentials
Customer data

Pass references, not values.

Observability for the remediation platform

The remediation control plane needs its own SLOs.

Recommended metrics:

RemediationProposalReceived
RemediationProposalRejected
RemediationProposalExpired
RemediationRiskScore
RemediationAutoExecuted
RemediationApprovalRequested
RemediationDenied
RemediationExecutionSucceeded
RemediationExecutionFailed
RemediationVerificationPassed
RemediationVerificationFailed
RemediationRollbackSucceeded
RemediationRollbackFailed
RemediationDuplicateSuppressed
RemediationConcurrentChangeDetected

Recommended dimensions:

Environment
Cluster
Namespace
Service
ActionType
RiskBand
DecisionRoute
RunbookVersion
PolicyVersion
Outcome

Do not use proposal IDs as metric dimensions. Store them in logs.

Alarm on:

Executor failure
Rollback failure
Workflow timeout
Policy-evaluation failure
Approval timeout
Audit-write failure
Unexpected increases in automatic remediation
Repeated remediation of the same workload

When the control plane fails internally, deny or pause remediation. Do not bypass the control.

Metrics that prove value

Do not publish invented performance claims.

Measure:

Metric	Definition
Mean time to detect	Incident start to alert
Mean time to investigate	Alert to probable root cause
Mean time to authorise	Proposal to policy decision
Mean time to remediate	Authorised action to recovery
Mean time to verify	Execution to verified recovery
Unsafe-action prevention rate	Unsafe proposals blocked
Automatic-remediation success rate	Auto actions passing verification
False-remediation rate	Executed actions based on incorrect RCA
Rollback success rate	Failed remediations reversed
Change-collision prevention rate	Conflicting changes blocked
Cost per investigation	AI, telemetry, workflow, and execution cost
Repeat-incident rate	Recurrence of the same failure mode

Compare:

Manual incident response
AI-assisted investigation with manual execution
AI-assisted investigation with risk-gated execution

The objective is:

Reduce recovery time without increasing the rate or blast radius of unsafe production changes.

Limitations

This pattern does not eliminate risk.

Root-cause analysis may be incomplete
Telemetry may be missing
Deterministic policies can still be wrong
Runbooks can contain defects
Human approval can delay recovery
Cross-account approval may require additional workflow design
Business validation may be difficult to automate safely
Break-glass actions still require human ownership

Do not expand autonomous permissions during an outage. Maintain a documented and audited break-glass process.

Production-readiness checklist

Investigation

Diagnostic access is read-only
MCP tools are narrowly scoped
Generic shell access is prohibited
Evidence sources are recorded
Output uses a versioned schema

Authorisation

Actions are allowlisted
Risk and urgency are separated
Live context is refreshed
Proposals expire
Default is deny
Forbidden actions have explicit deny policies
Policy changes use CI/CD
AI cannot change its own permissions

Execution

Runbooks are versioned
Parameters are validated
Free-form commands are rejected
IAM uses least privilege
Kubernetes RBAC is namespace-scoped
Secrets are inaccessible
Blast radius is bounded
Actions are idempotent
Concurrent changes are locked

Verification

Verification is independent
Infrastructure and business checks are included
Multiple successful observations are required
Failure triggers rollback
Rollback is verified
Success requires evidence

Audit

Proposal is immutable
Risk factors are stored
Policy and runbook versions are stored
Approvals are recorded
Execution logs are retained
Verification results are retained
Denied actions are searchable

Conclusion

AI can accelerate incident investigation, but investigation and production authority are different capabilities.

A system that proposes a change, authorises it, executes it, and judges its own success has collapsed too many trust boundaries.

The safer design separates responsibilities:

AWS DevOps Agent investigates
A structured proposal captures intent
A deterministic risk engine evaluates danger
Amazon Verified Permissions authorises
AWS Step Functions coordinates
Systems Manager executes allowlisted runbooks
IAM and Kubernetes RBAC contain access
Progressive delivery limits blast radius
Application Signals and Synthetics verify recovery

The result is not uncontrolled autonomy. It is bounded autonomy:

AI for investigation
Policy for authority
Runbooks for execution
SLOs and business tests for truth
Humans for ambiguity and high-risk decisions

AI proposes. Deterministic policy authorises. Controlled automation executes. Independent telemetry verifies.

That is the difference between an impressive AIOps demo and a production-safe reliability platform.

Disclaimer

The architecture, code samples, scoring weights, thresholds, IAM boundaries, Cedar policies, Kubernetes permissions, approval rules, and verification criteria in this article are reference examples.

Adapt and test them against your workload characteristics, security policies, compliance requirements, incident processes, service quotas, recovery objectives, and operational maturity.

Do not enable automatic production remediation until the actions have passed controlled failure injection, rollback testing, security review, and operational readiness exercises.