Originally published in AWSBuilder
The hardest problem in AI-driven operations is not getting an agent to
diagnose an incident.
Modern models can correlate logs, metrics, deployment events, traces,
Kubernetes state, and historical incidents well enough to produce
plausible remediation proposals. The harder question begins one step
later:
Who decides whether the proposed action is actually allowed to touch
production?
That distinction matters because reasoning quality and operational
authority are different properties.
An agent can be highly accurate and still eventually make a bad
decision. If that decision carries unrestricted production authority,
model accuracy becomes a weak safety boundary. A better architecture
assumes that recommendations can be wrong and constrains what happens
when they are.
Telemetry + Cluster State
|
v
AI Investigator
|
v
Structured Remediation Proposal
|
v
Deterministic Evidence Collector
|
v
Policy + Risk Evaluation
| | |
v v v
AUTO APPROVAL DENY
| |
| Human Decision
| |
+----> Revalidation
|
v
Bounded Runbook
|
v
EKS Executor
|
v
Independent Verification
The important architectural decision is not which model sits at the top.
It is that the model does not own the bottom half.
Start with an executable contract
An operations agent should never hand an executor a natural-language
instruction such as:
Fix the payments service.
There is almost nothing meaningful to authorize in that request.
Instead, require the agent to produce a typed remediation proposal:
{
"action": "RollbackDeployment",
"cluster": "prod-eks",
"namespace": "payments",
"workload": "payments-api",
"observedRevision": 42,
"targetRevision": 41,
"reason": "error_rate_regression",
"executionBounds": {
"timeoutSeconds": 180,
"maxUnavailable": 1
}
}
This document describes intent, not truth.
That distinction is critical.
The model may propose that revision 41 is the rollback target. It should
not be trusted to assert that revision 41 is healthy, that no
incompatible database migration occurred, or that revision 42 is still
running when execution begins.
Those facts need to come from deterministic systems.
A separate evidence collector can enrich the request:
{
"deploymentGeneration": 108,
"currentRevision": 42,
"previousRevisionHealthy": true,
"statefulMigrationDetected": false,
"maintenanceFreeze": false,
"evidenceTimestamp": "2026-08-19T13:01:14Z"
}
This gives us the first hard boundary:
The agent proposes the action. Trusted infrastructure establishes the
state under which that action may be considered.
Without that separation, policy enforcement becomes security theater. An
agent capable of supplying both the request and the evidence used to
authorize that request can effectively authorize itself.
Put authorization outside the agent
Amazon Verified Permissions is useful here because it externalizes
authorization decisions into policies written in Cedar. The application
asks whether a principal may perform an action against a resource in a
particular context, and receives an authorization decision.
Conceptually, a rollback policy could resemble:
permit (
principal == RemediationActor::"eks-remediator",
action == RemediationAction::"RollbackDeployment",
resource is KubernetesWorkload
)
when {
context.previousRevisionHealthy == true &&
context.statefulMigrationDetected == false &&
context.maintenanceFreeze == false &&
context.evidenceAgeSeconds <= 30
};
The exact schema will depend on the implementation, but notice what is
absent:
modelConfidence > 0.95
Confidence may be useful for deciding whether the system needs more
investigation. It is a poor substitute for operational invariants.
The control plane should ask questions such as:
Is the target revision known?
Is it healthy?
Has persistent state changed?
Is the request still current?
Is this namespace eligible for autonomous remediation?
Does the executor have authority for this resource?
Is a change freeze active?
These are much stronger authorization signals than whether an LLM
reports high confidence.
Cedar also supports explicit forbid policies and a default-deny
evaluation model. A matching forbid overrides permits, and a request
without an applicable permit is denied.
That makes certain boundaries straightforward:
forbid (
principal,
action,
resource
)
when {
context.production == true &&
context.modifiesPersistentData == true
};
Some operations should simply never enter an autonomous path.
Deleting persistent data, changing cluster-wide authorization, modifying
identity infrastructure, or executing arbitrary shell commands are
reasonable examples.
Do not turn Cedar into an incident workflow engine
There is an architectural trap here.
It is tempting to make the policy system return increasingly complicated
outcomes:
ALLOW_AUTO
ALLOW_WITH_APPROVAL
ALLOW_IF_SRE
INVESTIGATE_MORE
ESCALATE_SEV1
I would avoid that.
Authorization and operational risk classification are related, but they
are not the same responsibility.
Keep the classifier deterministic and separate:
Risk classifier
|
+--> AUTO
|
+--> HUMAN_APPROVAL
|
+--> ESCALATE
Then ask the authorization system whether the identified actor is
permitted to execute the requested operation in that path.
This keeps Cedar focused on authorization rather than turning policy
expressions into a hidden orchestration language.
Step Functions owns workflow state
Authorization tells us whether an operation may happen.
It does not solve what happens before or after the decision.
This is where AWS Step Functions fits naturally.
A remediation state machine could look like:
CollectEvidence
|
v
ValidateProposal
|
v
ClassifyRisk
/ | \
AUTO APPROVAL DENY
| |
| WaitForHuman
| |
+------+
|
v
RefreshEvidence
|
v
Reauthorize
|
v
ExecuteRunbook
|
v
VerifyRecovery
/ \
SUCCESS FAILED
|
v
ESCALATE
For human approval, Step Functions supports the callback-with-task-token
pattern. A workflow can pause until an external process returns the task
token using SendTaskSuccess or SendTaskFailure. Human approval is a
natural use case for this pattern.
The critical state in that diagram, however, is not WaitForHuman.
It is:
RefreshEvidence
Human approval expires
Suppose the system proposes:
Rollback payments-api from revision 42 to revision 41
An SRE reviews the evidence and approves the action.
Eight minutes pass before execution.
During those eight minutes, the deployment pipeline releases revision
43.
The original approval is now describing a production state that no
longer exists.
Executing the approved rollback blindly would mean modifying revision 43
based on analysis of revision 42.
This is a time-of-check versus time-of-use problem.
The solution is simple conceptually:
Approval authorizes intent. It does not freeze production state.
Immediately before execution, recollect the critical evidence and
evaluate authorization again.
The remediation contract should also carry preconditions:
{
"expectedGeneration": 108,
"expectedRevision": 42,
"targetRevision": 41
}
The executor reads the Deployment immediately before mutation.
If it finds:
expected generation: 108
observed generation: 109
execution stops.
Do not ask the model whether generation 109 is "probably okay."
Return the workflow to investigation.
Once the underlying state changes, the authorized operation is no longer
necessarily the same operation.
Make execution boring
The execution layer should be deliberately less intelligent than the
investigation layer.
AWS Systems Manager Automation provides runbooks consisting of defined
parameters and sequential actions. Custom runbooks can execute scripts,
invoke AWS APIs, call Lambda functions, and compose other automation
actions.
That allows us to expose something like:
Runbook:
EKS-BoundedDeploymentRollback
Inputs:
Cluster
Namespace
Deployment
ExpectedGeneration
ExpectedCurrentRevision
TargetRevision
TimeoutSeconds
Instead of:
execute(command)
That difference is one of the strongest controls in the architecture.
The AI cannot decide to append:
kubectl delete namespace payments
because there is no arbitrary command parameter.
The execution vocabulary should be intentionally small:
RollbackDeployment
RestartSelectedStatelessPods
ScaleDeploymentWithinBounds
CordonNode
RemoveKnownBadPod
Each action gets its own validation, permissions, preconditions,
timeout, and verification behavior.
Systems Manager Automation can invoke a purpose-built Lambda executor
through aws:invokeLambdaFunction.
I would prefer this pattern over turning EKS worker nodes into generic
administrative hosts using Run Command.
Systems Manager Run Command is designed to execute commands on managed
nodes.
Technically, it could be used to run Kubernetes administration commands
from a managed host. Architecturally, that creates a generic command
channel precisely where we are trying to remove one.
A dedicated executor with a narrow API is easier to reason about.
Least privilege must exist below policy
Policy alone is not enough.
Imagine that a Cedar policy is accidentally changed and begins
permitting an operation it should deny.
If the executor has cluster-admin privileges, the policy error
immediately becomes a production capability.
The better model is:
Policy boundary
+
IAM boundary
+
EKS authentication boundary
+
Kubernetes RBAC boundary
+
Runbook parameter boundary
Amazon EKS access entries associate IAM principals with Kubernetes
access. Permissions can be supplied through EKS access policies or by
mapping the principal to Kubernetes groups and implementing permissions
with Kubernetes RBAC.
For a tightly constrained remediation executor, custom Kubernetes RBAC
is often attractive because the exact verbs and resources can be
controlled.
For example:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: payments
name: bounded-remediator
rules:
- apiGroups: ["apps"]
resources:
- deployments
- deployments/scale
- replicasets verbs:
- get
- list
- watch
- patch
Then bind the Kubernetes group associated with the executor's EKS access
entry.
The executor should not receive access to:
Secrets
ClusterRoles
ClusterRoleBindings
PersistentVolumes
CustomResourceDefinitions
Namespaces
ServiceAccounts
unless a specific remediation capability genuinely requires it.
The principle is stronger than least privilege:
Make prohibited actions technically impossible from the remediation
identity.
Do not let the executor declare success
There is another subtle boundary after execution.
Suppose Kubernetes accepts:
PATCH deployment/payments-api
and returns success.
Did remediation succeed?
We know only that the API server accepted the mutation.
We do not know whether:
Pods became Ready
Service endpoints recovered
application errors dropped
latency normalized
dependencies remained healthy
the original SLI recovered
customer transactions succeeded
Execution success and recovery success are different states.
So verification should belong to a different component.
Executor
|
| mutation
v
Kubernetes
|
v
Independent Verifier
The verifier should preferably operate with read-only permissions and
consume both Kubernetes state and service-level telemetry.
For a rollback, it might evaluate:
Deployment available replicas
ReplicaSet convergence
Pod readiness
EndpointSlice endpoints
HTTP synthetic check
5xx rate
p95 latency
incident-specific SLI
The actual criteria must be workload-specific.
A batch processor cannot be validated like an HTTP API. A Kafka consumer
may require lag and processing-rate checks. A stateful service may
require consistency or replication signals.
This is why "AI remediation platform" is often too broad a product
boundary.
Safe remediation is easier when capabilities are designed around known
workload classes and explicit recovery contracts.
What happens when verification fails?
The obvious answer is:
Undo the remediation.
That answer is dangerous.
The remediation may have changed state. A second rollback may compound
the failure. The system may no longer satisfy the preconditions under
which the first action was approved.
Automatic reversal should therefore be treated as another remediation
request, with its own policy and preconditions.
A safer default is often:
Verification failed
|
v
Freeze further automation
|
v
Capture current state
|
v
Escalate to human
Autonomous systems need a reliable way to stop being autonomous.
Design failure behavior before success behavior
The control plane itself will fail.
That needs explicit semantics.
If Verified Permissions cannot return a decision:
DENY
not:
continue because this is an emergency
If the evidence collector cannot determine whether a stateful migration
occurred:
UNKNOWN
must not silently become:
FALSE
If human approval times out, block or escalate.
If the runbook receives an unexpected parameter:
ABORT
If production state changes between authorization and execution:
REINVESTIGATE
If verification is inconclusive:
ESCALATE
These decisions may reduce the percentage of incidents handled
autonomously.
That is acceptable.
Automation coverage is not the primary reliability metric for an
autonomous remediation system.
The more meaningful question is:
How much production authority can we safely delegate while keeping the
worst credible failure bounded?
The architectural lesson
The interesting part of AI operations is no longer whether an LLM can
call kubectl.
It can.
The harder engineering problem is building a system where a bad
recommendation cannot automatically become an unbounded production
change.
That requires separating responsibilities:
AI
Reason about the incident
Propose an action
Evidence collector
Establish current operational truth
Risk classifier
Determine the execution path
Amazon Verified Permissions + Cedar
Authorize the exact operation
AWS Step Functions
Own workflow state and approval boundaries
AWS Systems Manager Automation
Expose bounded operational runbooks
EKS access entries + Kubernetes RBAC
Constrain technical capability
Independent verifier
Determine whether the system recovered
None of those mechanisms makes the AI correct.
That is precisely why they matter.
A mature production architecture assumes that eventually the model will
be wrong, the evidence will become stale, a dependency will behave
unexpectedly, or a policy will contain a defect.
The architecture should make those failures survivable.
The goal is not an AI agent powerful enough to operate production
without humans.
The goal is an operations control plane disciplined enough that
increasingly capable agents can participate without inheriting
unrestricted production authority.
That is a much more useful definition of autonomous operations.
Top comments (0)