Building a self-healing EKS cluster with Amazon Bedrock AgentCore and Strands agents

It's 3 a.m. and a pod is stuck in OOMKilled. You know the fix: raise the memory limit and restart. You've done it a dozen times. So why is a human still doing it?

I built a small project to see how far an AI agent could take that on-call chore. It receives Prometheus alerts, investigates the live cluster, fixes what's safe to fix, and hands the rest to a human with a JIRA ticket and a Slack message.

The code is on GitHub: bedrock-agentcore-eks-monitor.

The idea

Most Kubernetes failures fall into two buckets:

• Mechanical: the fix is well known and low risk (OOMKilled, a transient CrashLoopBackOff).
• Needs a human: the fix needs judgment or access the agent shouldn't have (a bad image tag, a missing secret, a node that is down).

An agent is useful in both. For the first bucket it acts. For the second it does the investigation up front, so the human starts with context instead of a bare alert.

Architecture

Alertmanager → Webhook server (EKS) → AgentCore Runtime (ARM64) → Claude on Bedrock

                                          ┌──────────────────────────|
                                             ▼                          ▼
                                    Auto-remediate                  Escalate
                                 (patch memory + restart)      (JIRA ticket + Slack)

There are four moving parts:

1. kube-prometheus-stack runs Prometheus and Alertmanager, with alert rules for OOMKilled, CrashLoopBackOff, ImagePullBackOff, NodeNotReady, MemoryPressure, PodPending and CreateContainerConfigError.
2. A small FastAPI webhook server runs in the cluster. Alertmanager posts to it, and it forwards each alert to the agent. It authenticates to AWS with IRSA (IAM roles for service accounts), so there are no long-lived keys in the cluster.
3. Amazon Bedrock AgentCore Runtime hosts the agent as a container. AgentCore handles invocation, sessions and scaling, so I don't run agent infrastructure myself.
4. A Strands agent with Claude on Bedrock does the reasoning and calls the tools.

Terraform provisions the AgentCore Runtime, the IAM roles, the ECR repository and the webhook server resources.

The agent

The agent is only a few dozen lines. Strands provides the agent loop, and BedrockAgentCoreApp exposes it as an AgentCore entrypoint:

app = BedrockAgentCoreApp()

@app.entrypoint
async def handle_request(payload: dict) -> dict:
    prompt = payload.get("prompt", "")
    if not prompt:
        return {"error": "No prompt provided"}
    result = agent(prompt)
    return {"response": str(result)}

The agent has eleven tools, all plain Python functions decorated with @tool:

• Read: list pods, get logs, describe a pod, get events, get a deployment, get node status
• Write: patch deployment resources, rollout restart, cordon a node
• Escalate: create a JIRA ticket, send a Slack notification

Most of the behavior lives in the system prompt, not in code. It works as a runbook:

• Ignore anything in kube-system or monitoring.
• For ImagePullBackOff, escalate straight away and never restart. A restart can't fix a bad image tag.
• For OOMKilled on a Deployment, read the deployment, double the memory limit (capped at 2048Mi), then do a rollout restart.
• For CrashLoopBackOff, do one rollout restart.
• For a NotReady node, check its status, cordon it if needed, and always escalate.
• For anything unknown, escalate.
• Act once, then reply with a one-line summary: Auto-fixed: ... or Escalated: JIRA <key> + Slack notified.

Guardrails matter more than the model

An agent that can patch deployments is an agent that can break them. A few choices keep this one safe:

• Narrow tools. The agent can patch resources and restart, but it can't delete or exec.
• Hard caps in the prompt. Memory is doubled, never more than 2048Mi.
• Act once. One remediation per alert prevents restart loops.
• Explicit "never" rules. ImagePullBackOff never triggers a restart.
• System namespaces are off limits.
• Escalation degrades gracefully. If JIRA or Slack credentials are missing, the tools return a clear "not configured" message with what they would have sent, so the agent can still report what happened.

I'd treat the prompt rules as a first layer, not the only one. For a production cluster I'd also enforce limits in RBAC, so the agent's service account physically can't do more than the runbook allows.

Trying it out

The repo includes test manifests for six scenarios. Alert rules fire immediately and Alertmanager's group_wait is 10 seconds, so from pod crash to agent action takes roughly 60 to 90 seconds.

Scenario	What the agent does
OOMKilled (32Mi limit)	Patches the limit to 256Mi and restarts
CrashLoopBackOff	Runs a rollout restart
ImagePullBackOff	Creates a JIRA ticket and sends a Slack message
Node NotReady (stop kubelet)	Checks the node, escalates
Pod Pending (asks for 100Gi RAM)	Escalates
Missing secret	Escalates

You can run the first three end to end with bash demo.sh.

In the logs, a successful run looks like this:

INFO:__main__:Webhook received: status=firing alerts=2
INFO:__main__:Invoking AgentCore Runtime arn:aws:bedrock-agentcore:...
INFO:__main__:Agent response: {"response": "Escalated: Slack notified"}

Lessons learned

AgentCore Runtime runs on Graviton. Build the image for linux/arm64 (docker build --platform linux/arm64 --provenance=false) or the runtime won't start it. Container-based deployment also gave me a cold start of about 3 seconds, compared with about 3 minutes for an S3 zip that pip-installs on start.

Session IDs need at least 33 characters. I use the alert fingerprint, which has the nice side effect that repeat notifications for the same alert share a session.

IRSA in a private cluster needs an STS VPC endpoint. Without it, pods can't exchange their tokens for credentials.

terraform import can fail quietly. My deploy.sh imports existing resources, and a missing local data file made the imports fail while || true hid the error. terraform apply then tried to recreate resources that already existed. The fix is documented in the README's troubleshooting section.

kubectl cordon is not NotReady. It only sets SchedulingDisabled. To trigger a real NotReady alert in the demo, I stopped the kubelet through SSM.

A small, fast model is enough. The agent uses Claude Haiku on Bedrock. The task is tool use against a clear runbook, not open-ended reasoning, so speed and cost matter more than raw capability. The model ID is an environment variable (BEDROCK_MODEL_ID), so you can swap models without a rebuild.

What I'd do next

• Add RBAC scoping and an approval step for riskier actions.
• Deduplicate and rate-limit alerts before they reach the agent.
• Record what the agent did and why, so you can audit its decisions.
• Evaluate the runbook prompt against a larger set of failure scenarios.

Try it

Everything is in the repo, including the Terraform, the Kubernetes manifests and a step-by-step README: https://github.com/chandika-s/bedrock-agentcore-eks-monitor

JIRA and Slack are optional. Without them, auto-remediation of OOMKilled and CrashLoopBackOff still works.

If you build something similar, I'd like to hear which runbooks you'd trust an agent with.