Alejandro Velez for AWS Community Builders

Posted on Oct 31 Edited on Oct 31

Eyes, Not Hands: Adding AWS DevOps Agent, Bedrock AgentCore and Kiro to a GitOps Platform on EKS

#agents #ai #aws #devops

__Level 300 — Advanced | Amazon EKS, Argo CD, AWS DevOps Agent, Amazon Bedrock AgentCore, Kiro CLI, Amazon EventBridge

🎯 The Challenge

Scenario

Suppose you are the Platform Engineering lead of an enterprise that already runs Kubernetes the GitOps way: an Amazon EKS hub cluster with Argo CD manages a fleet of spoke clusters, and every team onboards its workloads through a single values file in Git. It works. Deployments are boring, in a good way.

Operations are not. On-call engineers keep getting paged for the same kind of incidents: a bad config is merged, pods start crash-looping, and someone spends 40 minutes reading logs and events before realizing the fix is "revert that commit". Then leadership asks the question every platform team is hearing:

"Can AI agents take the first part of on-call?"

Your mission: add agentic investigation and remediation to the existing platform, under these constraints:

✅ No agent gets write access to production. Not to the cluster, not to the AWS account.
✅ Every fix is a reviewable pull request. Git stays the single source of truth; Argo CD stays the only thing that changes the cluster.
✅ Don't rebuild the platform. The hub-spoke GitOps setup stays as it is; agents are added on top.
✅ Multi-account by design. Workloads and CI/CD live in different AWS accounts.
✅ Everything is observable and auditable. Every agent action leaves a trace.

This post is the architecture behind the TechLabVel video, Cap 2.

My Approach: Eyes, Not Hands

The fastest demo is always the same: connect an LLM to the cluster with an admin kubeconfig and let it "fix things". I've seen it in many conference talks, and it's also the one setup I would never take to a client's production. A model can be wrong, a prompt can be injected, a credential can leak. With write access, each of those becomes an outage.

So I split the problem the way with one load-bearing rule:

Reasoning is decoupled from execution. The agents can read and propose anything, but can mutate nothing directly. Every production change routes through a separate control plane: a GitOps pull request.

The agent gets eyes (logs, events, metrics, code) but no hands. GitOps is the hands, and a human holds them. I also decided not to build a new platform for this: everything is added on top of the hub-spoke GitOps setup I already had.

The Architecture

The six numbered steps in the diagram match the list below. Agents (teal) only read; the only write path is the pull request (green) that Argo CD pulls (orange).

Detect: a CloudWatch alarm on a Container Insights metric fires when a pod is OOMKilled.
Wake up the agent: EventBridge invokes a small Lambda that sends a signed webhook to the AWS DevOps Agent.
Investigate: the agent reads Kubernetes state through a read-only MCP server hosted on Amazon Bedrock AgentCore, plus CloudWatch, CloudTrail and the GitHub repos.
Hand off: the alarm event is forwarded cross-account to the DevSecOps account.
Propose: CodeBuild runs Kiro CLI headless; it edits the GitOps repo and opens a pull request.
Approve and apply: a human merges; Argo CD reconciles the spoke cluster.

Account	What lives there	Why
Workload account	EKS hub (Argo CD) + EKS spoke (workloads), CloudWatch, AWS DevOps Agent space, AgentCore Runtime + Gateway	The agent investigates where the signals are
DevSecOps account	GitHub runners, the Kiro remediation pipeline (EventBridge bus, Lambdas, SQS, CodeBuild)	The only component that writes code is isolated from the workloads
GitHub	`gitops` (platform + team values), `petadoptions` (app manifests)	Git is the source of truth and the approval gate

The Starting Point: GitOps Hub-Spoke

This is the platform from the GitOps and IaC at Scale series. Argo CD runs on the hub and registers each spoke as a cluster secret. Labels on that secret decide which add-ons each cluster gets, through ApplicationSets:

kubectl --context hub -n argocd get secrets -l argocd.argoproj.io/secret-type=cluster -o json \
  | jq -r '.items[] | "\(.data.name | @base64d)\t\(.metadata.labels.environment)"'

gitops-spoke-eks-dev    dev
in-cluster              control-plane

Teams onboard with one file in the platform repo. The workloads entry tells Argo CD which repo and which Kustomize overlay to deploy to the spoke:

# gitops/platform/teams/petadoptions/argoproject-values.yaml (excerpt)
workloads:
  - name: app
    repoURL: https://github.com/<org>/petadoptions
    targetRevision: main
    path: k8s/overlays/dev
    namespace: petadoptions

Keep this file in mind.

Hands On

Prerequisites

Tool / component	Version used
Amazon EKS (hub / spoke)	1.37 / 1.36
OpenTofu + Terragrunt	1.1.3 / 1.13.2 versions
Terraform AWS provider	6.69.1
`awslabs.eks-mcp-server`	0.2.1
Kiro CLI	latest at build time (TODO(you): pin it, see review)
Region	us-east-2

Deploy order for the agent side: S3 (evaluation bucket) → ECR → IAM roles → MCP server (AgentCore) → Agent Space → tool allowlist. Build and push the arm64 MCP image before the MCP server stack.

Step 1: The Alarm (A Signal That Actually Fires)

The first version of the alarm watched a custom SLO burn-rate metric. Nothing ever published it, so the alarm could never fire. The lesson came early: an agent is only as good as the signal that wakes it up. The final alarm uses a metric that the CloudWatch Observability add-on (Container Insights) already publishes for the spoke:

# environments/dev/observability.tfvars (excerpt)
alarm_name          = "petsearch-oomkilled"
metric_namespace    = "ContainerInsights"
metric_name         = "pod_container_status_terminated_reason_oom_killed"
statistic           = "Sum"
threshold           = 0      # any OOMKill in the period trips the alarm
period              = 300
evaluation_periods  = 1
comparison_operator = "GreaterThanThreshold"
treat_missing_data  = "notBreaching"   # no OOMKills reported = healthy
dimensions = {
  ClusterName = "<spoke-cluster>"
  Namespace   = "petadoptions"
  PodName     = "petsearch"
}

Step 2: The Trigger (Why a Lambda Signs the Webhook)

EventBridge can't call the AWS DevOps Agent webhook directly: the generic webhook requires an HMAC signature over the timestamp and payload, and EventBridge can't compute it. A small Lambda (about 100 lines) does:

# trigger/src/handler.py (excerpt)
def _sign(timestamp: str, body: bytes, secret: str) -> str:
    # AWS DevOps Agent HMAC: base64(HMAC-SHA256(secret, "{timestamp}:{payload}")).
    # The timestamp is part of the signed material (replay protection) and the
    # digest is base64 — NOT a bare hex of the body.
    mac = hmac.new(
        secret.encode("utf-8"),
        f"{timestamp}:".encode("utf-8") + body,
        hashlib.sha256,
    )
    return base64.b64encode(mac.digest()).decode("ascii")

The webhook URL and secret live in AWS Secrets Manager. The timestamp is part of the signed material, which protects against replays.

Step 3: The MCP Server on AgentCore (Eyes, Not Hands)

The AWS DevOps Agent needs to see inside the cluster: pods, events, logs. Instead of writing our own tools, we wrap the official awslabs.eks-mcp-server (pinned to 0.2.1) and host it on Amazon Bedrock AgentCore Runtime. AWS owns the tool logic; we own the posture.

The posture is read-only, enforced three times inside the server:

# src/mcp-server/server.py (excerpt)
# We NEVER enable write.
ALLOW_WRITE = False

# Defense in depth: refuse to start rather than silently expose mutating tools.
if os.environ.get("READ_ONLY", "true").lower() not in ("1", "true", "yes"):
    raise SystemExit("EKS MCP host is read-only by design; set READ_ONLY=true.")

# Remove mutating tools from the registry, so the agent never even SEES them.
_WRITE_TOOLS = {
    "manage_k8s_resource",   # create/replace/patch/delete K8s objects
    "apply_yaml",            # apply manifests
    "generate_app_manifest", # writes a manifest file
    "add_inline_policy",     # mutates IAM
    "manage_eks_stacks",     # CloudFormation create/update
}

Read-only is enforced at the listing layer, not only at call time: a tool the model never sees is a tool it can't be tricked into calling.

In front of the runtime sits an AgentCore Gateway with IAM (SigV4) authentication and semantic tool search:

# modules/agentcore-mcp/main.tf (simplified: count, JWT option and tags removed)
resource "aws_bedrockagentcore_gateway" "mcp" {
  name            = var.gateway_name
  role_arn        = var.execution_role_arn
  authorizer_type = "AWS_IAM"        # SigV4 — no shared secrets

  protocol_type = "MCP"
  protocol_configuration {
    mcp {
      instructions = "Read-only ops/K8s tools. Mutation is not exposed; actuation routes through the control plane."
      search_type  = "SEMANTIC"
    }
  }
}

Finally, the Agent Space only gets an allowlist of tools, managed as code (AWS::DevOpsAgent::Association, MCPServerSigV4):

mcp-runtime___list_k8s_resources
mcp-runtime___get_pod_logs
mcp-runtime___get_k8s_events
mcp-runtime___list_api_versions
x_amz_bedrock_agentcore_search

On the cluster side, the runtime's IAM role has an EKS access entry with the managed AmazonEKSViewPolicy: get, list and watch, nothing else.

Step 4: The Agent Space (What the Agent Can See)

The AWS DevOps Agent space is created with the native aws_devopsagent_agent_space resource, and its description states the contract:

# stacks/application/compute/devops-agent/main.tf (excerpt)
resource "aws_devopsagent_agent_space" "this" {
  name        = "${local.name_prefix}-space"
  description = "DevOps Agent space for ${var.project} ${var.environment} (read-only investigate + plan; proposes, never mutates)."
  locale      = var.locale
  kms_key_arn = var.kms_key_arn != "" ? var.kms_key_arn : null   # optional customer-managed key
}

Access	What the agent gets
Agent IAM role (read-only policy)	CloudWatch alarms, metrics and logs, X-Ray, CloudTrail `LookupEvents`, EKS describe
MCP server (SigV4, via AgentCore Gateway)	The 4 read-only Kubernetes tools above
GitHub: `gitops`, `petadoptions`, framework repo	The code and history to correlate "what changed"
Slack on-call channel	Where findings are posted for humans
Event channel (webhook)	The trigger from Step 2

Elevated actions are disabled in the space. The agent's IAM role trusts the aidevops.amazonaws.com service only for this account's agent spaces (aws:SourceAccount + aws:SourceArn), which protects against the confused-deputy problem.

Step 5: The Remediation Pipeline (Proposal to Pull Request)

The second EventBridge rule forwards the same alarm event cross-account to an event bus in the DevSecOps account. Its IAM role can do exactly one thing:

# stacks/application/integration/event-forwarder/main.tf (excerpt)
resource "aws_cloudwatch_event_target" "hub_bus" {
  rule      = aws_cloudwatch_event_rule.to_hub.name
  target_id = "kiro-hub-bus"
  arn       = var.hub_kiro_event_bus_arn
  role_arn  = aws_iam_role.forwarder.arn   # policy: events:PutEvents on that bus only
}

In the DevSecOps account, an extractor Lambda asks the AWS DevOps Agent API for the mitigation plan and queues it in SQS. A starter Lambda launches CodeBuild, which runs Kiro CLI headless with strict rules ("only edit files in this repo; do NOT run AWS CLI or kubectl"):

kiro-cli chat --no-interactive --trust-tools=read,write,grep,shell "$(cat /tmp/prompt.md)"

The buildspec wraps the agent with guardrails the agent can't change: no changes means no PR, more than 10 files means abort, and the branch is always a new kiro-fix/<timestamp> that goes through review. GitHub access uses an AWS CodeConnections GitHub App, with no personal access tokens.

Defense in Depth

No single control carries the safety. If one fails, the next one holds:

When It Breaks: Gotchas I Hit

Symptom	Cause	Fix
`Invalid STS role configuration for session sigv4McpServerSession`	The agent role's trust statement had `sts:TagSession`	Use `sts:AssumeRole` only for `aidevops.amazonaws.com`
`Role validation failed ... doesn't include either SourceArn or SourceAccount`	No confused-deputy guard	Add `aws:SourceAccount` and `aws:SourceArn` = `arn:aws:aidevops:<region>:<account>:service/` (not `agentspace/`)
`403 ... Insufficient permissions` from the gateway	Trust OK, but no invoke permission	Grant `bedrock-agentcore:InvokeGateway` on the gateway ARN
The MCP server shows zero tools	Gateway created without a target	Add the `mcp_server` target pointing at the runtime
SigV4 signature rejected	Service name set to `execute-api`	The SigV4 service for AgentCore is `bedrock-agentcore`
Read tools return `forbidden`	No EKS access entry for the runtime role on the spoke	Add the access entry with `AmazonEKSViewPolicy`
Alarm never fires	Alarm on a metric nothing publishes	Use a metric the Container Insights add-on already emits

Conclusion

We didn't give an AI agent the keys to production. We gave it the best possible view of production, and one narrow, reviewable way to suggest a change. The existing GitOps platform did the rest: the pull request is the dry-run and the approval, and Argo CD is the deterministic executor.

In the next episode we put this architecture under a real incident: a pod that's OOMKilled under load, an agent that investigates, and the two things it got wrong before the platform got it right.

How are you giving AI agents access to your clusters today: read-only, through PRs, or with the keys? I'd love to hear what's working for you 👇

Happy building! 🚀

References

The opinions expressed in this post are my own and do not necessarily reflect those of my employer or AWS.

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

DEV Community