A few weeks ago I found myself staring at a ClusterRoleBinding named vulnerable-sa-cluster-admin that I had created myself, on purpose, in a brand-new EKS cluster. The goal wasn't to break anything in production. It was to answer a question I keep running into whenever I talk about AI agents and infrastructure: if an agent can read your Kubernetes audit logs and understands exactly what a privilege escalation looks like, how much should it be allowed to do about it on its own?
So I built a small, throwaway EKS cluster, gave one pod way too much power, attacked my own cluster with a fixed script, and had an AI agent watch the whole thing happen. It correctly flagged the attack. It even knew how to fix it. But it couldn't fix anything until I clicked a button.
This is a write-up of that experiment, including the parts that didn't work on the first try.
Why RBAC is a strange fit for full autonomy
RBAC misconfiguration is one of those problems everyone nods along to in a conference talk and then still ships anyway. A ServiceAccount gets bound to cluster-admin because it's Friday afternoon and the "real" permissions can be figured out later. Nobody comes back to fix it. Eventually something reads a Secret it was never supposed to see.
Detecting that pattern is exactly the kind of task an LLM is good at: read a Kubernetes audit log entry, decide whether it looks like a legitimate operator action or something a human should look at. Reacting to it automatically is a different question. Deleting the wrong ClusterRoleBinding on a live cluster can take down a controller that everything else depends on. That asymmetry — cheap to detect, expensive to get wrong when acting — is why I wanted a human in the loop for the "fix it" step, even in a demo this small.
Building a cluster that's broken on purpose
I used eksctl to stand up a minimal cluster in ap-northeast-1: one t3.medium node, audit logging turned on from the start so I'd have something for the agent to read.
cloudWatch:
clusterLogging:
enableTypes:
- audit
- authenticator
managedNodeGroups:
- name: ng-min
instanceType: t3.medium
desiredCapacity: 1
On top of that I deployed a namespace called vulnerable-app with a pod that's meant to look like a public-facing web service — the kind of thing an attacker might land on first. Its ServiceAccount, vulnerable-sa, is bound to cluster-admin through a ClusterRoleBinding. In a separate payments namespace, I planted a Secret with fake payment provider credentials, just realistic enough to be worth stealing.
Small thing that cost me ten minutes: my first pod spec used bitnami/kubectl:1.31 as the image, which quietly doesn't exist anymore — Bitnami restructured their image tags and that pull just 404s. alpine/k8s:1.31.5 worked fine as a drop-in replacement.
The attack takes about a second
The attack script is intentionally boring and fully scripted, so it's reproducible: exec into the pod, enumerate what the ServiceAccount can do, read the Secret in the other namespace, and create a second, differently-named ClusterRoleBinding called backdoor-admin. That last step is the part I actually cared about. An attacker who has cluster-admin for a moment usually doesn't stop there; they plant a second, less obvious binding so that revoking the first one doesn't lock them out.
[attack] == Step 3: Steal the credentials in the payments namespace ==
dummy-sk-live-not-a-real-key-1234567890
[attack] == Step 4: Create a persistent backdoor via a new ClusterRoleBinding ==
clusterrolebinding.rbac.authorization.k8s.io/backdoor-admin created
All of that lands in the EKS audit log within seconds, tagged with the verb, the resource, and the identity that did it.
Teaching Claude to read an audit log entry
The detector is a plain polling loop, nothing fancy: query CloudWatch Logs Insights every 30 seconds for create events on clusterrolebindings/rolebindings/clusterroles and get events on secrets, then hand each new event to Claude Sonnet 4.6 through Bedrock's Converse API with a tool call it has to fill in — is_suspicious, risk_level, reasoning, proposed_action.
The interesting part wasn't getting a JSON-shaped response out of the model. It was watching it correctly separate cluster bootstrap noise from something worth flagging. EKS creates a handful of ClusterRoleBindings of its own on startup — eks:node-bootstrapper, vpc-resource-controller-rolebinding, and so on — all performed by eks:cluster-bootstrap. The model learned to wave those through and reserve is_suspicious: true for bindings created by identities that had no business creating them:
{
"is_suspicious": true,
"risk_level": "high",
"reasoning": "A non-system ServiceAccount created a ClusterRoleBinding named 'backdoor-admin', a pattern consistent with privilege escalation. The name itself suggests malicious intent.",
"proposed_action": "delete_clusterrolebinding"
}
It also flagged the original vulnerable-sa-cluster-admin binding (the one I'd created myself while setting up the demo) as suspicious, for the same reason. Which is fair. Nothing about "an IAM user creates a ClusterRoleBinding granting cluster-admin to a random ServiceAccount" screams "this was intentional and fine." A real audit log doesn't know the difference between a deliberately vulnerable demo and an actual mistake, and honestly, most actual mistakes look exactly like that.
Why the agent doesn't get to press delete
Every suspicious event goes into a DynamoDB table as a pending approval request instead of triggering an action directly. A separate executor process only picks up requests once their status flips to approved, and even then it validates the target resource name against a strict pattern before running kubectl delete clusterrolebinding — so a malformed or unexpected value from the model can't turn into an arbitrary shell command.
The approval itself happens in a small Next.js app deployed on Vercel, sitting behind Basic Auth, polling the same DynamoDB table every five seconds. Each pending item shows the risk level, what operation triggered it, who performed it, and the model's reasoning — the same explanation you'd want from a human analyst on call.
I clicked "Approve" on the backdoor-admin finding and watched the executor pick it up within fifteen seconds:
[executor] target=backdoor-admin
[executor] result: clusterrolebinding.rbac.authorization.k8s.io "backdoor-admin" deleted
$ kubectl get clusterrolebinding backdoor-admin
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "backdoor-admin" not found
That's the whole loop: audit event, Bedrock assessment, a human decision, then — and only then — a kubectl command with a name that's already been through a validator.
The reviewer doesn't have to read English
One thing I hadn't planned for: whoever is on call to click "Approve" at 3am isn't guaranteed to be a native English speaker. That's true of most approval-gate UIs, and it's easy to skip because the happy path ("I'll just write the UI in English") never fails a demo. It only fails the person actually staring at the screen during an incident.
The UI labels were the easy 90%. A small dictionary — eight languages, a dozen strings each — covers the buttons, the headings, the status pills. Text/JSON translation like that is nothing an LLM-era engineer should think twice about.
The harder part was the AI's reasoning itself, since that text doesn't exist until Bedrock generates it. Pre-translating a fixed set of strings doesn't work when the string is new every time. The fix turned out to be simpler than I expected: instead of asking for one reasoning string, I changed the tool schema so reasoning is an object with one field per language, and asked for all eight in the same call.
"reasoning": {
"type": "object",
"properties": {
"en": { "type": "string" },
"ja": { "type": "string" },
"zh": { "type": "string" }
// ... ko, fr, es, de, ar
},
"required": ["en", "ja", "zh", "ko", "fr", "es", "de", "ar"]
}
The prompt is explicit that these should read like something a native speaker wrote, not a literal translation of each other — and Claude is noticeably better at this than I expected. The Japanese version of a finding doesn't read like it was run through a translator after the English one; it reads like someone who thinks in Japanese wrote a security note. Same for the Arabic version, which came with a side quest I hadn't budgeted for: Arabic reads right-to-left, so the whole page layout, not just the text, has to flip. <main dir={locale === "ar" ? "rtl" : "ltr"}> and Tailwind mostly handled it, but it's a good reminder that "translate the strings" and "localize the interface" are not the same task.
The tradeoff is obvious: one Bedrock call now returns eight paragraphs instead of one, which costs more tokens per assessment. For a security queue that fires on genuinely suspicious events rather than every audit log line, that felt like a reasonable price for not assuming who's on the other end of the approval button.
One thing I didn't expect while building this
Unrelated to the security part, but worth mentioning: the Next.js version I scaffolded (16.2.11) renamed middleware.ts to proxy.ts. Same purpose — I used it for the Basic Auth check in front of the approval UI — but the file convention changed entirely, right down to the exported function name. It's a good reminder that "I already know Next.js" is a shrinking assumption if you're not reading the release notes for every major version.
What this pattern is actually for
RBAC misconfiguration was a convenient, self-contained scenario for this experiment, but the shape of the problem generalizes: an AI agent that can reliably detect something bad and confidently propose a fix is only half of a useful system. The other half is a queue, an audit trail, and a person who has to look at the reasoning before anything executes. That's the same pattern behind approval gates for AI agents making purchases, sending emails on someone's behalf, or touching production infrastructure — the detection can be as autonomous as you want; the blast radius of the action is what decides how much trust it gets by default.
The whole thing — cluster manifests, attack script, detector, executor, and the approval UI — is deliberately small enough to read in one sitting. If you're experimenting with the same "AI proposes, human disposes" pattern for your own infrastructure, I'd start exactly where I did: pick one narrow, well-understood attack, and resist the urge to let the agent skip the approval step even when you're fairly sure it's right.
Top comments (0)