DEV Community: Andrey Lesnikov

Don't Give AI Agents Root. Let Them Open Pull Requests.

Andrey Lesnikov — Wed, 21 Oct 2026 08:16:26 +0000

Would I give an AI agent root access to a production Kubernetes cluster? No. Would I let it open pull requests? Gladly.

That sounds like a contradiction until you look at where the real control point in a GitOps setup sits. It isn't the cluster. It's the path a change takes before it reaches the cluster.

The idea that clicked for me

Earlier this month Mauro Morales, a CNCF Ambassador, published Don't give AI agents root. Make them propose the next system state. on the CNCF blog. His core argument is simple: an agent should inspect the current state of a system and propose the next one. It should not apply that state itself. The proposal goes through the same software factory every human change goes through: review, CI, tests, build, signing, release, deployment.

One line from his post stuck with me: if you give an agent root and expect a well-written AGENTS.md to prevent an unrecoverable mistake, you are fooling yourself. The model is nondeterministic. You can't guarantee the outcome, so you have to guarantee the path instead.

His post focuses on the operating system layer and immutable hosts. I want to look at the same principle from the Kubernetes and GitOps side, because for many platform teams the boundary already exists. We just don't always enforce it properly for bots.

GitOps already drew the line

If you run Argo CD or Flux, this model is almost boring. The rule is already there: what is not in Git does not end up in the cluster. The controller reconciles toward the desired state in the repository, and drift gets corrected.

So the safe shape for an agent looks like this:

Read access to metrics, logs, events and the manifests in Git, so it can diagnose problems.
Write access to a branch, so it can propose a change as a pull request.
Nothing else. No cluster-admin, no kubectl apply against production, no direct access to the sync controller.

The agent becomes one more contributor. Its change is a diff that a human can read, CI can test and policy can reject. If it's wrong, you close the PR. If it was merged and turns out wrong, you revert a commit, and you know exactly which commit and which identity introduced it.

That traceability is the part people underestimate. It's not only about stopping bad changes. It's about knowing afterwards what changed, when and who or what proposed it.

The gap: "root with a few extra steps"

Here is the point from Mauro's post that I think matters most: the agent's identity may propose, but it must not approve, merge, or change the pipeline. Otherwise you have just replaced root with root plus a few extra steps.

This is exactly where real setups tend to fall short. A common pattern looks like this:

Someone creates a bot account or a GitHub App for the agent.
Setting up fine-grained permissions takes time, so it gets admin on the repository "for now".
Branch protection has an exception for that account, or the bot can approve its own PRs through a second token.
The pipeline definition lives in the same repo, and the bot can edit it.

At that point the PR flow is theater. The agent can propose a change, approve it, merge it and adjust the CI job that was supposed to catch it. Argo CD then faithfully syncs whatever landed on main. The cluster is protected by GitOps in theory and wide open in practice.

What I check in a setup like this

None of this needs new tooling. It's mostly configuration you probably already have, applied consistently to non-human identities:

Separate identity per agent. No shared tokens with humans or other automation, so the audit trail means something.
Branch protection without bypass for bots. Required reviews from humans, required status checks, no admin override for the agent account.
No self-approval. The identity that opened a PR can't be the one that approves it, and there is no second token lying around that makes that possible.
Pipeline files are protected. Changes to CI definitions, Argo CD Application objects or policy files go through CODEOWNERS with a human owner.
Short-lived, scoped credentials. The agent's read access to observability data should expire and be limited to what it needs.
Policy in CI fails closed. If a policy check can't run or can't decide, the result is "not approved", never a silent green.

The last point deserves extra emphasis. A gate that passes when it has missing data is not a gate. For agent-proposed changes that matters even more, because the volume of proposals can grow much faster than the attention of the reviewers.

Where this leaves us

I don't think the answer is to keep agents out of production work. They are useful for reading signals, correlating logs and proposing a fix, especially when a tired on-call engineer would be doing it at 3 a.m. The answer is to give them the same contract every engineer on the team already has: you can suggest anything, and the system decides what gets applied.

GitOps gave us that contract years ago. The job now is to make sure the bot account actually lives under it, instead of quietly sitting above it because it was quicker to set up that way.

So I'll ask the same question I asked on LinkedIn: how much is an agent allowed to do in your production environment today, honestly?

Rehearse the Change Before You Apply It: Fail-Closed Blast Radius for Kubernetes and Argo CD

Andrey Lesnikov — Sat, 17 Oct 2026 15:56:44 +0000

A Helm upgrade scales two deployments in a payments namespace: one goes from 6 to 18 replicas, the other from 4 to 12. Review is easy. CI is green. The diff is tiny. Argo CD syncs it, and a few minutes later the new pods are stuck. The events say FailedCreatePodSandBox: failed to assign an IP address. The pod IP pool was nearly full, and nobody in the review thought about it, because nobody in the review looks at IP capacity when they read a replica count.

That scenario is not hypothetical decoration. It is the main end-to-end fixture in my open-source project, Architecture Rehearsal, because it is the cleanest example of a failure class I care about: changes whose blast radius only shows up once production executes them. The review had opinions. Production had the actual dependency graph.

I'm a DevOps platform engineer, and this post explains what the tool does, how it is built, what it has actually been tested against, and where it falls short.

The idea: rehearse first, and treat "I don't know" as a result

The core idea is simple. Before a change is applied, build a model of the relevant part of the system, apply the change to the model, and run a set of deterministic failure scenarios against it. Then let a policy decide whether the change is allowed to proceed.

The part I spent the most time on is a rule from the project's first ADR:

Missing data never becomes false approve.

Most gates I have used are binary: pass or fail. The problem is what they do when they lack information. If the PDB inventory was never collected, a binary checker usually finds no PDB violations and passes. That is a false approve, and it is worse than having no gate, because now there is a green check to point at.

So the gate has four outcomes, each with its own exit code:

Decision	Meaning	Exit
`approve`	No matched high-risk patterns; required coverage present	0
`warn`	Medium-risk findings	1
`block`	High/critical findings with evidence	3
`unknown`	Required data missing, not safe to approve	4

unknown is fail-closed by default. CI has to treat exit 4 as a failing gate, or someone has to explicitly write a policy that says otherwise. The intended consequence is that operators fix their collectors and RBAC instead of ignoring the gap.

How it works

The pipeline, in the repo's own words, is:

Collect â†’ Model â†’ Propose â†’ Rehearse â†’ Gate â†’ Observe â†’ Verify â†’ Calibrate

A causal graph from a cluster baseline

rehearsal snapshot k8s turns a cluster dump (a kubectl List export, or live collection) into a baseline snapshot: workloads, services, PVCs, NetworkPolicies, PDBs and the edges between them. A change is compiled from rendered manifests (rehearsal change manifests), from Terraform plans, or from an Argo CD Application's desired state. The change is scoped: by default it cannot remove nodes, so a rendered chart for one namespace doesn't accidentally look like "delete everything else in GitOps".

The scenarios are plain Go code, each with a name and a risk model. The current set includes cni-ip-capacity, pdb-disruption, rwo-node-loss, volume-az, service-routing and prom-zero-match. For the Argo CD path there are dedicated detectors for scheduling capacity (CPU/memory requests vs allocatable), PVC zone conflicts, NetworkPolicies that cut a dependency, PDB-vs-rollout conflicts, and blast radius. Each one returns PASS, WARN, BLOCK or UNKNOWN. When a scenario doesn't have the facts it needs, it emits an unknown finding instead of staying silent.

There is no LLM in the risk path. The capability matrix lists that explicitly as not supported. The graph and the rules decide.

The policy gate

Scenarios produce facts. A small policy engine turns them into a decision. This is the built-in default, written as the YAML policy document the engine accepts:

apiVersion: rehearsal.io/v1beta1
kind: RehearsalPolicy
block:
  - 'risk in ["critical","high"]'
  - 'decision == "block"'
  - 'rollback == "unavailable"'
  - 'requiredMissing > 0'
warn:
  - 'risk == "medium"'
  - 'decision == "warn"'
unknownAsBlock: true

Two details matter more than the syntax. First, if you omit unknownAsBlock, it defaults to true; setting it to false is allowed but logged as a warning. Second, an expression the engine can't parse doesn't get skipped. It is rejected when the policy loads, and if one somehow reaches evaluation it produces block. Both of these were bugs I fixed in the latest round of hardening. Before that fix, an omitted unknownAsBlock meant false, and an unrecognised expression simply never matched, so a typo could quietly weaken a policy.

RehearsalRun: the gate as cluster state

For teams that want this inside the cluster, there is an operator built on controller-runtime and a RehearsalRun CRD (rehearsal.io/v1beta1). The example in the repo looks like this:

apiVersion: rehearsal.io/v1beta1
kind: RehearsalRun
metadata:
  name: payments-rollout
  namespace: default
spec:
  baselineRef: baselines/prod.json
  changeRef: changes/payments-v2.json
  # observedRef: observed/prod-after.json
  async: true

The status carries phase, decision, risk, controlPlaneRunId, jobId, evidenceDigest, a specDigest to detect drift, and standard conditions (Accepted, Running, Ready, Failed). Each spec generation gets its own run ID ({namespace}-{name}-{uid8}-g{generation}), so editing a run produces a new rehearsal instead of overwriting the old record.

One design decision started as a real security bug: the CR can never carry the control plane URL or token. An earlier version had a controlPlaneURL field. That meant anyone allowed to create a RehearsalRun could point the operator's bearer token at a server they controlled. The field was removed in v1.5.1, and now the URL and token come only from the operator's Deployment and Secret.

Verifying the prediction against reality

A gate that is never checked against reality slowly turns into folklore. After a deploy (or after a change was applied anyway), rehearsal verify compares the rehearsal report with an observed snapshot. It checks that the report is bound to the same baseline and change digests, that the change was actually applied, and that each predicted scenario is consistent with what was observed. Operator annotations such as "we saw failure X" count only as soft checks. The independent scenario check has to pass on its own, and pods stuck in Pending across the cluster don't count as confirmation. The outcome feeds a calibration step that tracks how accurate each scenario has been.

A content-addressed evidence chain

Every analysis writes an evidence chain:

baseline â†’ change â†’ proposed â†’ report â†’ observed â†’ verification

Each link is a SHA-256 digest computed from the live objects. The report digest is always recomputed from the report body, so if someone edits decision or risk in a stored report and leaves the old digest field alone, verification fails. The chain can be wrapped in a DSSE-style envelope signed with HMAC or Ed25519. I want to be precise here: this is not full Sigstore, and the threat model says plainly that HMAC does not give you non-repudiation.

The Argo CD PreSync hook

The newest piece came from an obvious gap. A CI check only runs if the change goes through CI. A manual sync from the Argo CD UI, or a direct commit to the tracked branch, skips it.

rehearsal argo-presync runs the same impact gate as an Argo CD PreSync Job (argocd.argoproj.io/hook: PreSync). The Job reads the desired manifests, takes a live snapshot, runs the detectors and prints a single JSON decision line. ALLOW and WARN exit 0, so the sync continues. BLOCK exits 3 and UNKNOWN exits 4, and Argo aborts the sync with the live objects untouched. Missing inputs, API errors and timeouts also fail closed.

There is an audited escape hatch for incidents: annotate the Application with rehearsal.io/break-glass: "true" plus a non-empty rehearsal.io/break-glass-reason. The Job logs the override and includes it in the JSON line. Break-glass without a reason is itself treated as a failure.

A real run

Everything below comes from the repo at commit 920ff4f, built locally (rehearsal 1.6.0; the PreSync gate shown later is part of v1.7.0). These are the commands from scripts/e2e_pipeline.sh (the same steps as make e2e), run against the payments fixture from the opening.

git clone https://github.com/justrunme/architecture-rehearsal.git
cd architecture-rehearsal
go build -o bin/rehearsal ./cmd/rehearsal
bash scripts/e2e_pipeline.sh

Snapshot, scoped change, then analysis:

==> 1/5 snapshot k8s (baseline from kubectl List dump)
wrote snapshot out/e2e-pipeline/baseline.json phase=baseline source=kubernetes-manifests (22 nodes)
==> 2/5 change manifests (scoped to payments; removals off)
wrote change out/e2e-pipeline/change.json (removals=false scope_ns=[payments])
  scope OK: 2 patch(es), 0 removals
==> 3/5 analyze (expect exit 3 block)
risk=critical decision=block findings=1 rollback=unknown digest=270e3b4f4ea2e25509461038cafe2e0fe804027a73abce74b77bc3fb6e55e5f9 evidence=out/e2e-pipeline/change-helm-scale-payments-e2e-20261017T133404Z
  analyze blocked as expected

The finding in latest-report.json explains the block with numbers instead of a vague severity label:

"title": "Pod IP capacity exhaustion (CNI / VPC)",
"summary": "Replica delta 20 + surge 6 = 26 IPs needed; available 10 (deficit 16). baseline=14 proposed=34",
"cascade": [
  "baseline_replicas=14",
  "proposed_replicas=34",
  "rollout_surge=6",
  "requested_ips=26 available=10 deficit=16",
  "IP exhaustion â†’ FailedCreatePodSandbox â†’ readiness timeout â†’ Helm pending-install"
],
"recommended_controls": [
  "increase max-pods / prefix delegation",
  "add nodes",
  "reduce maxSurge"
]

The same report includes a coverage block (overall 0.45 on this fixture) and lists its gaps, for example no meta.metric_labels â€” prom scenarios limited. I want the tool to say what it didn't look at.

The fixture also contains a post-deploy dump for the case where the change was applied anyway. Verification passes, with the independent scenario check succeeding apart from the operator annotation:

"name": "scenario:cni-ip-capacity",
"passed": true,
"detail": "meta.cni_ip_available=0 (explicit CNI provider)"
...
"name": "rollout_status:workload/payments/payments-api",
"detail": "rollout_not_converged (consistent with capacity/CNI prediction risk): readyReplicas=6 want=18",
...
  verify OK outcome=verified score=1 digest=72f7dbab61a69655
  decision=block risk=critical failures=['cni-ip-capacity']
e2e pipeline OK: dumpâ†’graphâ†’scoped changeâ†’blockâ†’verify

Signing and verifying the chain:

$ export REHEARSAL_HMAC_SECRET="$(openssl rand -hex 32)"
$ ./bin/rehearsal evidence sign-dsse --chain out/e2e-pipeline/latest-chain.json --out out/e2e-pipeline/evidence-dsse.json
wrote out/e2e-pipeline/evidence-dsse.json verified=true
$ ./bin/rehearsal evidence verify-dsse --envelope out/e2e-pipeline/evidence-dsse.json
signature OK keyId=hmac-env changeId=change-helm-scale-payments-e2e baseline=3035a1f23205 change=e73dd8fe9fa6

The Argo CD fixtures (bash scripts/e2e_argo_impact.sh) show the four outcomes. This is the case I care about most, where the baseline never recorded PDB collection:

==> unknown-pdb-inventory (expect 4)
verdict=UNKNOWN decision=unknown findings=1 confidence=low ...
unknown-pdb-inventory exit=4 expect=4

The generated markdown report says why:

| `pdb-rollout` | **UNKNOWN** | low | PDB inventory status unknown (baseline did not record PDB collection) |

## Missing data

Gate cannot safely PASS while these are missing:

- `pdb-inventory`

Four detectors passed. A binary gate would probably have approved this change. This one refuses.

What has actually been tested

I keep a claim-by-claim EVIDENCE.md that links each claim to a CI run. In short: the policy gate, the evidence chain and DSSE path are covered by the test job. The operator installs via Helm on kind in operator-kind-e2e. The native Argo CD gate runs against real Argo CD Applications on kind in argo-kind-e2e, where a benign Synced app passes and a risky OutOfSync Deployment is blocked.

The PreSync hook runs in argo-presync-kind-e2e with pinned Argo CD v2.14.9. On that fixture set (kind e2e plus an offline matrix), 5 of 5 risky changes were blocked and 0 of 3 safe changes were falsely blocked. The in-cluster hook added roughly 0.9 to 1.0 seconds per sync on a single-node kind cluster. Those are eight cases. That's a regression check. It tells you nothing about field accuracy, and I wouldn't quote it as an accuracy figure.

Limitations, honestly

The blast radius is simulated. It comes from a graph plus a change, not from fault injection against a live cluster. It complements chaos engineering and doesn't replace it.
The model is only as good as the inventory. Fail-closed protects you from false approves caused by missing data. It can't protect you from data that is present but wrong. The repo explicitly does not claim zero false approves on incomplete real-world inventories.
Live collection against arbitrary production clusters is still at the "reference / not yet" stage in the evidence table. CI uses fixtures and kind.
Single-cluster. Everything tested is single-node kind. Multi-cluster federation of evidence stores is out of scope for now.
The scenario catalogue is small. Six rehearsal scenarios and five Argo detectors cover failure classes I have seen often, not everything that can go wrong.
The PreSync hook expects desired manifests in a ConfigMap, which adds wiring work. It shipped in the v1.7.0 release.
No production SLOs yet. There are no published time-to-gate or false-approve-rate numbers beyond the fixture metrics above.
Signing is DSSE-style with HMAC/Ed25519, not a full Sigstore/transparency-log setup.

What's next

Next on my list: more evidence from live collection, more scenarios and detectors (especially around IAM and observability coverage, which score 0 on the main fixture), a less manual way to feed desired state into the PreSync hook, and turning calibration data into something a team can read as "this scenario has been right N out of M times on our clusters".

Try it, and tell me where it's wrong

The repo is Apache-2.0: github.com/justrunme/architecture-rehearsal. The fastest way in is:

make demo && make e2e
make e2e-argo

There is also a longer write-up on the case study page.

I'm especially interested in two kinds of feedback: failure classes from your own incidents that a graph-plus-scenario model should catch, and places where the gate says approve when you think it should have said unknown. Issues and PRs are welcome.

Andrey Lesnikov is a Senior DevOps Platform Engineer based in Halle, Germany, and a 3rd place Company Award recipient at the Cloud Native Rockstar Awards 2026.