DEV Community: Vitaliy Dmitriev

Terraform Applied Successfully. Did the AI Agent's Change Actually Work?

Vitaliy Dmitriev — Tue, 20 Oct 2026 08:31:01 +0000

Terraform Applied Successfully. Did the AI Agent's Change Actually Work?

Part 2 of 3 in "Terraform with an AI co-worker: access, testing, review".

Part 1 is here.

The task looked small: create an S3 bucket, two SQS queues and a service user across four test environments and production. An AI agent was doing the work.

Before deploying, it ran Terraform plans. In two test environments, those plans included other teams' access keys, permission changes and the deletion of two empty S3 folder markers. None of that belonged to our task.

That was the first useful result of testing, before we had deployed anything. Later, the agent would upload real objects, deliberately trigger access denials, and discover that some of its own checks were wrong.

In Part 1, I covered how we restricted the agent's AWS access. This part is about the evidence I wanted before accepting its work as done.

The plan included someone else's work

A Terraform plan previews proposed changes. It doesn't apply them, which makes it a useful first check. It also only covers what the configuration and state let Terraform see, so it isn't a complete inventory of an AWS account.

In our case, a colleague's cleanup had been merged into the repository but hadn't been deployed. A full apply would have removed two empty folder markers, created access keys and permission policies for other teams, and changed existing permission policies.

The folder-marker deletions were intended by their owner. This wasn't a confirmed data-loss scenario. The problem was that nobody deploying our change had reviewed or approved those unrelated changes for this rollout.

They would have gone live as a side effect of our task, at a time their owners hadn't chosen.

We used -target=module.<name> to limit this deployment, then checked the resulting plan against the expected resource addresses and actions. Anything outside that scope, including any deletion, stopped the deployment before approval. The unrelated changes were reported separately.

Targeting needs a caveat: it includes dependencies, and HashiCorp recommends it only for exceptional situations. It doesn't mean "touch exactly this module and nothing else." The resulting plan still needs inspection, and a subsequent full plan is needed to see what remains pending.

There is another distinction worth making. We used terraform plan -lock=false for the initial inspection with read-only credentials. The credentials restricted writes; -lock=false only disabled state locking. It isn't a general safety option. Keep locking enabled for normal deployment work, and don't use an unlocked inspection while another operation may be changing the same state.

Could we prove the bucket rule worked?

The bucket had a rule that mattered to the application.

Each hour, an upstream system uploads data files, followed by a small marker file saying the upload is finished. The marker triggers an S3 notification to an SQS queue. A retry must not overwrite that marker and trigger another object-created event for the same hour.

We enforced conditional writes for markers through a bucket policy. The uploader had to send If-None-Match: *, meaning "create this object only if the key doesn't already exist." Ordinary data files didn't need that header.

Reading the policy back would show that it had been saved. I wanted to know whether it actually enforced those rules.

Before deploying to the real environments, the agent created a throwaway bucket and queue in a test account and exercised the same policy:

Attempt	Expected	Observed
Upload a marker without the required header	`403 AccessDenied`	Refused
Upload a marker with `If-None-Match: *`	Success	Accepted
Upload the same marker again with the header	`412 PreconditionFailed`	Refused
Upload an ordinary data file without the header	Success, no notification	Accepted, no notification observed
Inspect queue messages	Marker notification, separate from AWS's setup test event	One marker notification observed

The agent then removed the resources it had created for the test.

There is an important limit here: preventing marker overwrites doesn't guarantee exactly-once processing. S3 notifications and standard SQS queues can deliver duplicates. Seeing one message in a test doesn't change that contract. The downstream consumer still needs to be idempotent, and we passed that requirement to its developers.

Test with the application's permissions

After each deployment, the agent ran a smoke test using the application's service-user credentials. A successful upload with a privileged deployment identity wouldn't tell us whether the application could do the same thing.

Here is the production output, with names shortened:

identity: user/<app-user>-prod
content folders: input/content/  output/content/
1 put without header (expect AccessDenied): AccessDenied
2 put with If-None-Match (expect OK):        OK
3 duplicate put (expect PreconditionFailed): PreconditionFailed
4 parquet without header (expect OK):        OK
5 other bucket (expect AccessDenied):        AccessDenied
6 input queue: smoke-test events=1 (expect 1)
7 output queue: non-test events=0 (expect 0)
cleanup: OK

Line 5 mattered as much as the successful upload: the identity couldn't access the other bucket we tested. Line 7 checked that the test hadn't produced an event in the wrong queue during the observation period.

The production queues were dedicated to this application, brand new and not yet in use. That made it possible to inspect the test messages without competing with a live consumer. This procedure would need a different design for an active queue.

The test objects used a year-2000 date to make them recognizable, and cleanup removed only the objects created by the test. The fake date was a naming convention, not protection against real processing. The fact that the queues weren't in use was the relevant condition here.

Some of the tests needed debugging too

In the first test environment, uploads initially produced no visible messages. The configuration looked correct, but it took roughly five to eight minutes before we observed notifications flowing.

That was our observation, not proof of a particular AWS root cause or a delay to expect every time. S3 notifications can be delayed, and SQS message counts are approximate. The useful response was bounded waiting and checking the actual messages, rather than immediately changing a configuration that might already be correct.

The agent also produced two false alarms in its safety checks. One treated an empty response as a failure; another expected a field that its query hadn't returned. There were smaller script bugs too: a check inspected the wrong part of the data, and extra command output broke a later step.

A guard that stops on uncertainty can be useful. But if the guard itself is broken, "the deployment was blocked" doesn't tell you much.

We needed to verify that it recognized valid results as well as unexpected ones.

Those script and checking bugs were fixed before the production rollout. AWS timing behaviour remained something the tests had to accommodate.

Lessons learned: what I now ask for before accepting "done"

For this change, the evidence was fairly concrete:

A reviewed plan whose proposed actions matched the task.
Behavioural tests on throwaway resources before deployment.
Smoke tests in every target environment using the application identity.
Negative tests for operations that should be refused.
An explicit account of test limitations, delays and cleanup.

These checks can live in scripts or a testing framework. They don't need an LLM to decide whether AccessDenied matches an expected denial. The agent was useful for investigating the environment, preparing the tests and working through unexpected results. The repeatable comparisons should remain explicit.

The most useful part wasn't the final successful apply. It was discovering what that apply would have included, and then checking that the deployed resources behaved as the application required.

That's the evidence I want alongside an agent's summary.

Next, in Part 3: reviewing the agent's work with independent agents, what those reviewers caught, and why we stop after three rounds.

Technical references: Terraform plan and targeting, S3 conditional-write enforcement, S3 notification delivery, and SQS approximate message counts.

One missing name, a Kubernetes namespace full of restarts

Vitaliy Dmitriev — Fri, 16 Oct 2026 07:40:39 +0000

One missing name, a Kubernetes namespace full of restarts

Compare these two commands:

kubectl delete deployment -n prod
# error: resource(s) were provided, but no name was specified

kubectl rollout restart deployment -n prod
# deployment.apps/api restarted
# deployment.apps/billing restarted
# deployment.apps/checkout restarted
# ...

The first refuses to run. The second attempts to restart every Deployment in the namespace, without confirmation or an explicit --all. It's documented behavior.

That makes this a surprisingly dangerous bit of shell:

kubectl rollout restart deployment $SERVICE -n prod

If SERVICE is empty, the command gets a much bigger audience than intended.

Each Deployment still follows its own update strategy. But with rolling updates, those rollouts can overlap across the namespace. Caches warm up, connections reconnect, and replacement pods compete for capacity together. Per-Deployment rollout settings don't coordinate that load across services.

We wanted to remove this particular footgun without taking away routine troubleshooting access. That led to a Jenkins job, and then to a more interesting question: what should its credential be allowed to do?

RBAC doesn't see your intention

There is no dedicated Kubernetes “restart” API. For a Deployment, kubectl patches the pod template with a timestamp:

spec:
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2026-01-01T12:00:00Z"

The changed template triggers a rollout. Without a name, kubectl lists the Deployments and sends a separate patch for each one.

RBAC authorizes those requests individually. It cannot express “allow any one Deployment, but reject a command that targets several.” A resourceNames allowlist can restrict which Deployments are patchable, but it still permits patching every allowed name.

We removed patch on Deployments from users and provided the restart through Jenkins instead. Users kept their existing scaling and pod-deletion access.

There is an important limit here: our users also retained update on Deployments. They could still change the template with a PUT request. We were preventing an easy accident, not preventing users from modifying workloads.

Make the target explicit

The job requires a resource name, validates its format, checks that the object exists, and takes the namespace from a fixed choice list. The restart path always uses kind/name:

// After validating name and checking ns against the allowed namespaces:
withEnv(["NS=${ns}", "NAME=${name}"]) {
    sh '''
        set -eu
        : "${NS:?Namespace is required}"
        : "${NAME:?Deployment name is required}"
        kubectl get "deployment/$NAME" -n "$NS" -o name
        kubectl rollout restart "deployment/$NAME" -n "$NS"
        kubectl rollout status "deployment/$NAME" -n "$NS" --timeout=180s
    '''
}

The shell rejects an empty name before kubectl runs. Even without that check, deployment/ fails instead of expanding the operation to the namespace.

The single-quoted Groovy script matters too: validated input reaches the shell through environment variables and stays quoted, rather than becoming part of the script text.

One easy permission to miss is watch, which rollout status needs. Otherwise the restart can succeed while the build fails to observe it. A status timeout also doesn't undo the restart; the rollout can continue after the build fails.

The job records who requested the operation, the namespace, and the target in its build history. It also supports scaling to zero and deleting a pod, but those have different failure modes. Neither a rolling update nor a controller-owned pod guarantees zero downtime.

Don't give the restart button cluster-admin

The existing Jenkins deployer credential in our setup had cluster-admin access. Reusing it would have made this a very expensive restart button.

Pipeline validation cannot compensate for that. Anyone who can modify or replay the Pipeline may be able to run different commands with credentials available to it. Jenkins permissions and credential scope matter as much as the form fields.

We used a dedicated service account, bound only in the intended namespaces through RoleBindings. The restart path needs this Deployment rule:

rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "patch"]

The other methods need their own permissions. None needs cluster-admin, Secret reads, or pods/exec.

This credential still permits a namespace-wide Deployment restart. The job limits the normal workflow to one target; Kubernetes permissions limit where a modified workflow could act. With unrestricted Deployment patch, it can also change more than the restart annotation.

That distinction is the useful part of the exercise. We made the everyday operation harder to mistype, while keeping the automation's access narrow enough to inspect and test.

And for a runbook or local script, the smallest useful improvement is just:

kubectl rollout restart "deployment/${SERVICE:?Set SERVICE first}" -n prod

An empty variable should produce an error, not a namespace full of restarts ;)

Resetting an Early 2011 MacBook Pro firmware password with a CH341B

Vitaliy Dmitriev — Wed, 14 Oct 2026 08:43:41 +0000

Resetting an Early 2011 MacBook Pro firmware password with a CH341B

I bought an Early 2011 MacBook Pro on OLX, listed as broken. It turned out to power on, but there was another problem: it had an EFI firmware password.

The laptop came without an SSD or HDD, so there was no previous owner's disk or data to recover. I wanted to get the machine usable again.

That turned into a small hardware project involving a CH341B programmer, an in-circuit clip, flashrom, and a Python script. After reflashing the firmware, the password was gone and I was able to boot macOS.

The hardware and the starting point

My machine was a 15-inch Early 2011 MacBook Pro with a Macronix MX25L6406E flash chip. Its capacity is 8 MiB, or 8,388,608 bytes.

I used a CH341B programmer and an in-circuit clip, so I didn't desolder the chip. For chip access and connection details, I followed this existing EFI firmware-password removal guide. The underlying technique isn't my discovery; my contribution was documenting my setup and adding a small helper for modifying the dump.

I didn't photograph the process before putting the laptop back together, so this is a write-up of my repair, not a complete board-level connection guide. Before connecting a programmer, check the chip's voltage requirements, the programmer's actual supply and signal levels, clip orientation, and the board's power-isolation requirements. A similar-looking programmer or logic board isn't enough to establish compatibility.

Read first, keep the original

Using flashrom on macOS, I read the chip twice:

sudo flashrom -p ch341a_spi \
  -c "MX25L6406E/MX25L6408E" -r dump1.bin

sudo flashrom -p ch341a_spi \
  -c "MX25L6406E/MX25L6408E" -r dump2.bin

The programmer was a CH341B, although flashrom's programmer option is named ch341a_spi.

Then I compared the dumps and kept an original copy:

cmp dump1.bin dump2.bin
shasum -a 256 dump1.bin dump2.bin
cp dump1.bin original.bin

cmp produces no output when the files match. The hashes provide another easy comparison. If the reads differ, stop and investigate before modifying anything.

Matching reads are a useful check, not proof that everything is correct. A consistently bad connection can still produce repeatable output. The expected file size and plausible contents matter too. Keep a separate private backup of the original, beyond the working directory.

Replace bytes, don't delete them

The modification concerns regions identified by the $SVS signature. The important practical detail is that the firmware image must keep its layout: removing bytes shifts everything that follows.

The helper in my gist replaces the selected bytes with 0xFF instead. The output stays the same size as the input.

python3 patch_svs.py dump1.bin modified.bin

The script checks for an 8 MiB input, locates the signatures, prints the proposed ranges and surrounding bytes, and asks for confirmation before creating the output.

There is a limitation worth making explicit: it finds the end of each region by looking for the next run of eight 0xFF bytes. That's a heuristic, not a full parser for the firmware format. The script also processes every matching signature it finds.

So don't treat it as a universal MacBook password-reset tool. Review the reported ranges for the specific image, check that the output size is unchanged, and inspect the differences before writing anything back. An unchanged size alone doesn't prove the modification is correct.

Also, use distinct input and output files. The current helper doesn't prevent you from choosing the original as the output, which would overwrite it.

Flashing and the result

Once the modified image was ready, I reflashed the chip. The write command in my notes is:

sudo flashrom -p ch341a_spi -V \
  -c "MX25L6406E/MX25L6408E" -w modified.bin

Flashrom normally verifies the contents after writing unless verification is disabled. The uppercase -V enables verbose output; it isn't the verification switch.

Writing firmware is the destructive part of this process. Keep the original available and remember that restoring it still depends on being able to communicate reliably with the chip.

In my case, reflashing removed the firmware password and I could boot macOS. That's the result I verified, rather than a claim that every function of the laptop was tested afterward.

What this applies to

This worked on my Early 2011 machine. I haven't established compatibility with other models or firmware revisions. It isn't an Activation Lock bypass or a way to decrypt FileVault-protected data.

Apple's documented procedure for a forgotten Intel Mac firmware password is an in-person service appointment with proof of purchase. The procedure above is an independent hardware-repair approach for a machine I purchased, not Apple's supported recovery workflow.

I've kept the commands and Python helper in this gist. If you're working on a similar repair, the useful starting point is an original dump you can trust, a backup you haven't overwritten, and a clear understanding of what the script proposes to change.

Cover photo: © Raimond Spekking / CC BY-SA 4.0 (via Wikimedia Commons). Reference photograph; not my repair.

I gave an AI agent a Terraform task. It found a credentials problem.

Vitaliy Dmitriev — Wed, 14 Oct 2026 08:31:37 +0000

I gave an AI agent a Terraform task. It found a credentials problem.

Part 1 of 3 in "Terraform with an AI co-worker: access, testing, review".

The ticket was ordinary: create an S3 bucket, two SQS queues and an IAM user across five environments, using the existing Terraform setup.

I gave it to an AI agent. It prepared the code and ran its checks, but deployment hit a problem. The AWS profiles Terraform expected weren't available on the machine. Investigating those profiles led to something more interesting than the ticket itself: permanent administrator keys shared with automated jobs.

The agent stopped and reported the problem. It didn't ask me to paste the keys into the chat or start replacing credentials that other workloads depended on.

That pause was probably the most useful part of this stage of the work.

This series follows the same Terraform task through three stages: sorting out access, testing the change, and reviewing the work. This first part is about the credentials the agent could use and what we could verify before allowing a deployment.

The profiles were familiar. The credentials behind them were less reassuring.

Our Terraform configuration expected one AWS profile per environment, with names like stage-user-terraform and prd-user-terraform. Each pointed to an IAM user with long-lived access keys and administrator permissions. The Terraform state was stored in S3 in the AWS organization's management account.

The keys weren't on the agent's machine. Using read-only queries, it found that the existing keys had been used that day by automated data jobs. In two environments, the users already had two keys each, which is the AWS limit per IAM user.

Creating another key for those users would first require deleting one. That would turn a small infrastructure ticket into a credential rotation for workloads outside its scope.

There were several tempting shortcuts: copy an existing secret from wherever it was stored, replace a key to make room, or use a more privileged identity because it happened to work. None belonged in this task.

Also, AWS doesn't let you retrieve an existing secret access key after creation. Inspecting key metadata and usage is different from having the credential itself.

The rules helped. They didn't remove the permissions.

The agent had standing instructions:

Read-only investigation was pre-approved; important changes needed human approval.
It was not to delete or modify resources it hadn't created without explicit authorization.
Missing credentials and ambiguous ownership were reasons to stop and ask.
A refused operation was not permission to find another way around the refusal.

These rules explain the behavior I wanted. They don't make an administrator role read-only.

That distinction matters here because the SSO role we eventually used still had administrator permissions. We improved credential handling, but the agent's restraint and the approval workflow remained part of the boundary. A dedicated role with narrower permissions is the next step.

There was also a small local example. A Terraform run left a generated file in the repository, and the safety layer blocked the agent's attempt to delete it. The agent moved it into a holding directory and reported the location, leaving final deletion to me.

That preserved the file, but I wouldn't present it as proof that the guardrail worked perfectly. Moving a file also changes the working tree. Whether that is an acceptable alternative depends on what the restriction is meant to protect. The useful detail was that the agent reported the alternative instead of silently removing the file another way.

Keep the profile names, change how we authenticate

The agent proposed four options. We chose one that left the Terraform code and the shared keys alone: keep the expected profile names, but resolve them through personal AWS IAM Identity Center logins, commonly called AWS SSO.

Those logins provide temporary role credentials instead of another permanent key. We put the profiles in a separate AWS config file, selected explicitly when running Terraform.

Here is an illustrative version, with placeholders:

# ~/.aws/config-terraform
# Profile configuration only, no secret access keys.
[sso-session company]
sso_start_url = https://<your-sso-portal>/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile stage-user-terraform]
sso_session = company
sso_account_id = <stage-account-id>
sso_role_name = <your-permission-set>
region = us-east-1

# Existing backend profile name; this is NOT the AWS root user.
[profile root-user-terraform]
sso_session = company
sso_account_id = <management-account-id>
sso_role_name = <your-permission-set>
region = us-east-1

The root-user-terraform name is unfortunate, but it was already what the backend expected. Here it refers to an SSO role in the management account, not root credentials.

We logged in with the separate configuration and selected it inside a subshell for the Terraform commands. An investigation command looks like this:

AWS_CONFIG_FILE="$HOME/.aws/config-terraform" \
  aws sso login --sso-session company

(
  export AWS_CONFIG_FILE="$HOME/.aws/config-terraform"
  export AWS_PROFILE=root-user-terraform
  cd infra || exit 1
  terraform workspace select stage || exit 1
  terraform plan -lock=false -target=module.my-change
)

The paths, workspace and module address are examples. This assumes an initialized backend and an AWS provider configuration that explicitly selects the environment profile. The backend and provider authenticate separately; setting AWS_PROFILE alone doesn't separate their access.

The two plan flags also need context. We used -lock=false during the investigation to avoid a remote lock write, and targeting to inspect the ticket's scope. These are not defaults I'd use for deployment. An unlocked plan can become stale if another operation changes the state, and targeting can omit changes outside the selected dependency graph.

The separate config file reduces accidental profile use, and the subshell keeps these environment settings from remaining in my interactive shell. Neither prevents a process with access to the file and cached SSO session from selecting the profile again. Exiting the subshell doesn't revoke the session.

Before using this pattern, check credential sources and precedence too. Existing environment credentials or static credentials under the same profile names can interfere with the intended login. The Terraform S3 backend and the AWS provider also need versions that support your SSO configuration.

So what did we actually improve? We avoided distributing another long-lived secret, left the pipeline credentials untouched, and used personal role sessions with better attribution than a shared IAM user. The remaining admin permissions still deserved attention.

Could it write the state? We gathered evidence without trying a write.

Reading infrastructure was only part of the access question. An eventual apply also needed permission to update the state in the management account and use the configured locking mechanism.

We kept this investigation read-only:

Run a Terraform plan with locking disabled for each environment we could authenticate to. This checked state reads and the AWS reads needed by that plan.
Inspect the state bucket's policy. There was no bucket policy.
Use the IAM policy simulator to evaluate the relevant S3 actions against the bucket and object resources. The simulated result was allowed.

The plan demonstrated that the reads it actually performed worked. The simulation supplied evidence for write permissions, but it wasn't an end-to-end write test. No bucket policy also doesn't, by itself, mean writes are allowed.

AWS documents limitations in the simulator, including resource-policy simulation for IAM roles and differences from live request evaluation. A result depends on the policies and context being evaluated. I wouldn't turn allowed into a promise that the eventual apply will succeed.

An independent reviewer agent rechecked the evidence and inspected encryption and organization controls. The reported bucket encryption didn't require a customer-managed KMS key. It also confirmed that SCPs don't restrict identities in the management account. That last point makes the account more sensitive, not less.

Locking permissions need their own check. S3 lock files require object permissions on the lock path; a DynamoDB-backed lock needs the relevant table permissions instead. An unlocked plan doesn't exercise either.

Two environments had expired SSO logins and weren't checked. We reported them as unverified rather than extending the result from the other environments.

Access wasn't the only thing blocking an apply

The real plan also exposed a mismatch between the shared Terraform configuration and the deployed environments. A full apply in three test environments would have included other teams' changes and deleted existing storage objects, well outside this ticket.

We chose a targeted deployment approach for this change and left the broader reconciliation as separate work. That didn't make the plan automatically safe: -target can include dependencies and leave other changes unapplied. I'll return to the plan's scope and the testing in Part 2.

Getting the right credentials was necessary. It wasn't permission to apply everything those credentials could reach.

What this changed for me

The agent prepared the change and investigated the access problem. A person still runs the apply.

We didn't rotate shared keys, create another administrator key or change the Terraform code just to get past the missing profiles. We found a way to use temporary personal credentials, while being explicit that those credentials still carried too much power for a narrowly scoped task.

For a team starting with AI-assisted infrastructure work, that is a useful first question: what identity will the agent use, and what can that identity actually do?

In this case, the missing login exposed an access arrangement worth revisiting. I would rather get that finding before deployment than discover the dependency on shared admin keys halfway through replacing one.

This is Part 1 of "Terraform with an AI co-worker: access, testing, review".

Part 2 will cover testing: using a temporary bucket and a real upload to check the change, cleaning up the test resources, and handling the unrelated changes in the plan.
Part 3 will cover review: what independent AI reviewers checked, what they caught, and which conclusions still needed human judgement.

Technical references

I let an AI agent make an infrastructure change

Vitaliy Dmitriev — Wed, 07 Oct 2026 07:59:37 +0000

I let an AI agent make an infrastructure change

An AI agent recently took an infrastructure ticket from "please create these queues" to deployed and tested.

It also:

found a bug I would probably have shipped;
briefly broke a consumer;
confidently gave me the wrong explanation for some infrastructure versus code drift.

All three were useful.

The first one showed me what agents can do.

The other two showed me what they need before you let them touch infrastructure.

The ticket

Typical BAU ticket:

Create a few message queues, configure dead-lettering and permissions, and make them available to the application.

Normally this is a small DevOps task.

But infrastructure changes are rarely isolated. The broker was shared, there were several environments, existing applications depended on it, and some configuration was generated from code.

So I decided not to do the change myself.

I gave it to an AI coding agent running in a terminal.

Its permissions were deliberately asymmetric:

It could read almost everything. It could change almost nothing without my approval.

My rules were simple:

production is read-only;
non-production changes require approval;
changes must be reversible;
don't guess when something is ambiguous.

The agent followed a fixed workflow:

explore
  ↓
plan + pre-mortem
  ↓
independent review
  ↓
questions to human
  ↓
implement
  ↓
non-production rollout
  ↓
test
  ↓
review
  ↓
handoff

This turned out to matter more than the model itself.

The first useful thing: it didn't start coding

The agent first explored the environment.

That immediately found several things I would have been tempted to skip.

The ticket was ambiguous

The non-production broker was shared by several environments.

The queue names in the ticket didn't contain an environment identifier, so blindly creating them would have made different test environments share the same queues.

The agent stopped and asked which environments actually needed the change.

Good catch.

There was already a pattern

Other team members and I had implemented something very similar before.

Instead of inventing a new configuration, the agent found the existing implementation and used it as a reference.

This is one of the least glamorous but most valuable things an agent can do:

look around before inventing things.

It found a potential time bomb

The planned queue configuration had a dangerous default. Under the expected failure scenario, messages could accumulate in memory while the consumer was unavailable. Enough messages could eventually trigger the broker's memory protection and affect unrelated applications. The agent suggested adding a size limit and changing the storage policy.

That was useful.

But I didn't trust the suggestion just because it sounded reasonable.

I asked for an independent review.

The second agent found the real problem

A separate agent reviewed the plan. Its first finding was marked CRITICAL. There was a regular expression containing an escaped dot.

Something like:

amq\.default

Perfectly valid regex. Except this value eventually ended up inside JSON. And \. isn't a valid JSON escape.

The interesting part was when this would fail. Not during deployment. Not during the usual validation.

The broker only read that particular configuration during a restart.

So the change could have been deployed successfully, tested successfully, and then caused a failure weeks later when somebody restarted a node.

That's exactly the kind of infrastructure bug I want automation to catch.

The fix was tiny.

The important change was in the process:

don't validate the template. Validate the actual artefact the system will consume.

Render it. Parse it. Do that for every environment.

The reviewer also found a few other problems in the planned tests and configuration.

This was a good reminder that an agent doesn't have to produce the final answer to be useful.

Sometimes its best job is simply:

"Here are five things you haven't thought about."

Then we changed the infrastructure

After I answered its questions, the agent implemented the change. It didn't import an entire configuration file.

It made targeted changes:

create these objects
change these permissions
leave everything else alone

That sounds obvious, but it's an important property for infrastructure automation.

A command that says "make the whole configuration look like this" can have a very different blast radius from:

create X
create Y
modify Z

Then it tested the change using the real application identity.

It tested the happy path:

publish
→ consume
→ fail
→ retry
→ retry
→ retry
→ dead-letter

It also tested the negative cases:

Could the application access things it shouldn't?
Could it create things it shouldn't?

The answer needed to be no.

And then came one of my favourite tests. The agent deliberately filled the queue until the configured limit was reached.

If you configure a limit but never actually hit it, you don't really know whether the limit works. A limit you haven't triggered is still partly a theory.

Then I let it restart the nodes

The infrastructure change also needed a rolling restart.

Before doing that, the agent checked for configuration differences between the repository and the live system. It also checked whether there were messages that could be affected by the restart. There was one queue containing messages that would not survive the restart.

It asked, and I approved it.

The restart began, and everything looked great: all nodes were healthy, replicas were synchronized, no broker alarms, the dashboard was green.

Then the agent found a problem.

The dashboard was green. The application wasn't.

One test consumer had stopped consuming, while its Pod was healthy.

The cluster was healthy, the monitoring dashboard was mostly green, BUT the application was still broken.

The restart had caused the application to reconnect and redeclare its queue. The queue already existed with slightly different settings from what the application expected.

The broker rejected the declaration and the application stopped consuming, while its health check didn't notice.

This lasted for about ten minutes. No customer impact, because it was a test environment.

But it exposed a very important problem with the original verification:

I was checking server health, not client health.

The agent had done what I asked, the cluster was healthy, but that wasn't enough. :(

So we changed the procedure - before every disruptive operation, we now capture the important client-side state:

consumers
connections
restart counts

After the operation, we compare it. Cluster health tells you that the cluster is healthy, but it doesn't tell you that your applications are happy.

Those are two different questions.

One more miss: the agent got drift wrong

Earlier, the agent had found three broker policies present in the live system but missing from Git. It confidently called them manual drift. I believed it and even warned the team.

Later, I asked it to check Git history. Turns out two policies had been in the repo, but were removed months earlier. The broker's config import added settings without deleting old ones, so the policies stayed live. Worse, those two policies were still protecting dead-letter queues. Removing them as “drift” would have made things less reliable.

Only one policy was actually created manually. The agent had checked the current repo, skipped its history, and filled the gap with a plausible explanation.

Plausible isn't verified. Now any drift claim needs a history check and evidence before anyone acts on it.

So we changed the rules

Each failure became a new rule.

Problem	New rule
A restart broke a consumer	Notify the human immediately when user-facing behaviour changes
Cluster was healthy but client was broken	Compare client-side state before and after disruptive operations
"Drift" was based on current Git state only	Check Git history before calling something drift
Review repeated facts from the plan	Reviewers must independently verify important facts
A plausible statement had no evidence	Every important fact must have a command, query or source behind it
A configuration file could fail only after restart	Validate the exact artefact consumed by the system

This is probably the most important part of the whole experiment.

The agent is allowed to be wrong.

The system around the agent shouldn't make the same mistake twice.

The workflow itself now contains these checks, so I don't have to remember them next time.

What this means for a business

If you're thinking about using AI to automate DevOps or infrastructure work, I wouldn't start by giving an agent production access. I'd start much more boringly.

1. Give it read access first

Let it investigate.

Let it correlate logs and metrics.

Let it find configuration.

Let it prepare a change.

You can learn a lot about the agent without giving it the ability to break anything.

2. Gate writes with credentials

This is important.

Don't rely on:

"Don't touch production."

That's a prompt.

Instead:

The production credential literally cannot modify production.

That's a control.

3. Keep approval where the blast radius begins

The agent can prepare the change.

The human approves it.

You can gradually automate more once you understand the failure modes.

4. Make the agent ask questions efficiently

One message with five numbered questions is useful.

Five separate interruptions are annoying.

A good agent should come back with something like:

1. These four environments need the queues. Agree?
2. Existing pattern A looks closest. Use it?
3. This policy may affect memory usage. Apply it?
4. One queue contains messages that won't survive restart. Proceed?

That turns the human into a decision maker instead of a human API.

5. Turn mistakes into guardrails

This is where I think agentic automation becomes really interesting.

A mistake shouldn't just produce:

"The AI made a mistake."

It should produce:

"What rule can we add so neither the AI nor the next engineer makes this mistake again?"

That's how the system gets better.

What actually changed for me

The agent didn't replace the DevOps engineer.

It changed what the DevOps engineer was doing.

Instead of spending the morning:

read ticket
find config
search Git
check cluster
write YAML
run commands
test
write documentation

I spent much more of the time asking:

What do we actually need to change?

What could go wrong?

How will we know it worked?

What must never be allowed?

Those are much more valuable questions to automate around.

And there is an interesting business implication here.

You don't necessarily need an AI that can autonomously run your entire infrastructure. You need an AI that can take a well-defined piece of infrastructure work and move it through a safe, repeatable process, while escalating the decisions that actually require human judgement.

That's a much smaller problem.

And, in my experience, a much more useful one.

If you're starting with AI in DevOps

Three habits are worth stealing even if you never use an AI agent:

Prove your limits. If you configure a timeout, retry count or size limit, trigger it deliberately and see what happens.
Check clients, not only servers. "The cluster is healthy" does not mean "the application works."

The agent didn't replace judgement. It made the places where judgement was missing much easier to see.

And that's probably the part of this experiment I found most valuable.

I'm experimenting with AI-assisted infrastructure automation, particularly around DevOps and on-prem environments.

Would you trust an AI agent to make a production infrastructure change if the permissions, approval gates and rollback path were designed correctly?

I'm curious where other teams draw that line.

Don't put an LLM where a bash script will do

Vitaliy Dmitriev — Tue, 06 Oct 2026 08:58:11 +0000

Don't put an LLM where a bash script will do

Or: how I almost gave a language model a login, a service account and a cron job to check four numbers.

Recently I was debugging a production Prometheus exporter that was restarting during the night. It wasn't crashing in the usual sense. It was getting killed by Kubernetes because its liveness probe timed out.

The reason was fairly simple: the exporter depended on another service that became slow when the nightly jobs started. The probe had a short timeout, so Kubernetes eventually decided the exporter was dead and restarted it.

The fix was boring: increase the probe timeout, adjust the monitoring timeout, deploy the change.

The interesting question was what happens after the deployment.

The fix only really matters at 3 a.m. That's when the nightly load happens. I didn't want to wake up at 3 a.m. just to check whether the exporter survived. So I needed some kind of automated verification.

The obvious AI solution

I had been using an AI agent during the investigation and it was actually quite useful. It helped correlate the restarts with the nightly load, look through metrics and logs, find a problem with one of my assumptions, and prepare the ticket and merge requests.

So the next idea seemed reasonable:

Run the agent every morning for a few days and ask it whether the fix worked.

The agent could query Prometheus, look at the deployment and send me a message. Then I started thinking about what would actually be required to run an agent unattended.

It would need:

a dedicated non-root user;
its own authentication to the model provider;
read-only access to the monitoring system;
some kind of K8S credentials if it needed its data;
a restricted tool set;
a timeout and a way to stop it;
logs;
and probably some tests around the whole thing.

That's a lot of infrastructure for a question that turned out to be:

Did the exporter restart?
Was it up during the whole observation period?
What was the maximum successful scrape duration?
Did the alert fire?

That's four PromQL queries and four comparisons.

At that point I stopped.

The boring solution

I wrote a shell script instead. It was around 80 lines, most of which were notification and error handling. The actual check was tiny.

Something like this:

query() {
    curl -sk --max-time 25 -G "$PROM/api/v1/query" \
        --data-urlencode "query=$1" |
        jq -r '
            select(.status == "success") |
            .data.result[0].value[1] // "none"
        '
}

check() {
    local name="$1"
    local query="$2"
    local condition="$3"

    local value
    value=$(query "$query") || {
        errors+=("$name: query failed")
        return
    }

    [[ "$value" == "none" ]] && {
        errors+=("$name: no data")
        return
    }

    jq -en --arg v "$value" \
        '($v | tonumber) as $v | '"$condition" >/dev/null ||
        failures+=("$name = $value")
}

check restarts \
    '...' \
    '$v == 0'

check uptime \
    '...' \
    '$v == 1'

check scrape_time \
    '...' \
    '$v < 25'

The real script had a few more checks and notification handling, but the principle was the same.

There were three possible results:

CONFIRMED

Everything was within the expected limits.

No notification during the observation period. On the last day, send one short confirmation.

REGRESSED

Something went wrong.

Send me a message with the relevant result and the rollback procedure.

INCONCLUSIVE

The monitoring query failed, there was no data, or the thing being checked no longer existed.

Send a message too.

This last case is important:

Silence should mean "checked and everything is fine", not "the checker failed".

So I also wrote one JSONL record for every run. Nothing fancy, just enough to know that the check actually happened.

One useful detail

The script also checked that the deployment being verified was still the one I intended to verify.

I used an identifier associated with the newly deployed workload rather than simply asking Prometheus about "the exporter".

That matters because deployments can change while your verification job is waiting.

If somebody rolls back or deploys another version, the original workload disappears. In that case the script reports INCONCLUSIVE instead of accidentally verifying the new deployment.

It's a small detail, but it prevents a particularly annoying kind of false positive.

systemd instead of cron

I could have used cron, but systemd made the rest of this surprisingly simple.

The service looked roughly like this:

[Unit]
ConditionPathExists=!/etc/example/PAUSE

[Service]
Type=oneshot
User=example-verify
ExecStart=/usr/local/libexec/example-verify/verify.sh
TimeoutStartSec=300

StateDirectory=example-verify
LoadCredential=notifier.env:/path/to/notifier.env

ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes

And the timer contained the few dates on which I wanted the check to run:

[Timer]
OnCalendar=...
OnCalendar=...
OnCalendar=...

Persistent=true

There were a couple of nice things here.

LoadCredential= lets systemd provide the notification credential to the service without copying it into the script's environment or configuration.
ConditionPathExists=!PAUSE gives me a simple kill switch. Create the file and systemd doesn't start the service.

And because the timer had a finite number of scheduled runs, it didn't become another forgotten cron job that would still be running six months later ;)

There was also a small systemd gotcha. I initially used:

RuntimeMaxSec=300

because that's what I wanted: kill the process after five minutes.

Turns out RuntimeMaxSec doesn't apply to Type=oneshot services.

The correct option here was:

TimeoutStartSec=300

systemd-analyze verify happily caught it. It's worth running, costs nothing.

Where the LLM actually helped

This isn't an anti-LLM story, on the contrary the LLM was useful during the investigation.

The interesting part was figuring out what should be checked.

For example, one of the initial metrics I was looking at seemed to show that the exporter had enough headroom, while second review showed that this conclusion was wrong.

The apparent maximum duration was based on successful scrapes. When a scrape exceeds the timeout, it can fail instead of giving you a nice metric saying "this scrape took 27 seconds". So a graph showing a maximum of 9 seconds doesn't necessarily mean the exporter never took longer than 9 seconds. It can mean that 9 seconds was the longest successful scrape.

The failed scrapes were the important part. That kind of investigation is where an LLM is useful:

correlate different signals;
suggest what to look at;
challenge assumptions;
generate queries;
review a proposed solution;
write the boring documentation and ticket text.

My current rule

I ended up with a fairly simple rule:

If the pass/fail condition can be written as a comparison, use a script. / If you still need to figure out what should be compared, an LLM can be useful.

The important part is not "bash good, AI bad". It's rather about using the right amount of machinery.

An LLM is useful when there is ambiguity and you need judgement. A script is useful when the decision has already been made and needs to be repeated reliably (not "re-decided each and every time").

In this case, using an agent would have meant building a small security boundary around a system whose job was basically:

query
query
query
query
compare
notify

The shell script was easier to understand, easier to secure and easier to debug.

And it was considerably cheaper ;)

One last surprise

After three nights the check reported exactly what I wanted: no restarts and scrape times comfortably below the new timeout. So I deleted the timer, the service, the script and the temporary user.

Except I didn't.

The cleanup command appeared to run successfully but didn't remove the files. The reason was my shell configuration.

My interactive shell had an rm -i alias. The non-interactive environment inherited it. Without a terminal to answer the confirmation prompt, rm -i effectively did nothing.

That was a good reminder that automation doesn't necessarily run in a clean environment. Your shell configuration is part of the environment too.

If you use interactive-only aliases, make sure they are actually limited to interactive shells:

if [[ $- == *i* && -z ${AGENT_SHELL:-} ]]; then
    alias rm='rm -i'
fi

Use whatever environment variable or detection mechanism makes sense for your agent.

The broader lesson was probably the same as the one from the beginning:

Use the clever tool to figure out what needs to be checked. Use the boring tool to check it.