DEV Community: Vitaliy Dmitriev

I let an AI agent make an infrastructure change

Vitaliy Dmitriev — Wed, 07 Oct 2026 07:59:37 +0000

I let an AI agent make an infrastructure change

An AI agent recently took an infrastructure ticket from "please create these queues" to deployed and tested.

It also:

found a bug I would probably have shipped;
briefly broke a consumer;
confidently gave me the wrong explanation for some infrastructure versus code drift.

All three were useful.

The first one showed me what agents can do.

The other two showed me what they need before you let them touch infrastructure.

The ticket

Typical BAU ticket:

Create a few message queues, configure dead-lettering and permissions, and make them available to the application.

Normally this is a small DevOps task.

But infrastructure changes are rarely isolated. The broker was shared, there were several environments, existing applications depended on it, and some configuration was generated from code.

So I decided not to do the change myself.

I gave it to an AI coding agent running in a terminal.

Its permissions were deliberately asymmetric:

It could read almost everything. It could change almost nothing without my approval.

My rules were simple:

production is read-only;
non-production changes require approval;
changes must be reversible;
don't guess when something is ambiguous.

The agent followed a fixed workflow:

explore
  ↓
plan + pre-mortem
  ↓
independent review
  ↓
questions to human
  ↓
implement
  ↓
non-production rollout
  ↓
test
  ↓
review
  ↓
handoff

This turned out to matter more than the model itself.

The first useful thing: it didn't start coding

The agent first explored the environment.

That immediately found several things I would have been tempted to skip.

The ticket was ambiguous

The non-production broker was shared by several environments.

The queue names in the ticket didn't contain an environment identifier, so blindly creating them would have made different test environments share the same queues.

The agent stopped and asked which environments actually needed the change.

Good catch.

There was already a pattern

Myself and other team members had implemented something very similar before.

Instead of inventing a new configuration, the agent found the existing implementation and used it as a reference.

This is one of the least glamorous but most valuable things an agent can do:

look around before inventing things.

It found a potential time bomb

The planned queue configuration had a dangerous default. Under the expected failure scenario, messages could accumulate in memory while the consumer was unavailable. Enough messages could eventually trigger the broker's memory protection and affect unrelated applications. The agent suggested adding a size limit and changing the storage policy.

That was useful.

But I didn't trust the suggestion just because it sounded reasonable.

I asked for an independent review.

The second agent found the real problem

A separate agent reviewed the plan. Its first finding was marked CRITICAL. There was a regular expression containing an escaped dot.

Something like:

amq\.default

Perfectly valid regex. Except this value eventually ended up inside JSON. And \. isn't a valid JSON escape.

The interesting part was when this would fail. Not during deployment. Not during the usual validation.

The broker only read that particular configuration during a restart.

So the change could have been deployed successfully, tested successfully, and then caused a failure weeks later when somebody restarted a node.

That's exactly the kind of infrastructure bug I want automation to catch.

The fix was tiny.

The important change was in the process:

don't validate the template. Validate the actual artefact the system will consume.

Render it. Parse it. Do that for every environment.

The reviewer also found a few other problems in the planned tests and configuration.

This was a good reminder that an agent doesn't have to produce the final answer to be useful.

Sometimes its best job is simply:

"Here are five things you haven't thought about."

Then we changed the infrastructure

After I answered its questions, the agent implemented the change. It didn't import an entire configuration file.

It made targeted changes:

create these objects
change these permissions
leave everything else alone

That sounds obvious, but it's an important property for infrastructure automation.

A command that says "make the whole configuration look like this" can have a very different blast radius from:

create X
create Y
modify Z

Then it tested the change using the real application identity.

It tested the happy path:

publish
→ consume
→ fail
→ retry
→ retry
→ retry
→ dead-letter

It also tested the negative cases:

Could the application access things it shouldn't?
Could it create things it shouldn't?

The answer needed to be no.

And then came one of my favourite tests. The agent deliberately filled the queue until the configured limit was reached.

If you configure a limit but never actually hit it, you don't really know whether the limit works. A limit you haven't triggered is still partly a theory.

Then I let it restart the nodes

The infrastructure change also needed a rolling restart.

Before doing that, the agent checked for configuration differences between the repository and the live system. It also checked whether there were messages that could be affected by the restart. There was one queue containing messages that would not survive the restart.

So it asked me andI approved it.

The restart began, and everything looked great: all nodes were healthy, replicas were synchronized, no broker alarms, the dashboard was green.

Then the agent found a problem.

The dashboard was green. The application wasn't.

One test consumer had stopped consuming, while it's Pod was healthy.

The cluster was healthy, the monitoring dashboard was mostly green, BUT the application was still broken.

The restart had caused the application to reconnect and redeclare its queue. The queue already existed with slightly different settings from what the application expected.

The broker rejected the declaration and the application stopped consuming, while its health check didn't notice.

This lasted for about ten minutes. No customer impact, because it was a test environment.

But it exposed a very important problem with the original verification:

I was checking server health, not client health.

The agent had done what I asked, the cluster was healthy, but that wasn't enough. :(

So we changed the procedure - before every disruptive operation, we now capture the important client-side state:

consumers
connections
restart counts

After the operation, we compare it. Cluster health tells you that the cluster is healthy, but it doesn't tell you that your applications are happy.

Those are two different questions.

So we changed the rules

Each failure became a new rule.

Problem	New rule
A restart broke a consumer	Notify the human immediately when user-facing behaviour changes
Cluster was healthy but client was broken	Compare client-side state before and after disruptive operations
"Drift" was based on current Git state only	Check Git history before calling something drift
Review repeated facts from the plan	Reviewers must independently verify important facts
A plausible statement had no evidence	Every important fact must have a command, query or source behind it
A configuration file could fail only after restart	Validate the exact artefact consumed by the system

This is probably the most important part of the whole experiment.

The agent is allowed to be wrong.

The system around the agent shouldn't make the same mistake twice.

The workflow itself now contains these checks, so I don't have to remember them next time.

What this means for a business

If you're thinking about using AI to automate DevOps or infrastructure work, I wouldn't start by giving an agent production access. I'd start much more boringly.

1. Give it read access first

Let it investigate.

Let it correlate logs and metrics.

Let it find configuration.

Let it prepare a change.

You can learn a lot about the agent without giving it the ability to break anything.

2. Gate writes with credentials

This is important.

Don't rely on:

"Don't touch production."

That's a prompt.

Instead:

The production credential literally cannot modify production.

That's a control.

3. Keep approval where the blast radius begins

The agent can prepare the change.

The human approves it.

You can gradually automate more once you understand the failure modes.

4. Make the agent ask questions efficiently

One message with five numbered questions is useful.

Five separate interruptions are annoying.

A good agent should come back with something like:

1. These four environments need the queues. Agree?
2. Existing pattern A looks closest. Use it?
3. This policy may affect memory usage. Apply it?
4. One queue contains messages that won't survive restart. Proceed?

That turns the human into a decision maker instead of a human API.

5. Turn mistakes into guardrails

This is where I think agentic automation becomes really interesting.

A mistake shouldn't just produce:

"The AI made a mistake."

It should produce:

"What rule can we add so neither the AI nor the next engineer makes this mistake again?"

That's how the system gets better.

What actually changed for me

The agent didn't replace the DevOps engineer.

It changed what the DevOps engineer was doing.

Instead of spending the morning:

read ticket
find config
search Git
check cluster
write YAML
run commands
test
write documentation

I spent much more of the time asking:

What do we actually need to change?

What could go wrong?

How will we know it worked?

What must never be allowed?

Those are much more valuable questions to automate around.

And there is an interesting business implication here.

You don't necessarily need an AI that can autonomously run your entire infrastructure. You need an AI that can take a well-defined piece of infrastructure work and move it through a safe, repeatable process, while escalating the decisions that actually require human judgement.

That's a much smaller problem.

And, in my experience, a much more useful one.

If you're starting with AI in DevOps

Three habits are worth stealing even if you never use an AI agent:

Prove your limits. If you configure a timeout, retry count or size limit, trigger it deliberately and see what happens.
Check clients, not only servers. "The cluster is healthy" does not mean "the application works."

The agent didn't replace judgement. It made the places where judgement was missing much easier to see.

And that's probably the part of this experiment I found most valuable.

I'm experimenting with AI-assisted infrastructure automation, particularly around DevOps and on-prem environments.

Would you trust an AI agent to make a production infrastructure change if the permissions, approval gates and rollback path were designed correctly?

I'm curious where other teams draw that line.

Don't put an LLM where a bash script will do

Vitaliy Dmitriev — Tue, 06 Oct 2026 08:58:11 +0000

Don't put an LLM where a bash script will do

Or: how I almost gave a language model a login, a service account and a cron job to check four numbers.

Recently I was debugging a production Prometheus exporter that was restarting during the night. It wasn't crashing in the usual sense. It was getting killed by Kubernetes because its liveness probe timed out.

The reason was fairly simple: the exporter depended on another service that became slow when the nightly jobs started. The probe had a short timeout, so Kubernetes eventually decided the exporter was dead and restarted it.

The fix was boring: increase the probe timeout, adjust the monitoring timeout, deploy the change.

The interesting question was what happens after the deployment.

The fix only really matters at 3 a.m. That's when the nightly load happens. I didn't want to wake up at 3 a.m. just to check whether the exporter survived. So I needed some kind of automated verification.

The obvious AI solution

I had been using an AI agent during the investigation and it was actually quite useful. It helped correlate the restarts with the nightly load, look through metrics and logs, find a problem with one of my assumptions, and prepare the ticket and merge requests.

So the next idea seemed reasonable:

Run the agent every morning for a few days and ask it whether the fix worked.

The agent could query the Prometheus, look at the deployment and send me a message. Then I started thinking about what would actually be required to run an agent unattended.

It would need:

a dedicated non-root user;
its own authentication to the model provider;
read-only access to the monitoring system;
some kind of K8S credentials if it needed its data;
a restricted tool set;
a timeout and a way to stop it;
logs;
and probably some tests around the whole thing.

That's a lot of infrastructure for a question that turned out to be:

Did the exporter restart?
Was it up during the whole observation period?
What was the maximum successful scrape duration?
Did the alert fire?

That's four PromQL queries and four comparisons.

At that point I stopped.

The boring solution

I wrote a shell script instead. It was around 80 lines, most of which were notification and error handling. The actual check was tiny.

Something like this:

query() {
    curl -sk --max-time 25 -G "$PROM/api/v1/query" \
        --data-urlencode "query=$1" |
        jq -r '
            select(.status == "success") |
            .data.result[0].value[1] // "none"
        '
}

check() {
    local name="$1"
    local query="$2"
    local condition="$3"

    local value
    value=$(query "$query") || {
        errors+=("$name: query failed")
        return
    }

    [[ "$value" == "none" ]] && {
        errors+=("$name: no data")
        return
    }

    jq -en --arg v "$value" \
        '($v | tonumber) as $v | '"$condition" >/dev/null ||
        failures+=("$name = $value")
}

check restarts \
    '...' \
    '$v == 0'

check uptime \
    '...' \
    '$v == 1'

check scrape_time \
    '...' \
    '$v < 25'

The real script had a few more checks and notification handling, but the principle was the same.

There were three possible results:

CONFIRMED

Everything was within the expected limits.

No notification during the observation period. On the last day, send one short confirmation.

REGRESSED

Something went wrong.

Send me a message with the relevant result and the rollback procedure.

INCONCLUSIVE

The monitoring query failed, there was no data, or the thing being checked no longer existed.

Send a message too.

This last case is important:

Silence should mean "checked and everything is fine", not "the checker failed".

So I also wrote one JSONL record for every run. Nothing fancy, just enough to know that the check actually happened.

One useful detail

The script also checked that the deployment being verified was still the one I intended to verify.

I used an identifier associated with the newly deployed workload rather than simply asking Prometheus about "the exporter".

That matters because deployments can change while your verification job is waiting.

If somebody rolls back or deploys another version, the original workload disappears. In that case the script reports INCONCLUSIVE instead of accidentally verifying the new deployment.

It's a small detail, but it prevents a particularly annoying kind of false positive.

systemd instead of cron

I could have used cron, but systemd made the rest of this surprisingly simple.

The service looked roughly like this:

[Unit]
ConditionPathExists=!/etc/example/PAUSE

[Service]
Type=oneshot
User=example-verify
ExecStart=/usr/local/libexec/example-verify/verify.sh
TimeoutStartSec=300

StateDirectory=example-verify
LoadCredential=notifier.env:/path/to/notifier.env

ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes

And the timer contained the few dates on which I wanted the check to run:

[Timer]
OnCalendar=...
OnCalendar=...
OnCalendar=...

Persistent=true

There were a couple of nice things here.

LoadCredential= lets systemd provide the notification credential to the service without copying it into the script's environment or configuration.
ConditionPathExists=!PAUSE gives me a simple kill switch. Create the file and systemd doesn't start the service.

And because the timer had a finite number of scheduled runs, it didn't become another forgotten cron job that would still be running six months later ;)

There was also a small systemd gotcha. I initially used:

RuntimeMaxSec=300

because that's what I wanted: kill the process after five minutes. But it appeared that if service is Type=oneshot, that isn't the right setting for limiting its startup/run phase.

The correct option here was:

TimeoutStartSec=300

systemd-analyze verify happily caught it. It's worth running, costs nothing.

Where the LLM actually helped

This isn't an anti-LLM story, on the contrary the LLM was useful during the investigation.

The interesting part was figuring out what should be checked.

For example, one of the initial metrics I was looking at seemed to show that the exporter had enough headroom, while second review showed that this conclusion was wrong.

The apparent maximum duration was based on successful scrapes. When a scrape exceeds the timeout, it can fail instead of giving you a nice metric saying "this scrape took 27 seconds". So a graph showing a maximum of 9 seconds doesn't necessarily mean the exporter never took longer than 9 seconds. It can mean that 9 seconds was the longest successful scrape.

The failed scrapes were the important part. That kind of investigation is where an LLM is useful:

correlate different signals;
suggest what to look at;
challenge assumptions;
generate queries;
review a proposed solution;
write the boring documentation and ticket text.

My current rule

I ended up with a fairly simple rule:

If the pass/fail condition can be written as a comparison, use a script. / If you still need to figure out what should be compared, an LLM can be useful.

The important part is not "bash good, AI bad". It's rather about using the right amount of machinery.

An LLM is useful when there is ambiguity and you need judgement. A script is useful when the decision has already been made and needs to be repeated reliably (not "re-decided each and every time").

In this case, using an agent would have meant building a small security boundary around a system whose job was basically:

query
query
query
query
compare
notify

The shell script was easier to understand, easier to secure and easier to debug.

And it was considerably cheaper ;)

One last surprise

After three nights the check reported exactly what I wanted: no restarts and scrape times comfortably below the new timeout. So I deleted the timer, the service, the script and the temporary user.

Except I didn't.

The cleanup command appeared to run successfully but didn't remove the files. The reason was my shell configuration.

My interactive shell had an rm -i alias. The non-interactive environment inherited it. Without a terminal to answer the confirmation prompt, rm -i effectively did nothing.

That was a good reminder that automation doesn't necessarily run in a clean environment. Your shell configuration is part of the environment too.

If you use interactive-only aliases, make sure they are actually limited to interactive shells:

if [[ $- == *i* && -z ${AGENT_SHELL:-} ]]; then
    alias rm='rm -i'
fi

Use whatever environment variable or detection mechanism makes sense for your agent.

The broader lesson was probably the same as the one from the beginning:

Use the clever tool to figure out what needs to be checked. Use the boring tool to check it.