I gave an AI eight real DevOps jobs. Here's where it shipped — and where it quietly got it wrong.

Sambit Mallick — Mon, 31 Aug 2026 10:35:51 +0000

For the last couple of months I've been handing an AI agent (Claude Code) real DevOps work
— not toy prompts, but actual failing pipelines, real pull requests, a live Kubernetes
cluster — and filming what happens. Everything ran locally at $0 (no managed API bill), on
real repos, with a human reviewing every change before it merged.

The interesting part was never "look, AI did the thing." It was the seam between where it
genuinely shipped and where it produced something that looked right and wasn't. That
seam is the whole job. Here's what I found.

The result that made me stop trusting it as a gate

I ran the same AI code review on the same pull request, twice. I got two different
verdicts — one run approved it, the other flagged a blocking issue. Same diff, same prompt,
same model.

That's not a bug, it's the nature of the tool: it's a sampler, not a linter. And it has a
direct consequence for how you're allowed to wire it in. An AI reviewer cannot be a
required status check that blocks merges — not while the same input can yield "approve"
and "request changes" on two runs. It's a fantastic second pair of eyes that catches
things humans skim past. It is not a deterministic gate. Treat it like a very sharp junior
who's occasionally, confidently wrong.

What I do now:

Use it as an advisory reviewer that comments, never one that blocks.
Keep temperature/settings pinned and prompts version-controlled so at least the inputs are stable.
Assume any single run can miss things; the value is in the aggregate signal, not the verdict.

(Full run: https://youtu.be/lox5bc7zCqM)

Where it genuinely earned its keep

Reading CI logs. I pointed it at a red GitHub Actions pipeline with three real failures
— a dependency conflict, a genuine off-by-one in a test, and a workflow pinned to the wrong
Python. It read each failing step's log, named the root cause, and proposed the smallest fix
for each. It was legitimately faster than I am at the boring triage. The honest caveat is in
the title of that episode: "Almost." — on one of them it fixed the symptom, and a human
still had to redirect it to the actual cause. (https://youtu.be/pIkPsiBZQ1M)

On-call, with hard guardrails. The one I trust most is an on-call agent that wakes up on
a broken build, finds the fix, and opens a PR explaining what broke — and then stops. It
has no merge rights. A human reads the PR over coffee and clicks merge. The autonomy ends at
the merge button, on purpose. (https://youtu.be/UcOt5JrQVco)

The guardrails that made agentic DevOps safe enough to film

Every "the AI did DevOps and it was fine" story is really a story about constraints. The set
that worked for me:

Edit-only tools. No shell, no network, no installs. It can read and edit files and open PRs — that's it.
Throwaway branches only. It never commits to main.
PR-only, no merge rights. A human reviews every change. This is the load-bearing one.
One attempt per failure. No token-burning retry loops that "eventually" pass by luck.
Tests are the spec. It never silences or weakens a failing test to go green.

Give an agent hands but not the keys, and most of the scary failure modes just… can't happen.

Scaling review without paying for it

Three episodes are about making review cheap and constant:

Instant review on every PR, free, via a GitHub Actions workflow — the review lands seconds after the PR opens. (https://youtu.be/8xEto5pcoJY)
Ephemeral self-hosted runners on Kubernetes (Actions Runner Controller): a fresh pod per PR that runs the review and self-destructs. $0, no idle runners. (https://youtu.be/PA6NoR7Vc6A)
A reviewer with memory — a git-tracked file of your team's conventions that it learns and applies, so it stops re-flagging things you've already decided. (https://youtu.be/-9g8sSO0fL0)

The takeaways, if you skipped to the bottom

AI review is advisory, not a gate. Same PR can get opposite verdicts; never make it a required check.
Humans keep the merge button. Agents get hands (read/edit/PR), never the keys (merge/deploy).
Constraints are the product. Edit-only, throwaway branches, one attempt, tests-as-spec.
Verify every change. "Valid YAML" and "the tests pass" are not the same as "correct."
It's genuinely useful for the boring middle — log triage, first-pass review, drafting fixes — which is most of the toil.