Agentic DevOps means the agent verifies its own work
I spent a night watching an agent run an ops loop end to end. Six servers, two releases, one incident. The interesting part was not that it could do the work. It was which step made the difference, and it was not the one I expected.
The incident
A box froze twice overnight. I was sure it had rebooted.
It had not. uptime said seven days, continuously, and there was no reboot in the logs. That is the kind of contradiction that usually ends an investigation, because the machine sounds authoritative and the person sounds vague.
The person was right. What actually happened was two out of memory kills, at 00:14 and 01:55. The second one is the one that matters. Linux runs its OOM killer in the context of whatever process asks for memory next, and at 01:55 that was sshd. So the session went dead for several seconds and came back with things missing. From where you are sitting that is indistinguishable from a reboot.
So the first useful thing an ops agent can do is refuse to let uptime win the argument. Both facts were true. Only one of them explained what the human saw.
The number nobody looks at
Linux has kept pressure stall information since 4.20, and almost nobody reads it. /proc/pressure/io on that box had logged roughly ten hours of full stall over seven days. Ten hours where every process was blocked on the disk at once.
Nothing had crashed. No alert had fired. There was no error anywhere to grep for. The box was just periodically unusable, and the only artifact of that was a counter nobody had checked.
That is the shape of a lot of infrastructure problems. Not a failure, a degradation with no error attached. It is also exactly what an agent is good at, because reading forty diagnostic counters is boring and a machine does not get bored.
Two changes, and they fix different halves
The box had 16 GB of RAM, 2 GB of swap, and no early OOM protection at all. So:
zram is a compressed block device in RAM used as swap. Cold pages get compressed at roughly three to one instead of written to disk. Reclaim stops costing seeks and starts costing a little CPU. It does not add memory. It moves the wall further out. On that box it is currently holding 6.1 GB of pages in 1.5 GB of actual RAM.
earlyoom watches free memory and kills the biggest process while the box is still responsive. The kernel's own OOM killer is not wrong about what to kill. It is late. It only fires when allocation genuinely fails, and by then you have been thrashing for minutes and cannot type. earlyoom fires at ten percent free instead.
Swap went from 2 GB to 10 GB. Six boxes got the same treatment. Two of them had been running with zero swap, and one of those has 961 MB of RAM and was sitting at 575 MB. That box was one traffic spike from an OOM kill with no runway at all.
The step that actually matters
Here is what I got wrong about agentic work before this.
I assumed the hard part was the fix. It is not. The fix is the easy part, because it is the part that is written down somewhere. The hard part is verification, and specifically verification that does not trust the thing being verified.
Three things went wrong that night, and every single one of them reported success:
The systemd unit for zram failed with a dependency error naming dev-zram0.device. The real cause was that the DigitalOcean kernel image ships no zram module at all. The error named the symptom and never the cause.
A zram device that already has a size set refuses to be reconfigured. It returns Device or resource busy and quietly keeps its old settings. Your new config appears to apply. It has not.
And systemd expands variables from an EnvironmentFile by splitting on whitespace with no shell quote removal. So writing --avoid '^(sshd|systemd)$' puts the quote characters inside the regex, where they match nothing. The service starts perfectly happily. The only way to know is ps -o args= on the running process.
That last one is the useful lesson. Every layer said fine. The daemon was running. The config file was correct on disk. And the protection was not actually in place. If the loop had stopped at "systemctl is-active says active" it would have shipped a box that was still going to freeze, and the freeze would have been more confusing next time because now there was a fix in place that everyone would assume was working.
An agent that cannot verify its own work is just a faster way to be wrong. That is the whole thing. Speed multiplies whatever your verification step actually catches.
Generalise or you have deferred an outage
Six boxes fixed by hand is six boxes that drift. The change went into root-ubuntu.sh, the provisioning script, so the next box is born with it.
Idempotence is what makes that safe to re-run, and it is worth being specific about what idempotent means here. It is not "does not crash on the second run". It is "changes nothing on the second run". An already correct zram device is never cycled, because cycling it would drop whatever the box had paged out. Config goes through a write-if-changed helper. The service is only restarted when something actually changed.
I proved it by running the thing twice and diffing what it claimed to change. First run, three changes. Second run, zero. That is a two minute test that turns a belief into a fact.
Merged is not shipped
The other thing worth automating is the tail end, because it is where work quietly dies.
A merged pull request reaches nobody. The release has to be cut, the registry has to actually serve the new version, and someone has to say it exists. Two releases went out that night, and for one of them the announcement copy had to be pulled back and rewritten, because a follow up change landed two hours later and made the queued posts describe an interface that no longer existed.
That is a small thing and it is the kind of small thing that only gets caught if the loop includes the announcement rather than stopping at the merge.
What the loop looks like
Observe. Diagnose. Fix. Verify on the machine. Generalise into provisioning. Ship, then say so.
Six stages, and the value is concentrated in the fourth and fifth. Anyone can get an agent to make a change. The question is whether it checks the change on the real box with a command that could return bad news, and whether the change outlives the box it was made on.
One more thing
The last piece of this shipped as a moshcode command. If the agent you are supervising is itself the thing eating the machine, you want to be able to hold it back:
/nice agents claude
/nice pnpm -r build
/nice mem 2G
It covers three resources rather than one, and the reason is the whole point of the night. nice reorders the CPU. It does nothing about memory, and memory is the half that kills processes. So it is nice for CPU, ionice for the disk, and a systemd scope with a hard ceiling for the part that actually takes the box down.
npm i -g moshcode for 0.93.0.
Top comments (0)