I asked an agent to build a monitor: something to watch the background jobs across my servers and tell me when one broke. It came back and told me not to build it.
Not out of laziness. It had spent its first hour running probes against the live machines instead of writing code, and the probes said the thing I asked for already existed five times over. One platform watched the hosts. A container metrics collector watched the containers. The init system watched its own units. Two more scripts watched the edges. What I was actually missing was not another detector. It was the layer that reads five noisy signals and says "these three are the same incident, here is the likely cause, here is the runbook that fixes it."
That refusal was the single most valuable thing the agent did on the whole project. A detector I did not need would have shipped in an afternoon, run green forever, and watched nothing that mattered. The hour of recon that killed it is the reason I trust what got built instead.
This guide is the anatomy of that trust. It is worth nothing that an agent seems careful. What matters is the specific, checkable structure that let this one build a piece of production monitoring I now depend on: a recon step that argues with the task, a finish line a model can actually verify, a plan I approved before any code existed, autonomy bounded by construction rather than by a polite prompt, and acceptance by firing every control instead of watching it sit green.
Who this is for: anyone pointing a coding agent at real infrastructure, especially self-hosted systems where a quiet failure hides for weeks and "it ran without erroring" is not the same as "it works."
Why should an agent's first job be to refute your task?
In a 2026-07 build, the recon phase refuted the premise of the task before a line of code existed. I had asked for a holistic watcher because "nothing watches the jobs holistically," and a live sweep found five watchers already running. Four background units were failing at that exact moment, invisible because the watcher's coverage list was fixed at install time and never grew as new units shipped.
This is the crown rule of pointing an agent at infrastructure: recon is not information-gathering, it is error-correction. A first draft of any plan is full of confident, wrong premises, because the person writing the task does not have the live state in front of them. The job of the recon step is to take each premise and try to break it against the real system. "Nothing watches this" breaks the instant you list what is watching it. "Add a host-metrics exporter to one of the nodes" breaks when you find that node already runs a container metrics collector and ships its host metrics through a separate platform, so your exporter would double-count and page on itself.
The failure mode to design against is the opposite: an agent that takes the task at face value, builds exactly what you said, and hands you a clean diff for the wrong thing. A clean diff describes the code, not the world it runs in. Those four silently-failing units were the proof. Every one of them had been shipped after the monitor's coverage list was written, and the list was a static allowlist that rots the moment the estate grows. The highest-value item on the whole project turned out to be a watcher-of-the-wiring, an audit that asks whether newly-shipped units are actually covered. I never would have thought to ask for it. The recon found it by refusing to believe the task.
What makes a finish line a model can actually check?
The eleven files this run produced were never seen by the thing that graded it. A small, fast model scored the finish from the transcript alone, never running a command or opening a file. That one constraint rewrites how you write a done condition. "The eleven files exist and the tests pass" is unverifiable to that grader, so the run either loops forever or passes on the agent's word. "One final turn shows fresh command output and a self-audit table" is checkable in a single read.
An autonomous agent needs a terminal condition, the signal that tells the loop it is finished. The naive version describes the world: files present, tests green, service deployed. The problem is that whatever checks the condition is usually another model, and it has no eyes on your disk. It sees the transcript. A done condition phrased as a fact about the filesystem is therefore either uncheckable, so the run never converges, or taken on faith, so the agent can declare victory while nothing works.
The fix is to make the finish line transcript-demonstrable. Require the run to surface, in one final turn, the fresh output of real commands: the file listing, the diff, the passing test log, the grep that proves the wiring is live. Then require a self-audit table that maps each acceptance criterion to the evidence for it. Now the grader has something to grade. This is not ceremony. It is the difference between a run that ends because a model felt finished and a run that ends because it showed its work. Write the condition from the grader's real capabilities, not from a template you copied without asking what the grader can see.
Why gate the plan instead of the finished code?
Before any implementation, the recon collapsed nineteen candidate work-items into a design with exactly four genuinely-open decisions, and I resolved all four in a single structured prompt. Zero of the nineteen items collided with a project already in flight. The approval gate sat on the design, in the middle of the arc, not on the finished pull request at the end, because the middle is the only place an approval is cheap enough to be honest.
There is a strong pull toward letting an agent run start to finish and reviewing the pull request at the end. For infrastructure, that is the most expensive possible place to learn the plan was wrong. By the time a diff exists, the agent has committed to an architecture, and your choices are approve-with-misgivings or throw away real work. Reviewing a plan costs minutes. Reviewing and then unwinding a wrong build costs a day.
So the arc has one hard human gate, between research and implementation, and it is the only step that requires me. The agent produces a map: what it will build, what it deliberately will not, every prior decision it reconciled against, and a short list of the forks it genuinely cannot resolve alone. I answer the forks. Everything already settled is asserted plainly so it does not get re-litigated, and only the real decisions reach me. This is also where scope discipline lives. Nineteen items were dispositioned explicitly, each exactly once, so nothing was silently dropped and nothing raced work already running elsewhere. The plan is the artifact you can still change cheaply. Gate there, not at the end.
How do you make autonomy safe by construction rather than by prompt?
The remediation layer runs in three tiers, and none of them can take an action, because the capability to act was never written into any of them. The triage model sees only captured text and is never handed a tool it could call. The advice layer emits a proposal into a chat thread and holds no code path that executes. The incident-filing step can open a draft but has no path that closes. Each limit is structural, so a jailbroken or simply confused model hits a wall, not a moment of discretion.
"Propose-only" written in a system prompt is a request, and a request is not a safety property. The version that holds is the one where the capability is absent. The triage step is the clearest case. Instead of an agent with shell access reasoning about my servers, it is a deterministic collector that captures read-only output and a model that receives that text and returns structured JSON. The model cannot run anything, because nothing runnable was ever put in its hands. Its blast radius is the size of a JSON object.
The same shape repeats up the stack. The layer that suggests a fix writes the suggestion to a thread for a human to read; no branch anywhere in it executes the fix, so no prompt can talk it into one. The step that records an incident can draft a writeup, and the rule that an incident closes only after a human has confirmed the root cause and verified the fix is enforced by that step having no close path at all. This is graduated autonomy done honestly: each tier holds exactly the capability it has earned, and the tiers above it are not one clever sentence away. When you catch yourself trusting a model because you told it to behave, you are holding a prompt, not a gate.
Why isn't "armed and green" the same as working?
Every control that shipped was accepted by firing it, never by watching it sit healthy. We injected a synthetic fault and watched it travel the whole path to a chat thread. We ran one real triage cycle end to end and read the verdict it produced. We handed the refusal paths a case they should reject and confirmed they rejected it. A monitor that is armed and green has told you nothing at all until you have made it catch something.
The most seductive lie in operations is the green dashboard. An alerting rule with no alert, a remediation daemon with a healthy badge, a heartbeat that has never missed: all of them look identical whether they work or are quietly dead. "Armed and green" is a claim about a control's own status, not about whether it does its job. I have been burned by a reaper that reported healthy for weeks while reaping nothing, because nothing had ever forced it to act and reveal that it could not.
So the acceptance bar for the controls here was a fired proof. For the coverage audit, ship a unit it should flag and confirm the flag. For the triage path, run a real cycle and read the result. For the propose-only guarantee, feed it a case that must be refused and confirm the refusal returns an error instead of an action. The heartbeat is the honest exception worth naming: rather than trust a new bespoke alert, wire it onto the estate's already-proven dead-man monitor, the same one that pages when any other producer goes silent, so it inherits a firing that was tested long ago. A control you have watched catch a real event is a fact. A control you have only watched stay green is a hope wearing the costume of a fact. The extra hour spent writing synthetic faults is the cheapest insurance you will buy on infrastructure, because the alternative is discovering the control was decorative during the incident it existed to catch.
What do the five checkpoints add up to?
Across the project, five practices did the load-bearing work, and each one guards against a different way agent-built infrastructure fails silently. Not one of them depends on the model being smart. Every one of them is structure the human keeps: an argument before the build, a finish line a grader can read, a gate on the plan, capability bounded by construction, and acceptance by firing the control.
| Checkpoint | The silent failure it prevents |
|---|---|
| Recon that refutes the task | building the wrong thing, cleanly |
| Transcript-demonstrable finish line | a run that ends on the model's own word |
| Approval gate on the plan, not the PR | learning the architecture was wrong after the work is done |
| Autonomy bounded by construction | a prompt talked into taking an action |
| Acceptance by firing the control | a decorative monitor that is green and dead |
The through-line is that trust in agent-built infrastructure is not a feeling you develop about a capable model. It is a property of the harness around it. The model in this arc was the same model that wrote the confident, wrong first draft of the plan. What made its output trustworthy was that every step had a place where a wrong assumption got caught by something other than the model's own judgment. Recon caught the wrong premise. The grader caught the unfinished run. I caught the wrong plan. Construction caught the unsafe action. A fired test caught the decorative control.
Point a fast writer at your infrastructure with none of that, and you get exactly what the speed implies: the wrong thing, built well, running green, watching nothing. Keep the five checkpoints, and the agent becomes what it should be, a very fast pair of hands on a short and very well-lit leash. That is not a limit on what agents can do for operations. It is the thing that makes handing them the keys a sane decision instead of a hopeful one.
Originally published at danmercede.com.
Top comments (0)