Every agent demo ends with a result. Almost none end with a mess.
But a failed run leaves residue: a half-written config, a service that is now stopped, a file renamed to something nobody recognises, a database with three rows the agent created while "exploring". The next run starts from that state and fails differently, and now you are debugging two problems that were never the same problem.
This is the part I underestimated most.
Snapshot before, not after
Before the first action of any task that touches state, take a cheap snapshot: file hashes, the output of the commands that describe the world (git status, service states, listening ports), and a copy of anything the task will modify.
Cheap is the operative word. A snapshot expensive enough that I skip it is a snapshot that does not exist.
Idempotency is not optional
My rule: every step must be safe to run twice. Create-if-missing, not create. Append-once, not append. If a step cannot be made idempotent, it needs a check before it runs, and that check's failure must be loud rather than silent.
The failure mode here is subtle: a non-idempotent step does not fail on the retry, it succeeds twice, and the damage shows up three steps later.
Make the failure path a first-class path
The recovery needs to be a task the agent can run, not a paragraph in a runbook I read at midnight:
- list what changed since the snapshot
- revert only what the task itself did - not everything that differs
- re-run the verification that failed, to confirm the environment is back
That third one matters. Reverting and then assuming you are back is how you get a fix that works once.
Log the intent, not just the action
A log line that says ran rm -rf build/ is useless after the fact. A line that says intent: clean build output, so the next build is from source; target: build/ (created 3 min ago by this task) is debuggable.
If I cannot reconstruct why a step ran, I cannot decide whether to revert it. Intent is what makes a rollback safe.
What this bought me
Failure rate barely moved. Recovery time dropped a lot. Most of my bad days were never about the agent being wrong - they were about a wrong result I could not cleanly undo, so I kept going forward instead of back.
None of this is agent-specific. It is just the cost of letting something act on your behalf, and the bill comes due the first time it acts wrong.
If you run agents against real state, what does your snapshot actually contain? Mine is 4 hashes, 2 command outputs and one tarball, and I am not sure that is the right set.
Top comments (0)