An agent that fails loudly is annoying.
An agent that says "done" while nothing ran is expensive.
I've been building with AI agents near real systems, and the failure mode I worry about most isn't a crash — it's silent success.
The agent returns something that looks fine. The UI says the task completed. But when you dig in, the command never ran, the wrong thing ran, or it checked its own assumption instead of the actual system state.
Isolation still matters — I wouldn't run arbitrary agent code without boundaries.
But a sandbox alone doesn't answer: "did the thing I care about actually happen?"
That's where I've shifted my thinking: less "is it isolated?" and more "can I reconstruct what executed — command, args, when, and outcome?"
Curious: if you've shipped agents near real systems, how do you catch the "looks fine, did nothing" cases?
Top comments (4)
The cleanest pattern I use is asserting on an out-of-band state delta rather than anything the tool or model reports. Before the tool executes, the harness snapshots the target state (the git tree hash, file inode timestamps, or a row count). After the agent claims completion, an external verifier checks that delta directly against the disk or database. If the model emits a clean exit code but the underlying inode never changed or the git diff is empty, the run gets marked as failed regardless of what the trace says.
The subtle issue is that “success” should be a property of the observed side effect, not the agent’s response. I’d make the execution boundary emit an immutable receipt containing the intended action, actual command/tool invocation, target state, and postcondition check. Then the UI can only mark a run successful when the postcondition is independently verified. That separation prevents the agent from effectively becoming both the actor and the judge of its own work.
An independently verified postcondition still loses if the read is stale. In a posting pipeline I run, the write returned success while the page rendered right after it still showed the old count, so a checker reading the normal path would have agreed with the wrong answer; the record only turned up on a re-read forced past the cache. So the receipt needs the read to be provably fresh as well as external, otherwise the agent's claim and the verification can be wrong together, for the same reason.
I check the thing the task was supposed to change, not the agent's report of it. For code that means the acceptance criteria are written as tests or scripts before the agent starts, and the run only counts as done when they pass on a fresh checkout. If the command never ran the test fails, if the wrong thing ran the test fails, and the agent doesn't get to mark its own work. Reid's snapshot-and-diff approach is the same idea for anything that isn't code. The gap I still have is when the check itself reads from a cache, which Vinh's example covers well.