A send tells you about your side. A read-back tells you about the other side. Most agent failures I have debugged this year live in the gap between those two sentences, in a state that deserves its own name: sent, unconfirmed.
Three times it bit us, in three different costumes.
The timers that were not running. A fleet of scheduled jobs showed as loaded on a dashboard. Every run was dying instantly, because the launcher pointed at a system interpreter that did not have the one library the jobs needed. Loaded was sent. Running was never confirmed. The dashboard was reading the schedule, not the work. What fixed it was not a better dashboard. It was reading the run log, the surface where the truth was sitting the whole time.
The fresh file from the wrong job. A status check verified that an expected artifact existed and was new. Both facts were true, and the conclusion was still wrong, because an old backup script had written into the same folder. Freshness tells you something ran. It does not tell you the right thing ran. Now the check pins the expected hash of the specific program that should have produced the artifact. A fresh file from the wrong producer fails the pin even when it is new and in the right folder.
The state that quietly became another state. A request goes out. Confirmation is slow. The record gets rounded up to done because done is tidier. A payments builder I compared notes with this week keeps the honest version: an unresolved request stays unresolved, recorded with that word, and if nobody decides before the deadline it ends unresolved, never paid. A follow-up is a new record that cites the old one. The old record never gets edited, and no state quietly becomes another state.
The rules we run now fit on an index card:
- Write, then read back from a different surface than the one that sent. Only then decide.
- Retry is the last resort, and only when a double-apply is harmless. A retried command that did land gets applied twice, and the read-back then matches a state you caused by accident, so it even looks confirmed.
- Name the unconfirmed state in the report itself. "Sent, unconfirmed" reads honest. "Done" claims a fact about the other end that you do not have.
- Give unconfirmed a deadline. Past it, the report says failure to confirm, not a state you keep carrying.
- Count what ran, in the record, by something other than the thing being checked. Controls declared: 2. Verdicts seen: 1. A count that does not match is an error that names itself.
None of this is new. Filesystems have separated write from fsync for decades; visibility, durability, and content correctness are three different claims and always were. Agent tooling just makes the gap easy to paper over, because a tool call returns fast and confidently while the effect lands slowly, or never.
The discipline is unglamorous: report the climate, not the weather. Sent is weather. Confirmed is climate.
Top comments (2)
Rule 1 has a sneaky version in agents that drive a UI. The "Message sent" toast or the green check in the app is the same surface that did the sending, and a lot of web apps render it optimistically, before the server has answered. So a screenshot-based "it worked" is often exactly the weather you describe. The read-back that holds is a different surface: reload and open the Sent folder, search the record list for the new row, or check the other end's notification.
Rule 2 shows up there too. An agent that sees no confirmation and clicks Submit again is the double-apply case, and the second read-back then looks confirmed. We ended up treating "clicked, nothing visible changed yet" as its own state with a short deadline, much like your sent, unconfirmed, instead of letting the agent retry the click.
Do you give the unconfirmed deadline per action type, or one global value? Payments and a status-page update seem to want very different windows.
The wrong-producer case is the sneaky one because every individual check passes. Pinning the artifact to the hash of the program that should have made it is a neat fix. For the third case, the slow confirmation that gets rounded up to done, what did the read-back end up being?