We spent a long time arguing about which source of truth our unattended agent should consult before it publishes, and the whole argument was framed wrong. We kept asking which record was more authoritative. The question that actually mattered was which record was more simultaneous.
The setup is ordinary. A job produces an artifact, and before producing it, the job should confirm it has not already produced that same artifact. The obvious way to confirm this is to ask the place where the artifact lives: give me the list of everything I have there, and I will check whether this one is in it. That reads like the responsible choice. It consults the system of record rather than trusting our own bookkeeping. It is the version of the check you would defend in a review.
It is also the version that broke. A listing endpoint we used to check for duplicates returned a list that was missing the three most recent items, more than six hours after they were published. The request carried a cache-busting parameter, which we had added earlier precisely because we suspected staleness, and which bought us nothing. The remote service was not lying. It was answering from a view of the world that had not caught up, and it had no obligation to tell us how far behind that view was.
Here is the part worth sitting with: nothing about the endpoint's authority was in question. It really is the system of record. If you asked it a week later it would give you the right answer. The property it lacked was not correctness but timing — it updates asynchronously, on someone else's schedule, with someone else's cache in the middle. Our agent, meanwhile, needs an answer within milliseconds of doing the thing.
A local record does not have this problem, and the reason is almost boring. If the same job that produces the output also writes a line saying it produced the output, that line updates synchronously with the write. There is no interval during which the write has happened and the record does not know. The interval is what kills you, and the interval is zero by construction, not by luck.
I want to be precise about what this is not. It is not "don't trust external services." It is not caching, or eventual consistency lectures, or an argument that our bookkeeping is better than theirs. It is narrower: a check that asks a remote API "do I already have this?" inherits that API's staleness as its own failure mode, and inherited failure modes do not show up in your own tests. Our duplicate check was working perfectly. It was faithfully reporting what it was told. The bug lived in the gap between two clocks, which is exactly the kind of place a unit test never looks.
Under supervision, this class of bug is nearly invisible, because a person glancing at the output catches the duplicate before it matters and never learns why it happened. Unattended, the same gap becomes a mechanism. The agent asks, receives a confident negative, and proceeds — and it will proceed the same way tomorrow, for the same reason, with no accumulating evidence that anything is wrong. Confident negatives are the most dangerous thing an autonomous system can receive, because they terminate inquiry.
So the design rule we landed on is stated in terms of time rather than trust. For any fact your agent must know immediately after acting, the record it consults must be one that changes at the same moment the action does. Anything else is a fact about the past wearing the clothes of the present. Remote listings are still useful — for reconciliation, for auditing, for catching drift between our record and theirs over a longer window. They are simply the wrong instrument for a decision that has to be made in the same breath as the write.
Write your own receipt at the moment you do the thing. Then ask the world later, at leisure, whether it agrees.
Top comments (0)