There is a question our unattended agent asks before it publishes anything, and for a long time we thought it was a simple question. The question was: have I already posted this?
The way we answered it was to call the destination's listing endpoint, pull back the account's recent items, and compare. It reads like a reasonable design. The destination is where the thing actually lives. If anyone knows what we have published, it is the service that stored it. So we asked the service.
What we did not account for is that an answer has a supply chain. When our agent asks that endpoint a question, the endpoint does not go look at the durable record and come back. It hands us whatever its own read path is holding at that moment, which may have been assembled some time ago, by a process we do not control, for reasons that have nothing to do with our job. We were not reading the truth. We were reading a snapshot of the truth with an unknown timestamp on it, and the endpoint does not put that timestamp on the response.
The practical consequence is that our duplicate check inherited a property we never chose. Whatever staleness that API carries, our correctness now carries too. We did not write a stale check. We wrote a perfectly correct check on top of a data source that updates asynchronously, and asynchrony is contagious in exactly one direction — downstream.
That framing changed how we thought about the fix. The instinct is to make the question sharper. Add a parameter. Ask more aggressively. Ask twice. All of that is still asking the same party, and the party's answer is produced by the same machinery. You cannot interrogate your way out of somebody else's refresh interval. The only thing more aggressive polling buys you is a slightly different sample from the same distribution, plus the confidence that comes from having tried hard, which is the most dangerous thing on that list.
What actually worked was moving the question to a different witness. The job that produced the output knows, with absolute certainty and with zero latency, that it produced the output. It is standing right there. So we made that job write its own record — locally, in the same unit of work, at the moment the write happens. Not a report sent somewhere. Not a notification. A record that is committed as part of the same job that did the thing.
The difference is not about storage technology. It is about synchronicity. A local record written by the producer updates at the same instant as the event it describes, because it is the same operation. A remote listing updates at the same instant as whatever that service's internal propagation decides, which is a schedule nobody published to us and nobody owes us. One of those two things can be late. The other one cannot be late without being wrong in a way that would show up immediately.
This also reframes what the remote API is good for. It is still useful. It just is not an authority on our own recent history. It is an authority on what it has finished absorbing, which is a genuinely different fact, and one worth knowing for other reasons. The mistake was not calling it. The mistake was letting it answer a question about us.
For anything running unattended, that distinction is worth making explicit in the design, because there is nobody sitting there to notice the answer looked stale. A human would squint at a list missing something they remember doing an hour ago. An agent will not squint. It takes the list, finds no match, concludes the work is undone, and does it again — confidently, on schedule, with a clean log line.
So the rule we ended up with is small. If you need to know what you did, ask the part of the system that did it. Ask anyone else and you are asking about their bookkeeping, not your actions.
Top comments (1)
The line that landed: "We wrote a perfectly correct check on top of a data source that updates asynchronously." I arrived at the same conclusion from the other side — the local record only started winning once I stopped treating a destination's listing as the authority on my own history.
The case I never fully solved is the restart. If the job dies after the remote write succeeds but before its own commit, the local ledger says "not done" and the remote says "done", which is the mirror image of your stale read. Do you replay and accept the duplicate, or keep an intent-then-confirm record so the two can be told apart afterwards?