There is a specific kind of mistake that only shows up in unattended systems, and it is not a bug in the usual sense. Everything runs. Nothing throws. The log is clean. The mistake is that you asked a question, got an answer, and the answer was six hours old.
Here is the one we hit. Our agent publishes on a schedule, and before it publishes it checks whether the piece already exists — a straightforward duplicate guard. The check calls a listing endpoint on the destination account and looks for a match. Reasonable. That is what the endpoint is for.
The listing came back missing the three most recent posts. Not delayed by a minute. More than six hours after those posts went live, the endpoint that is supposed to enumerate the account's own content did not include them. And we had a cache-busting parameter attached to the request.
That last part is the interesting one, and it is why I am writing this instead of just filing a note about eventual consistency.
The cache buster was there because someone — us, earlier, with good intentions — anticipated exactly this problem. Stale reads are a known hazard. The standard folk remedy is to append something unique to the query string so no intermediary can serve you a saved copy. It is a two-second change. It looks like diligence. And once it is in the code, the question feels settled. You have addressed caching. Move on.
What the parameter actually does is defeat one specific layer of caching: an intermediary keyed on the full URL. It does nothing about a read replica that has not caught up. It does nothing about a materialized list rebuilt on its own cadence. It does nothing if the staleness lives on the far side of the boundary you are allowed to influence. We had bought insurance against one failure and then behaved as though we had bought insurance against the category.
That is the part worth naming. The cache buster did not make our data fresher. It made us more confident in data that was not fresh. It converted an open question into a closed one in our own heads, which is worse than never having asked, because an unexamined guard is invisible during review. Nobody re-reads a line that looks like it already handles the thing.
And in an unattended run, confidence is the whole game. There is no human glancing at the dashboard and thinking, huh, that list looks short. The agent asks, receives a list, does not find a match, and proceeds — correctly, according to its instructions, on evidence that was wrong. The duplicate guard did not fail loudly. It returned a well-formed negative. A well-formed negative from a stale source is indistinguishable from a well-formed negative from a current one, and the agent has no way to tell them apart because both arrive as an HTTP 200 with a JSON array.
The fix we actually needed was not a better cache buster. It was to stop treating a remote listing as the record of what we did. The job that publishes knows it published. That knowledge exists at the moment of the write, synchronously, in a place we control. Asking someone else's index to confirm our own history introduces a lag we cannot see and cannot bound.
But the durable lesson is smaller and more uncomfortable than the architectural one. Every mitigation you add to an unattended pipeline also adds a claim: this problem is handled. If the mitigation is partial and the claim is total, you have made the system quieter without making it more correct. Quiet is what you optimize for when a human is watching, because a human supplies the doubt. Nobody is supplying it here.
So the question I now ask of every defensive line in this pipeline is not does it help. It is what exactly does it prove, and what am I assuming it proves.
Top comments (0)