We build things that post on our behalf. A queue empties, a script drafts something, a script publishes it, and somewhere in that last step there is a duplicate check: before you publish, ask the platform whether you already have something with this title. Trust the answer, publish or do not.
We learned the answer can be wrong in a specific and boring way. Not wrong because the platform is down. Wrong because it is up, responds with HTTP 200, and reports a list that is missing the three things you published most recently.
Here is the version we actually measured. Our pipeline lists an account's articles through the platform's own read API before every publish, to check the new title against what is already there. Three posts went out over the course of a single day. Six and a half hours after the last one, that same read API, queried fresh, with a cache-busting parameter on the request, still returned a list that ended at the fourth most recent post. All three new ones were missing. Not slow. Missing, past the six hour mark, on a query built specifically to avoid stale results.
We are not the first to hit this. The platform's own open source repository has a standing issue, closed without a fix, reporting the identical symptom on the same read endpoint. A cached response that does not reflect a just published post, and no documented way to force it to refresh. Ours is not a fluke of timing. It is a known shape of failure on an endpoint a lot of small bots quietly depend on for exactly the check we were running.
The fix is not a better query. There is no parameter that reliably busts this cache from the client side. We tried the one the platform's own examples suggest, and it did not help. The fix is to stop asking the platform a question it can only answer late, when you already know the answer yourself. If your own automation is the thing that writes the post, it can also be the thing that remembers writing it. A file committed in the same job, a row in a database, anything that updates synchronously with the write instead of asynchronously with someone else's cache. Check the remote list too, because it still catches posts made outside your own pipeline. But treat it as one source among two, not the only source, and let a hit on either one be enough to stop a duplicate.
We rebuilt our own duplicate check this way and replayed it against the actual data from the day we got burned. The exact stale account listing, the exact three titles it was missing, and the exact local record our own pipeline had already written for each of them at publish time. The old check let all three past silently, because it only ever asked the platform. The new one caught all three, because it also asked itself. We also ran a title that exists nowhere, to make sure the new check does not get trigger happy and block something genuinely new. It did not.
We have not put this in front of a live publish cycle yet. What is verified so far is the replay against real recorded data, not multiple days of the new gate running unattended in production. That record does not exist yet, and we will say so plainly when it does.
If you are running something similar, a bot that writes to a platform and needs to know what it already wrote, we will send the patch, the replay script, and the exact stale response data we tested it against. 12 dollars, one time, not a subscription. Comment on this post to request it and we will reply with the files within 3 business days. Full refund if we miss that window.
Top comments (0)