Your dedupe key is fine. Your dedupe state is the problem.

What I built

I run a small fleet of about twenty personal sites. A bot collects the articles published each day and announces them on social media. Nothing fancy: gather URLs, skip the ones already announced, queue the rest, post them on a schedule.

The "skip the ones already announced" part is where this story happens. It broke in a way that I think is worth writing down, because every component behaved correctly and the system still produced duplicates.

Idempotency, for anyone who wants the term pinned down, means running the same operation more than once leaves you in the same state as running it once. "Do not announce the same article twice" is exactly that.

The shape of the system

The bot runs in two phases, shown below. enqueue decides what to announce and writes it to queue.json; process reads that queue later and does the sending.

enqueue: collect today's article URLs
         -> keep only those absent from posted.json and queue.json
         -> append to queue.json
process: send queue entries whose scheduled time has arrived
         -> move them into posted.json

So a URL travels collected -> queue.json -> posted.json, and the duplicate check happens once, at the front, against both files. Checking both files matters. If you only check posted.json, anything queued but not yet sent gets queued a second time.

Step one: the key was too coarse

The original key was the URL alone. That worked until I added per-genre accounts, where announcing one article to two different accounts became the correct behaviour.

With a URL-only key, the second account was rejected because the first had already claimed the URL. The duplicate guard was blocking legitimate posts. The fix was to key on the pair of destination and URL:

function buildGenreDedupe(posted = [], queue = []) {
  const acctKey = (webhook, url) => `${webhook}|${url}`;
  return {
    acctKey,
    postedLegacy: new Set(posted.filter((r) => !r.webhook).map((r) => r.url)),
    seen: new Set([
      ...posted.filter((r) => r.webhook).map((r) => acctKey(r.webhook, r.url)),
      ...queue
        .filter((q) => !q.posted && q.webhook)
        .map((q) => acctKey(q.webhook, q.url)),
    ]),
  };
}

Concretely, for article https://example.jp/a going to webhooks W1 and W2:

• Before: queue a, then reject the second one. Wrong.
• After: W1|a and W2|a are distinct, so both queue. A repeat of W1|a is rejected. Correct.

Note postedLegacy. Older records in posted.json predate the webhook field. Folding them into the new key space would produce undefined|a, which matches nothing, so previously announced articles would look unannounced and go out again. Old records are read in the old shape, alongside the new ones. When you change the shape of a key, decide how pre-existing records are read before you ship it.

Step two: the key was never the problem

The key design is sound. The actual incident was elsewhere.

posted.json and queue.json are in .gitignore. They are local to the machine and shared with nothing. On one machine that is invisible. The day I started running the same job on a second machine:

Machine A: posted.json = { W1|a }  -> skip (correct)
Machine B: posted.json = { }       -> "not announced yet" -> queue (also correct)

Both machines evaluated their own records correctly. Both reached a correct conclusion. The announcement went out twice, and posts cannot be unsent.

The generalisation I took from this: idempotency is not a property of the key, it is a property of the scope in which the key is recorded. Normalise the key all you like. If the record lives per actor, you get one independent idempotency guarantee per actor and none for the system.

The real fix is moving that state somewhere shared: Firestore, object storage, a shared filesystem. That takes time, so the immediate measure was a guard that pins queueing to a single machine, named in an environment variable.

The interesting decision was the default when the variable is unset. Allowing it means a forgotten setting on machine two turns straight into duplicate posts, and forgotten settings always happen. Denying it means the failure is "nothing got queued this run", which is fully recoverable by setting the variable and queueing again. Choose the recoverable failure over the unrecoverable one. Fail closed.

What bit me

Duplicates had not actually happened yet, and when I looked into why, it was three unrelated accidents stacked on top of each other: a config path hardcoded as an absolute path from machine one, the posting daemon only being installed on machine one, and machine two missing the account definitions file.

The absolute path is the dangerous one. It looks like an obvious defect. Anyone would fix it, and fixing it removes a blindfold rather than a bug. So the guard carries a long comment at the top stating the condition for removing it: only after queue and posted state live somewhere both machines can read. Making the path relative does not satisfy that, because config is input, not dedupe state.

One more thing I had to walk back. The guard's rejection message originally printed FLEET_TWEET_HOST=<this machine's hostname>, ready to paste. That was actively harmful: the guard does not know which machine is supposed to be the owner. It only knows its own name. Prompting a paste on the wrong machine gets you two owners and the duplicate posting the guard exists to prevent. It now says to set the owner's hostname, without offering a value.

Try it

One of the sites this bot announces: https://manga.autoarticles.net

Takeaway

When you write a duplicate guard, the attention naturally goes to the key. That work is necessary but not sufficient. Check in this order:

1. Does the key include every dimension you need to distinguish?
2. How are records written before the key change read afterwards?
3. Where is the record kept?
4. Can every actor that might run this see that record?
5. If not, pin execution to one actor until it is shared, and fail closed when unconfigured.

Steps 1 and 2 show up in code review. Steps 3 to 5 do not, because they are not visible in the code. "Correct on one machine" does not imply "correct on N machines".

---

This article is about my own side project. It was written with AI assistance.