API Publishing Deduplication with Failure Recovery: Reconcile Before You Retry

A publishing job sends a request. The connection closes before a response arrives. Should the job send the same request again?

The answer depends on something the worker does not yet know: whether the remote platform accepted the first request. A network timeout describes the worker's observation. It does not prove that publication failed. Treating every timeout as permission to retry is an easy way to create duplicate posts.

This guide describes a small worker design. It is an implementation pattern, not a claim that every publishing platform offers the same API features. Adapt the remote lookup and idempotency mechanism to the platform you actually use.

Separate content identity from delivery attempts

Give each reviewed content item a stable internal identifier. Then create a separate delivery record for each destination account. The combination of content ID, platform and account identifies the intended publication; an attempt ID identifies one execution of the worker.

Do not generate a new content ID merely because a request failed. That would make a retry look like a new publication. Likewise, do not use the current timestamp as the only identity: each execution would receive a different one.

A minimal delivery record might look like this:

{
  "contentId": "guide-2026-10-16-01",
  "platform": "example-network",
  "accountId": "brand-account",
  "state": "approved",
  "externalId": null,
  "lastAttemptId": null
}

Enforce uniqueness for that destination identity in the database. Application-level checks alone can race when two workers read the same queue.

Claim work atomically

The worker should change an eligible delivery from approved to claimed in one atomic operation. If another worker has already claimed it, stop processing that delivery. A read followed by an unrelated write leaves a gap in which both workers can believe they own the item.

Keep the claim's timestamp and worker identifier. A crashed worker can leave a claim behind, so a recovery process needs to identify stale claims. However, an expired claim is not itself permission to publish again. The first worker might have reached the remote platform before it crashed.

When recovering an expired claim, move it into reconciliation. Check remote state before creating another public post.

Model uncertainty as its own state

Useful states include approved, claimed, published, rejected and uncertain. Define their transitions explicitly. In particular, do not collapse uncertain into rejected simply to make the retry queue easier to implement.

Examples help distinguish them:

• A local validation failure before any network request is a known failure.
• A confirmed remote rejection can be recorded as rejected, following that API's documented response semantics.
• A timeout after sending a publication request is uncertain.
• A remote ID followed by a failed database update means the remote publication succeeded while local recording failed.

That last case deserves special handling. Record the external ID in durable logs or another recovery store before acknowledging the job. Do not issue a second publication request just because the final database write needs retrying.

Reconcile using the strongest available evidence

If the platform provides an idempotency key for this exact operation, reuse a stable key according to its documented scope and retention rules. Do not assume an arbitrary request header creates idempotency.

If you have an external ID, query that resource directly. Otherwise, use an authenticated listing for the correct account and compare the intended publication with remote records. A stable marker can help where supported. Exact text alone is weaker evidence: legitimate posts can share wording, and platforms can transform URLs or formatting.

Read all pages needed to cover the relevant history. If pagination stops unexpectedly, authentication fails or a safety limit is reached, preserve the uncertain state. An incomplete search cannot establish absence. It is safer to surface that delivery for investigation than to silently duplicate it.

Retry the failed stage

Structure the implementation around stages rather than wrapping the entire worker in a broad retry loop:

async function recover(delivery, remote, store) {
  if (delivery.externalId) {
    const post = await remote.get(delivery.externalId);
    return store.confirmPublished(delivery.id, post.id);
  }

  const result = await remote.findDelivery(delivery);
  if (!result.complete) return store.markUncertain(delivery.id);
  if (result.match) {
    return store.confirmPublished(delivery.id, result.match.id);
  }

  return store.markForReviewedRetry(delivery.id);
}

The example deliberately does not publish. It resolves what is known and hands a confirmed absence to the retry policy. A production adapter must define what a complete search means and how it handles eventual visibility of newly created posts.

Use bounded backoff for transient read failures. Avoid repeatedly querying a platform in a tight loop. Authentication problems need credential recovery, not more publication attempts.

Test the failure between each pair of stages

The most useful tests interrupt the worker at meaningful boundaries: after the claim, after the remote write, and before local confirmation. Simulate a remote service that creates the post but throws a timeout to the caller. Recovery should discover the existing publication and avoid a second write.

Also test concurrent claims, multi-page history, an incomplete history response and a failed local confirmation. Count remote writes in the test, not just successful worker exits. A worker can exit successfully after publishing nothing, or fail after publishing something; neither exit code alone tells the full story.

Report publication separately from job health

For each run, report selected, published, rejected, uncertain and skipped counts. Include the external ID and public URL when available. A green scheduler badge means the job completed according to its exit policy. It should never be presented as proof that a new post exists.

Keep credentials out of logs, and retain enough non-secret delivery metadata to reconcile an interrupted attempt. This approach makes recovery a controlled state transition instead of a guess. It also makes a quiet day explainable: an empty reviewed queue is a content-supply issue, while an uncertain publication is a delivery issue. Each needs a different fix.