Failure Recovery for Publishing APIs: Reconcile Before Retrying

A network timeout does not tell you whether a publishing request failed. The remote service may have rejected the request, accepted it and lost the response, or created the record while your process was already disconnecting. Retrying immediately can turn one uncertain delivery into two public posts.

This guide presents a limited recovery pattern for workers that publish approved content to remote APIs. The design combines stable operation keys, explicit delivery states and reconciliation against remote history. It is intended for systems that can search recent records or attach a client-supplied identifier. If the destination provides neither capability, the worker must remain conservative and escalate uncertain writes instead of guessing.

Model uncertainty as a real state

A delivery table often begins with pending, published and failed. That model is incomplete because a transport error can leave the remote result unknown.

Add an uncertain state. Enter it when the request may have reached the destination but the response cannot be trusted. Do not automatically return an uncertain item to the normal retry queue.

A useful record contains the internal content ID, destination, operation key, request start time, response status when known, remote ID, public URL and last reconciliation time. Store error summaries, but never store access tokens or complete authorization headers.

Create a stable operation key

Generate the operation key before the first network request and persist it with the delivery attempt. It must remain stable across process restarts and operator-approved retries.

For example:

import { createHash } from "node:crypto";

function operationKey({ destination, contentId, revision }) {
  return createHash("sha256")
    .update(`${destination}:${contentId}:${revision}`)
    .digest("hex");
}

The revision matters. A corrected article can be a legitimate new operation, while restarting the worker for the same revision should not create another delivery.

If the API supports an idempotency header, send this key in the documented field. That is the strongest option because the destination can reject duplicate requests. Keep local reconciliation anyway; providers can have retention windows or endpoint-specific behaviour.

Separate selection from delivery

Claim one approved item transactionally before making a remote call. The claim prevents two runners from selecting the same item at the same time.

The worker should then use a small state machine:

async function deliver(item, adapter, store) {
  const attempt = await store.beginAttempt(item);

  try {
    const result = await adapter.publish(item, attempt.operationKey);
    await store.markPublished(attempt.id, {
      remoteId: result.id,
      publicUrl: result.url
    });
  } catch (error) {
    if (isDefiniteRejection(error)) {
      await store.markRejected(attempt.id, safeSummary(error));
      return;
    }

    await store.markUncertain(attempt.id, safeSummary(error));
  }
}

A validated 400 response is normally a definite rejection. A timeout, connection reset or invalid response body after the request was sent is not. A 500 response may also be uncertain if the provider could have completed the write before returning the error.

Reconcile before any retry

A reconciliation worker reads uncertain attempts and searches the destination using the strongest available evidence.

First, search by operation key when the platform exposes custom fields or request metadata. Otherwise, compare a bounded time window using destination account, normalized title or text, source URL and creation time. Require more than one weak attribute before declaring a match.

Do not compare only the first few recent posts. Paginate until the search crosses the attempt time plus a documented safety margin. Stop at a hard page limit and report that history verification was incomplete.

When exactly one credible match exists, save its remote ID and URL and mark the attempt published. When no match exists and the provider's consistency delay has passed, move the attempt to retry-approved rather than publishing immediately. When several matches exist, flag the attempt for review because a duplicate may already exist.

Make retry a deliberate transition

Only attempts in retry-approved may return to the delivery worker. Reuse the original operation key. Record who or what approved the retry and the evidence produced during reconciliation.

Limit attempts with a small retry budget and exponential backoff. Backoff protects the provider, but it does not solve duplication by itself. The safety property comes from stable identity and reconciliation.

For systems without remote search or idempotency support, the correct automatic action after an uncertain write may be no action. Manual inspection is slower than a retry, but safer than publishing repeated content under an account you cannot reconcile.

Verify the response before reporting success

A successful HTTP status is not enough. Require the documented identifier field and validate its type. Prefer the public URL supplied by the API. If the service returns only an ID, perform a follow-up read and use the verified URL from that response.

Keep publication counts tied to these receipts. Authentication checks, queue polling and dry runs should never increment a published counter.

Return a structured result from every adapter:

{
  "outcome": "published",
  "operationKey": "stable-key",
  "remoteId": "remote-record-id",
  "publicUrl": "https://example.test/posts/remote-record-id"
}

The URL is illustrative. A production adapter must use the actual destination response or a verified lookup.

Test the difficult boundaries

Use a fake adapter that records requests and can simulate a timeout after creating a remote record. The first run should leave the attempt uncertain. Reconciliation should discover the fake record, and the retry worker should make no second publish call.

Add tests for a definite rejection, an uncertain attempt with no remote match, paginated history, multiple candidate matches and a process crash after the remote write but before the local commit.

Also test two concurrent workers. Only one should acquire the claim. A reliable single-worker test does not prove that scheduled jobs cannot overlap in production.

Operate with evidence

A useful operational report separates four facts: the scheduler ran, a candidate was approved, the remote write was attempted, and a public record was verified. A green workflow badge proves only the first fact.

Log the operation key, internal content ID, outcome and remote receipt. Keep credentials redacted. Alert when uncertain attempts exceed their reconciliation window or when remote history cannot be read completely.

Recovery is not the same as repeating the request. The safe order is identify the intended operation, inspect the destination, reconcile the result and only then decide whether a retry is justified.