DEV Community: FlowPatch Reliability

When an n8n API timeout may already have committed

FlowPatch Reliability — Thu, 03 Sep 2026 16:51:28 +0000

An n8n HTTP Request node can fail even when the remote system has already accepted the request.

That is the most dangerous kind of failure: not a clean rejection, but an ambiguous commit.

Suppose a workflow does this:

Receive a lead webhook.
POST the lead to a CRM.
Wait 30 seconds.
The request times out.
Retry the workflow.

The retry may be correct if the CRM never received the first request. It may create a duplicate if the CRM committed the lead at second 29 but the response never reached n8n.

A green retry is not proof that the workflow is correct. It can mean the same business action happened twice.

The rule that prevents most duplicates

For every side-effecting request, generate one stable operation identity before the first attempt and reuse it for every retry.

operation_key = "create-lead:" + source_event_id

Do not generate the key inside the retrying node. Do not use the execution attempt ID if it changes across retries. The key must identify the intended business action, not the transport attempt.

If the provider supports idempotency keys, send that value in the provider's documented header. If it does not, store the key and provider result in a durable table you control.

A useful state model is:

prepared -> attempted -> confirmed
                    \-> ambiguous -> reconciled
                    \-> terminal_failure

ambiguous is a real state. Treating a timeout as a simple failed state erases the most important fact: the provider may have committed.

What the n8n workflow should persist

Before calling the provider, persist:

stable operation key;
source event ID;
request fingerprint;
provider name and endpoint;
state = prepared.

After sending, record the attempt. On a definite success, store the provider object ID and mark confirmed. On a definite rejection, record the safe-to-retry or terminal result.

On a timeout, broken connection, or malformed response, mark ambiguous. Do not immediately create a new business operation.

The next step should reconcile:

Look up the operation by provider idempotency key, external reference, or request fingerprint.
If the provider object exists, store its ID and mark confirmed.
If it definitely does not exist, retry with the same operation key.
If the provider cannot answer conclusively, route to bounded delayed retry or operator review.

Webhook callbacks do not remove the need for state

A common design is "POST the job, then wait for the callback." That still has two failure windows:

The provider accepts the job, but the initial response is lost.
The provider sends the callback, but your workflow crashes after the side effect and before durable acknowledgement.

Webhook handlers should durably accept first, deduplicate on the provider event ID, and make downstream side effects replay-safe. Returning 200 before durable acceptance can lose an event. Returning an error after the side effect can invite a duplicate delivery.

Five failure-path tests worth running

The happy path is only one test. Add these:

Timeout after provider commit

Simulate a committed object with a lost response. Verify the next run reconciles instead of creating a second object.
Duplicate source webhook

Deliver the same source event twice. Verify one business operation and one stored provider ID.
Crash after side effect

Stop execution after the provider accepts but before the workflow records success. Verify replay uses the same operation key.
Callback before listener registration

Deliver the provider callback immediately. Verify it is durably stored even if the waiting execution has not registered its in-memory listener yet.
Permanent 4xx

Return an authentication or validation error. Verify it does not enter the same retry loop as a transient 429 or 503.

A quick boundary review

For one n8n-to-provider boundary, write down four facts:

provider/API;
trigger;
observed failure;
current retry behavior.

Those four facts are usually enough to define the first failure map: where identity is created, what can commit, which outcomes are ambiguous, and how reconciliation should work.

I offer a fixed $99 n8n/API failure-path diagnostic for one boundary. It includes a boundary-specific risk matrix, retry and state rules, patch plan, failure-path tests, and an implementation estimate. Delivery is within two business days after written scope and cleared ACH; the fee is credited toward a $299 implementation patch purchased within 14 days.

The service is independent and is not affiliated with n8n.

The crash window that turns webhook retries into duplicate side effects

FlowPatch Reliability — Thu, 03 Sep 2026 14:27:08 +0000

A webhook handler can return 200 OK, log "processed", and still create the same charge, email, or entitlement twice.

The bug usually lives in a small crash window between acceptance and completion.

A provider only knows whether your endpoint accepted a delivery. It does not know whether your business operation completed exactly once. If the connection drops, your process crashes, or the response times out, the provider may retry. That retry is correct behavior. The duplicate side effect is our bug.

The dangerous sequence

A common handler looks harmless:

async function handleWebhook(event: ProviderEvent) {
  const seen = await db.events.find(event.id);
  if (seen) return;

  await sendWelcomeEmail(event.customerId);
  await db.events.insert({ id: event.id, status: "done" });
}

Now place a crash after sendWelcomeEmail succeeds but before the insert commits.

The provider sees no successful response and retries. The database still says the event is new, so the handler sends a second email. Replace email with createShipment, issueRefund, or grantCredits and the incident becomes expensive.

This is also why a check-then-insert guard is not enough. Two workers can receive the same event at nearly the same time. Both check, both see nothing, and both execute the effect before either inserts the marker.

Acceptance is not completion

Separate the inbound transport decision from the business-work decision.

Verify the signature and parse the event.
Atomically record the provider event ID under a unique constraint.
Return a success response once durable acceptance is complete.
Process the durable record asynchronously.

The unique insert decides which delivery owns the work. A duplicate delivery becomes a no-op at the database boundary rather than after an external side effect.

async function acceptWebhook(raw: Buffer, headers: Headers) {
  const event = verifyAndParse(raw, headers);

  const inserted = await db.transaction(async tx => {
    return tx.inbox.insertIfAbsent({
      provider: "stripe",
      eventId: event.id,
      payload: event,
      state: "pending",
      attempts: 0,
    });
  });

  return { status: 200, duplicate: !inserted };
}

The database needs a unique key such as (provider, event_id). Application-level checks without a constraint still race.

Make completion durable too

The worker processing the inbox record needs explicit states. pending, leased, completed, and terminal_failure are more useful than a single boolean.

A lease lets a crashed worker's job become eligible again. Store lease_until, increment attempts, and let another worker claim the row only after the lease expires. Do not hold a database transaction open while calling a remote provider.

If processing must update local state and enqueue another action, use a transactional outbox:

await db.transaction(async tx => {
  await tx.accounts.applyEntitlement(accountId, plan);
  await tx.outbox.insert({
    kind: "send_receipt",
    key: `receipt:${event.id}`,
    payload: { accountId, plan },
  });
  await tx.inbox.markCompleted(event.id);
});

The local state change, outbox entry, and completion marker now commit together. A separate dispatcher sends the receipt.

Stable idempotency keys cross the final boundary

An outbox makes local intent durable, but the dispatcher can still crash after the remote provider accepts a request and before local completion is recorded.

When the remote API supports idempotency, send a stable key derived from the logical operation, not from the attempt:

await emailProvider.send(message, {
  idempotencyKey: outbox.key,
});

Every retry of the same logical operation must reuse the same key. A random key per attempt defeats the mechanism.

When the remote API does not support idempotency, classify the operation honestly. Some actions are naturally safe to repeat. Some can be reconciled by querying the remote system. Some remain ambiguous and require a manual review state instead of blind retries.

Retry classification matters

A retry policy should distinguish:

transient failures: timeouts, connection resets, 429, and selected 5xx responses;
permanent failures: invalid payloads, authentication failures, and most 4xx responses;
ambiguous outcomes: the request may have succeeded, but the response was lost.

Use bounded exponential backoff with jitter. Honor Retry-After. After the attempt budget is exhausted, move the operation to a visible terminal state with enough context to investigate. Infinite retry is not resilience; it is a quiet denial-of-service against your own queue and the provider.

Reconciliation closes the gap

Even a strong design needs a repair loop. Periodically compare local records with the provider's authoritative state. Find accepted events stuck in leased, outbox records with ambiguous outcomes, and local entitlements that disagree with payment state.

Reconciliation turns an invisible inconsistency into a queued repair or an explicit alert.

Test the crash points, not only the happy path

A useful failure-path suite kills the process at each boundary:

before the inbox insert;
after the insert but before the response;
after claiming work but before the side effect;
after the side effect but before completion is recorded;
during outbox dispatch;
while two workers race on the same event.

Then replay the same provider event and assert the business invariant: one shipment, one entitlement transition, one refund, or one message.

Exactly-once delivery is rarely available end to end. What you can build is durable acceptance, atomic local transitions, stable idempotency at remote boundaries, bounded retries, and reconciliation. That combination makes at-least-once delivery safe enough for real systems.

A concrete implementation example

I applied the same contract in Posnic/POS PR #476, a focused webhook retry and dead-letter contribution now under maintainer review. I also isolated a separate lost sync-response race in Activepieces: a fast worker can publish before the API registers its listener, causing a false timeout after a successful run. The patch adds deterministic tests for timeout, 400, 500, success after retry, and permanent failure. It also verifies that retries reuse the same delivery ID, stop at a fixed attempt budget, persist only safe error categories, and strip shop/time details from dead-letter records.

That distinction is worth testing explicitly: a transport timeout and a 500 are candidates for bounded retry; most 4xx responses are permanent until the request changes. Treating them all alike either drops recoverable work or hammers a request that can never succeed.

Use this 7-question preflight checklist to review an integration. If one provider boundary is already failing, FlowPatch offers a $99 fixed-scope failure-path diagnostic with a risk matrix, retry/state rules, patch plan, and failure-path tests.