Webhook retries: test the gap between database commit and external calls

A webhook sender can retry after your application has already committed an order update. Perhaps the response was lost. If the handler also creates a shipment, the retry can create another shipment unless each persistence boundary has its own protection.

This article and illustrative code were prepared with AI agents for Ekyon. The local experiment was executed; its scope and limitations are stated below.

This tutorial uses an original Node and SQLite experiment with a fictitious payment event. It exercises database rollback, concurrent deliveries and a simulated lost acknowledgment. It describes no customer incident or production result. You can run the complete file below without an HTTP service, external provider or package installation.

Give the event and the business operation different identities

The receipt key is (account, event). SQLite's uniqueness-conflict handling lets the receipt insert skip an existing key. Its unique constraint answers: “Have we already committed processing for this upstream event?” Scope it to the provider's actual identity rules; an event identifier may only be unique within an account.

The operation key is tenant-a/order-1/fulfill/v1. It answers a different question: “Have we already scheduled this business effect?” Two distinct payment events can refer to the same paid order. Receipt deduplication alone would accept both.

Here the business guard only transitions awaiting_payment to paid. Only that transition creates a fulfillment obligation in the outbox. Real partial payments, refunds and replacement shipments need additional states and identities. Never reuse a fulfillment key for an intentionally different shipment.

Keep the receipt, transition and obligation together

The receiver uses one transaction for all three changes. A failure before outbox insertion rolls back the receipt and order update too. Committing the receipt first would be dangerous: a retry could see “already processed” even though the required work never committed.

The outbox row means that work is owed. It does not mean that the external side effect succeeded. An HTTP handler built around this approach should acknowledge successful processing after its transaction commits. Authentication, signature verification, event validation and account-to-tenant authorization belong before the illustrated receiver.

Run the complete experiment

Save this as webhook-retries.mjs and run node webhook-retries.mjs. It was tested with Node 22.17.1, whose built-in SQLite module prints an experimental-feature warning. The script uses a temporary database and removes it afterward. The hardcoded order and account keep the failure boundaries visible; they are not a reusable payment implementation.

// Local illustrative experiment. Not an HTTP server or production integration.
import assert from 'node:assert/strict';
import { DatabaseSync } from 'node:sqlite';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { spawn } from 'node:child_process';

function connect(path) {
  const db = new DatabaseSync(path);
  db.exec('PRAGMA busy_timeout = 10000;');
  return db;
}

function receive(db, eventId, fail = false) {
  db.exec('BEGIN IMMEDIATE');
  try {
    const receipt = db.prepare(`INSERT INTO receipts VALUES ('account-a', ?)
      ON CONFLICT DO NOTHING`).run(eventId);
    if (receipt.changes) {
      const changed = db.prepare(`UPDATE orders SET status='paid'
        WHERE id='order-1' AND status='awaiting_payment'`).run();
      if (changed.changes) {
        if (fail) throw new Error('injected before outbox insertion');
        db.prepare('INSERT INTO outbox VALUES (?, ?, 0)').run(
          'tenant-a/order-1/fulfill/v1', '{"order":"order-1"}');
      }
    }
    db.exec('COMMIT');
  } catch (error) {
    db.exec('ROLLBACK');
    throw error;
  }
}

function concurrentDelivery(path, eventId) {
  return new Promise((resolve, reject) => {
    const child = spawn(process.execPath,
      [fileURLToPath(import.meta.url), 'worker', path, eventId],
      { stdio: ['ignore', 'ignore', 'pipe'] });
    let stderr = '';
    child.stderr.on('data', chunk => { stderr += chunk; });
    child.on('error', reject);
    child.on('exit', code => code === 0 ? resolve() : reject(new Error(stderr)));
  });
}

if (process.argv[2] === 'worker') {
  const db = connect(process.argv[3]);
  try { receive(db, process.argv[4]); } finally { db.close(); }
} else {
  const dir = mkdtempSync(join(tmpdir(), 'webhook-retries-'));
  const path = join(dir, 'local.sqlite');
  const db = connect(path);
  try {
    db.exec(`
      PRAGMA journal_mode=WAL;
      CREATE TABLE receipts (
        account TEXT NOT NULL, event TEXT NOT NULL,
        PRIMARY KEY(account,event));
      CREATE TABLE orders (id TEXT PRIMARY KEY, status TEXT NOT NULL);
      CREATE TABLE outbox (
        operation_key TEXT PRIMARY KEY, payload TEXT NOT NULL, sent INTEGER NOT NULL);
      CREATE TABLE fake_provider_effects (
        operation_key TEXT PRIMARY KEY, payload TEXT NOT NULL);
      INSERT INTO orders VALUES ('order-1', 'awaiting_payment');
    `);
    const count = table => db.prepare(`SELECT count(*) AS n FROM ${table}`).get().n;
    assert.throws(() => receive(db, 'evt-17', true), /injected/);
    assert.equal(count('receipts'), 0);
    assert.equal(count('outbox'), 0);
    assert.equal(db.prepare('SELECT status FROM orders').get().status, 'awaiting_payment');
    console.log('PASS: injected transaction failure leaves receipt, order and outbox unchanged');

    await Promise.all(Array.from({ length: 16 }, () => concurrentDelivery(path, 'evt-17')));
    assert.equal(count('receipts'), 1);
    assert.equal(count('outbox'), 1);
    assert.equal(db.prepare('SELECT status FROM orders').get().status, 'paid');
    console.log('PASS: 16 concurrent processes produce one receipt and one obligation');

    receive(db, 'evt-18');
    assert.equal(count('receipts'), 2);
    assert.equal(count('outbox'), 1);
    console.log('PASS: a distinct event for the paid order creates no second obligation');

    // Model a provider with stable keys. Commit its effect separately, then
    // omit the local completion update to simulate an acknowledgment gap.
    function fakeSend(operation) {
      db.prepare(`INSERT INTO fake_provider_effects VALUES (?, ?)
        ON CONFLICT DO NOTHING`).run(operation.operation_key, operation.payload);
      const stored = db.prepare(`SELECT payload FROM fake_provider_effects
        WHERE operation_key = ?`).get(operation.operation_key);
      assert.equal(stored.payload, operation.payload, 'key reused with changed payload');
    }
    const operation = db.prepare('SELECT * FROM outbox').get();
    fakeSend(operation); // success, then simulate lost acknowledgment
    assert.equal(db.prepare('SELECT sent FROM outbox').get().sent, 0);
    fakeSend(operation); // retry uses persisted key and payload
    db.prepare('UPDATE outbox SET sent=1 WHERE operation_key=?').run(operation.operation_key);
    assert.equal(count('fake_provider_effects'), 1);
    console.log('PASS: simulated lost acknowledgment retries without a second fake effect');

    db.exec('DELETE FROM receipts');
    receive(db, 'evt-17');
    assert.equal(count('receipts'), 1);
    assert.equal(count('outbox'), 1);
    console.log('PASS: receipt deletion allows replay but surviving business state prevents a second obligation');
  } finally {
    db.close();
    rmSync(dir, { recursive: true, force: true });
  }
}

Read the five results as separate guarantees

The executed experiment exited successfully with all five PASS groups:

1. Rollback: injecting failure before outbox insertion leaves zero receipts, zero obligations and the order awaiting payment.
2. Concurrent duplicate delivery: 16 separate Node processes contend for the same event. They leave one receipt, one paid order and one obligation.
3. Distinct event, same order: processing evt-18 creates another receipt but no second obligation, because the order has already transitioned.
4. Lost acknowledgment: the fake provider accepts the persisted key twice while retaining one effect. The outbox remains unsent between those calls, modeling the decision to retry after an ambiguous outcome.
5. Receipt deletion: deleting receipts allows evt-17 back in. The surviving order state prevents another obligation.

The concurrency test uses SQLite's BEGIN IMMEDIATE and busy timeout to serialize contending writers. SQLite permits one simultaneous writer, including in WAL mode. It does not establish PostgreSQL isolation behavior or prove scaling characteristics. The lost-acknowledgment test simulates the persistence gap by withholding the local status update. It does not kill a process, send HTTP requests or reproduce a real provider's failure modes.

The external call needs another contract

A database transaction cannot atomically commit an unrelated provider's shipment and your local sent flag. If the provider accepts the request and the worker crashes, the next attempt cannot tell from that flag whether the shipment exists.

The fake provider solves this with a unique operation key. A real integration needs a documented idempotency contract or a reliable lookup and reconciliation protocol. Persist the key and immutable request before the first attempt. Retry both together; do not rebuild the payload from mutable order data. Handle key/payload mismatches explicitly.

The fake provider retains keys indefinitely. A real provider may expire them. Retrying an old obligation after that window can create a second effect. An outbox alone does not remove this risk.

What to add before deploying this pattern

Multiple workers need a claim protocol, such as short database transactions with expiring leases. Commit a claim before making the network call. Require the current lease token when recording completion so a stale worker cannot overwrite a newer claim. A lease can expire while an old worker is still running, so downstream idempotency remains necessary. This experiment implements no leases.

Keep pending obligations observable: age, attempts, failures and ambiguous provider outcomes. Back off transient errors and give permanent failures a recoverable operator workflow. A failed obligation is still owed work.

Choose receipt retention from the sender's redelivery horizon and your operational replay needs. The deletion test demonstrates why a receipt's lifetime differs from the lifetime of the business guard. Deleting receipts, order history and operation identities together removes the protection that made this replay safe.

Finally, test reordered events, missing orders, conflicting payloads, tenant boundaries, expired provider keys and crash recovery against the actual systems. This receiver assumes one known order; it does not reject a missing order or reconcile a refund arriving first. The five passing groups support this small experiment's claims, not production readiness or universal exactly-once delivery.

---

Prepared for Ekyon, co-founded by Dhairya Purohit. The team's code audit overview explains the scope, evidence and limitations of a review.