A silently dropped webhook, recovered five minutes later: a reconciliation drill you can run locally

Webhooks are at-least-once, not exactly-once. They get retried, duplicated, delayed, and sometimes they never arrive at all. When that happens the money has moved but the merchant order has not, and nobody notices until a customer complains.

I built a small drill that reproduces the worst version of this on purpose, on a laptop, in about a minute of real time:

1. The merchant creates a Stripe order in a local emulator.
2. The emulator drops payment_intent.succeeded for that payment.
3. The provider intent is succeeded, while the merchant order stays PENDING — money moved, business state did not.
4. The sandbox clock advances five minutes. The scheduled reconciliation task pulls the payment intent, sees succeeded, and recovers the order: status=PAID, paidSource=reconciliation.

The recording shows the drill steps on the left and the matching provider and merchant log lines on the right. Every step has its own log lines and starts with a dashed separator and a short pause: order created, intent confirmed, dropped delivery, zero webhook deliveries, clock advanced, and finally the reconciliation scan that recovers the order.

The whole project is a multi-provider payment sandbox: Stripe-style, PayPal-style, card acquirer and crypto emulators, plus a reference merchant that verifies signatures and handles webhooks idempotently.

Why reconciliation, and not just retries

Retries only help when the delivery reaches you and your handler fails. They do not help when:

• the webhook is dropped or lost,
• your service is down for the whole retry window,
• the endpoint was misconfigured and every attempt was rejected,
• the platform itself has an incident.

In all of those cases the provider still knows the truth. A scheduled pull that compares local state against provider state is the fallback that closes the gap.

What the drill shows

merchant order : order_102
payment intent : pi_1001
provider says  : succeeded
merchant says  : PENDING
webhook        : never arrived (dropped by the emulator)

...five minutes of sandbox time later...

status         : PAID
paid source    : reconciliation
recovered at   : 2026-09-11T08:57:16Z

summary
  webhook deliveries   : 0 (dropped)
  reconciliation scans : 1
  orders recovered     : 1

Run it locally:

git clone https://github.com/pqa-labs/pqa-payments-sandbox
cd pqa-payments-sandbox
./scripts/reconciliation-drill.sh

Design notes worth copying

• Reconciliation is pull-based and idempotent. A second scan does not touch an already recovered order. There is a test for the double-recovery case.
• Provider status is the source of truth: succeeded (Stripe), COMPLETED (PayPal), CAPTURED (card).
• Recovered orders carry paidSource=reconciliation, so you can tell webhook-driven state from reconciliation-driven state in reports and audits.
• The sandbox clock makes a five minute window visible in seconds, which is what makes the demo watchable.
• One deliberate gap: crypto deposits are skipped because the merchant stores a deposit address, not a queryable deposit id. Finding and fixing that is a good exercise.

What to test in your own system

1. Drop a webhook on purpose and confirm the order still ends up correct.
2. Deliver the same webhook twice and confirm nothing is written twice.
3. Restart the service in the middle of the retry window and check recovery.
4. Assert that a recovered order is not recovered twice.
5. Make sure a reconciliation run is visible in metrics or logs, not silent.

The repo is pqa-labs/pqa-payments-sandbox. It is a local sandbox, not a production system, and it should never be used with real payment data.

I spent years doing both development and testing for cross-border payment systems: payment integrations and SDK upgrades, test automation frameworks, payment flow testing, dropped-order triage, and reconciliation fallbacks. I now focus on payment verification for products going global, and I also take on integration and automation work. Happy to compare notes.

Comments

[pqa-labs]

One design detail I left open on purpose: the reconciliation window. The drill uses five minutes, but the right number depends on the provider's retry policy and how long an order can stay payment-pending before it hurts the customer. Curious how others set this: a fixed schedule (every N minutes), event-driven after a timeout, or tied to provider settlement? And do you page someone when the provider says paid but the local order stays pending beyond a threshold, or is a metric enough?