Webhook order is not guaranteed, even from one provider

Assumed webhooks would arrive in the order events happened. They don't, and the gap shows up on refunds.

Sequence that broke a reconciliation job: charge succeeds, provider fires the charge webhook, our endpoint is slow and times out, provider queues a retry with backoff. Merchant issues a refund seconds later. Refund webhook has no retry needed, goes out immediately, lands before the delayed charge webhook retry completes. Our handler processes "refund" for a charge that, according to our own database, does not exist yet.

Fix was not bigger retries or a longer timeout. It was making the handler check current state before applying any event instead of assuming events arrive as a sequence. Refund webhook checks: does this charge exist as completed? If not, park the event and reprocess after the charge webhook lands, don't drop it and don't error into a dead letter queue silently.

Provider docs list delivery guarantees (at-least-once, retry schedule) but almost none document ordering guarantees between different event types tied to the same object. Timestamps in the payload help, but only if you're willing to hold and resequence, which most webhook handlers aren't built to do.

Anyone handling this with a full event-sourced ledger instead of state checks per event type? Curious if that's overkill for a mid-volume system or the actual right answer.