Provider webhooks and your own write path are two separate async processes, and nothing guarantees which one finishes first. Fire off a charge, get a 200 back, kick off the DB write to create the order — and the provider's webhook can land on your endpoint before that write commits, especially under load or a slow transaction.
If the handler does a lookup-or-404, you just told a payment provider your endpoint is broken, and depending on their retry policy you might not see that event again for minutes.
Options I've seen: retry-with-backoff inside the handler, an upsert that tolerates arriving first, or a short queue that holds unmatched events for a few seconds before giving up. Each trades latency for complexity differently.
Genuinely curious what other teams landed on — especially anyone who measured how often this race actually fires in production versus how much handling code it justified.
Top comments (0)