Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run
Every webhook provider's documentation has a sentence like this somewhere: "We don't guarantee delivery of events in the order in which they were generated."
Most teams read it, nod, and build a handler that assumes order anyway. It works for months, because most of the time events do arrive in order. Then one day customer.subscription.deleted arrives two seconds before customer.subscription.updated, the handler applies the update last, and a cancelled customer is active again.
There are two broad ways to deal with this, and people feel strongly about both. Let's argue about it.
Camp 1: put them back in order
The idea: buffer incoming events, sort them by the provider's timestamp or sequence number, and apply them in order.
Arguments for:
- Business logic stays simple. Handlers can assume "created, then updated, then deleted". No defensive code in every handler.
- It matches the provider's state machine. If the provider's object went through states A, B, C, applying events in that order reproduces it faithfully, including intermediate states you might care about (an "upgraded" event that triggers an email, for example).
- Audit trails make sense. Your event log reads like a history, not a puzzle.
Arguments against:
-
Timestamps aren't sequence numbers. Two events created in the same second have the same timestamp. Clocks on the provider's side may not even be monotonic across their own servers. Sorting by
createdis approximately right, which is the worst kind of right. - How long do you wait? To reorder, you have to hold events until you're "sure" nothing earlier is coming. Hold too briefly and you still get misordered events. Hold too long and everything is delayed. Retries can deliver an event minutes or hours later; you can't buffer that long.
- It adds a stateful component. A buffer that sorts events needs storage, locking per object, and its own failure handling. It's a small queueing system you now have to operate.
Camp 2: make order irrelevant
The idea: treat each webhook as a notification that something changed about an object, and make the handler converge on the correct state no matter what order events arrive in.
The two common techniques:
-
Fetch current state. On any event for subscription
sub_123, call the provider's API to get the subscription as it is now, and store that. The event's payload is a hint; the API response is the truth. -
Compare versions. If the provider includes a version, sequence number or reliable
updated_aton the object, store it and ignore any event whose object is older than what you already have.
Arguments for:
- It's correct regardless of timing. Late events, duplicates and retries all converge to the same end state.
- No buffering, no waiting. Each event is processed when it arrives.
- It handles missed events. If an event never arrives at all, the next event for the same object still brings you up to date.
Arguments against:
- Extra API calls. Fetching on every event costs requests, latency and rate-limit budget. For high-volume event types that can be a real problem.
- You lose intermediate states. If a subscription went from "trialing" to "active" to "past_due" in a minute, fetching current state shows only "past_due". If you needed to react to "active" (send a receipt), you missed it.
- Not every event is a state change. "Payment failed, attempt 2 of 4" is an occurrence, not a state. You can't fetch an occurrence.
Where I've landed
I've ended up with a split that I'd defend in most systems:
-
For objects with state (subscriptions, customers, orders), make order irrelevant. Use the provider's object version or
updated_atwhen it's reliable, fetch current state when it isn't, and never let an older version overwrite a newer one. - For occurrences (a payment attempt failed, an email bounced, a refund was issued), process each event once, idempotently by event ID, and don't depend on order at all. These are facts, and facts don't conflict.
- For side effects that depend on transitions ("send a welcome email when the subscription becomes active"), trigger them from your own state change, not from the incoming event. When your stored state goes from not-active to active, you emit the side effect, once. That works no matter which event caused it.
- Don't build a reorder buffer unless the provider gives you real sequence numbers per object. Sorting by timestamp just moves the bug.
Fetching current state after an event is a good fit for a short delayed job, not work inside the webhook request. delay.run can schedule that follow-up HTTP call a few seconds later, retry it if the provider's API is slow, and log every attempt.
The part that's still messy
-
Deletes. If
deletedarrives first and you remove the row, a lateupdatedmight recreate it. You need tombstones, a "deleted at" marker that wins over any older update. - Objects that span several providers. An order is created in your app, paid through a payment provider, and shipped through a logistics API. There's no single version number to compare.
- Rate limits. "Fetch on every event" is great until a bulk operation sends you 50,000 events in a minute. Debouncing (fetch once per object per few seconds) helps, but it's another moving part.
A quick test for your handlers
Take your three most important webhook event types and ask:
- What happens if the second event arrives before the first?
- What happens if the same event arrives twice, a minute apart?
- What happens if one event never arrives at all?
If any answer involves "the data would be wrong until someone notices", you're in camp 1 without the buffer, which is the one place nobody wants to be.
Related reading
- Should webhook handlers trust the payload, or fetch the latest state from the API? Let's argue about it
- Case study (hypothetical): the payment webhook that arrived before the order existed
- How to fix 'No signatures found matching the expected signature for payload' (Stripe webhooks in Express, Next.js and Flask)
- Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging
- Case study (hypothetical): the month-end usage billing job that charged 300 customers for the wrong month
Which camp are you in? Have you built a reorder buffer that actually worked, or did you end up fetching current state like most of us?
Top comments (0)