DEV Community

delay.run
delay.run

Posted on Edited on

Case study (hypothetical): one checkout, three webhook retries, three subscriptions for the same customer

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

This is a hypothetical case study. The product and numbers are made up, but the sequence of events will be familiar to anyone who has run a payment webhook in production.

The setup

A B2B SaaS app sells team plans. When a customer completes checkout on the payment provider's hosted page, the provider sends a checkout.session.completed webhook. The app's handler:

  1. Verifies the signature.
  2. Looks up the customer's organisation.
  3. Creates a Subscription row in the app's database.
  4. Calls the provider's API to add the purchased number of seats to that subscription.
  5. Sends a welcome email to every user in the organisation.
  6. Returns 200 OK.

It had worked for two years. Then, one Tuesday, a large customer bought 40 seats and their finance team noticed three active subscriptions on the next invoice.

What happened

10:14:02. The customer completes checkout. The provider sends the webhook.

10:14:02 to 10:14:33. The handler runs. Step 5, the welcome emails, calls the email provider once per user. The organisation has 40 users, and the email API is having a slow morning: about 700 ms per call. The handler is still sending emails at the 20-second mark.

10:14:22. The payment provider's delivery timeout (in this hypothetical, 20 seconds) passes without a response. From its point of view, the delivery failed. It schedules a retry.

10:14:33. The first handler finishes and returns 200, to a connection nobody is listening to anymore.

10:15:24. The retry arrives. Same event ID, same payload. The handler doesn't check for that. It creates a second Subscription row, adds 40 more seats through the API, and starts sending welcome emails again. The email API is still slow. The handler times out again.

10:19:30. A third delivery. Third subscription. By now, every user has received three welcome emails, and someone on the customer's side has opened a support ticket asking why.

The retries stopped there only because the third attempt finished just under the timeout.

Nothing was "broken"

This is what makes these incidents frustrating. Every component did what it was supposed to:

  • The provider guarantees at-least-once delivery. If it isn't sure you got the event, it sends it again. That's the correct behaviour; the alternative is losing payments.
  • The handler processed every request it received correctly, in isolation.
  • The email API was slow, but up.

The bug was in the design: a handler that wasn't safe to run twice, doing slow work before responding.

A webhook handler should acknowledge fast and hand the slow part to something that can retry it safely. delay.run takes that follow-up as a delayed HTTP job, retries it with backoff, and keeps a log of every attempt, so the provider gets its 200 in milliseconds.

The fix, in three parts

1. Record the event ID first, and make it unique

The first thing the handler now does after verifying the signature is insert the event ID into a processed_events table with a unique constraint:

CREATE TABLE processed_events (
  event_id     text PRIMARY KEY,
  received_at  timestamptz NOT NULL DEFAULT now()
);
Enter fullscreen mode Exit fullscreen mode
try:
    db.execute("INSERT INTO processed_events (event_id) VALUES (%s)", [event.id])
except UniqueViolation:
    return 200   # already handled (or being handled): acknowledge and stop
Enter fullscreen mode Exit fullscreen mode

The insert and the business logic run in the same transaction, so if the handler crashes before committing, the event ID isn't recorded and a retry will process it properly.

2. Make the business operation idempotent too

Event-level deduplication covers retries of the same event. It doesn't cover two different events that mean the same thing, which some providers send (a checkout.session.completed and an invoice.paid for the same purchase, for example). So the subscription itself got a natural key: the provider's subscription ID, with a unique index.

CREATE UNIQUE INDEX subscriptions_provider_id ON subscriptions (provider_subscription_id);
Enter fullscreen mode Exit fullscreen mode

Creating the subscription became an upsert. Adding seats changed from "add 40 seats" (not idempotent) to "set seats to 40" (idempotent), and the API call passes an idempotency key derived from the event ID, so a repeated call doesn't add seats twice on the provider's side either.

3. Return 200 quickly, do the slow work later

The handler now does only the database work, which takes milliseconds, and returns 200. The welcome emails go into a delayed job, keyed by subscription ID, that runs a few seconds later with its own retries. If the email API is slow, the emails are slow. The webhook isn't.

Cleaning up

The team refunded the duplicate subscriptions, merged the seat counts, and apologised for the triple emails. They also searched for other duplicates: a query for organisations with more than one active subscription found four more customers, from smaller incidents over the previous year that nobody had connected.

What I'd take from it

  • Assume every webhook will arrive at least twice. Sometimes it will. Design the handler so the second time is a no-op.
  • Dedupe on the event ID, and make the underlying operation idempotent. You need both.
  • "Add N" is dangerous; "set to N" is safe. The same applies to credits, seats, balances and quantities.
  • Know your provider's timeout, and stay far below it. Do slow work after responding, in a job that can be retried on its own.
  • Look for duplicates periodically. A daily query for "more than one active subscription per organisation" would have found this pattern a year earlier.

Related reading

Have you been bitten by a webhook retry creating duplicate records? What was the slow step that caused the timeout in your case?

Top comments (0)