Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run
Sooner or later this happens: a deploy breaks your webhook handler for a few hours. The provider retries for a while, then gives up. You fix the bug, and now there's a gap. Payments that happened, subscriptions that changed, orders that shipped, and your database knows nothing about them.
Some providers let you resend events from their dashboard or CLI, one by one or in bulk. That's the first thing to check. But not all do, retention windows are often short, and resending through the provider means waiting for their delivery queue.
The more reliable approach is to keep your own copy of every webhook request, and replay from it. This post shows the table and a small replay script.
Step 1: store every webhook before processing it
The trick is to separate receiving from processing. The HTTP handler does as little as possible: verify the signature, store the raw request, return 200.
CREATE TABLE webhook_inbox (
id bigserial PRIMARY KEY,
provider text NOT NULL,
event_id text NOT NULL,
event_type text NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
headers jsonb NOT NULL,
body bytea NOT NULL, -- raw bytes, exactly as received
status text NOT NULL DEFAULT 'pending', -- pending | done | failed
attempts int NOT NULL DEFAULT 0,
last_error text,
UNIQUE (provider, event_id)
);
Store the raw bytes of the body, not parsed JSON. Signature verification works on raw bytes, and you want to be able to re-verify later if you need to prove an event is genuine.
The UNIQUE (provider, event_id) constraint gives you free deduplication of provider retries: insert with ON CONFLICT DO NOTHING.
A separate worker (or a short delayed job per event) reads pending rows, runs the business logic, and marks them done or failed with the error.
Step 2: make the processing function callable directly
The business logic should be a plain function that takes the stored row, not something buried in an HTTP route:
def process_event(row) -> None:
event = json.loads(bytes(row["body"]))
handler = HANDLERS[event["type"]] # e.g. "invoice.paid" -> handle_invoice_paid
handler(event)
This is what makes replays easy and safe: the replay script calls exactly the same function the worker does. No re-signing, no fake HTTP requests, no special code path.
Step 3: the replay script
#!/usr/bin/env python3
"""Replay failed or unprocessed webhooks from webhook_inbox."""
import argparse, time, traceback
import psycopg
from psycopg.rows import dict_row
from app.webhooks import process_event # the same function the worker uses
p = argparse.ArgumentParser()
p.add_argument("--since", required=True, help="e.g. 2026-11-16T10:00Z")
p.add_argument("--until", required=True)
p.add_argument("--status", default="failed", choices=["failed", "pending"])
p.add_argument("--type", help="only this event type")
p.add_argument("--rate", type=float, default=5.0, help="events per second")
p.add_argument("--apply", action="store_true", help="actually process (default: dry run)")
args = p.parse_args()
sql = """SELECT * FROM webhook_inbox
WHERE status = %(status)s AND received_at >= %(since)s AND received_at < %(until)s
AND (%(type)s::text IS NULL OR event_type = %(type)s)
ORDER BY received_at, id"""
with psycopg.connect(autocommit=True, row_factory=dict_row) as conn:
rows = conn.execute(sql, vars(args)).fetchall()
print(f"{len(rows)} event(s) to replay{'' if args.apply else ' (dry run)'}")
ok = failed = 0
for row in rows:
label = f"{row['id']} {row['event_type']} {row['event_id']}"
if not args.apply:
print("would replay", label); continue
try:
with conn.transaction():
process_event(row)
conn.execute("UPDATE webhook_inbox SET status='done', attempts=attempts+1, "
"last_error=NULL WHERE id=%s", [row["id"]])
ok += 1; print("ok ", label)
except Exception:
failed += 1
conn.execute("UPDATE webhook_inbox SET attempts=attempts+1, last_error=%s "
"WHERE id=%s", [traceback.format_exc()[-2000:], row["id"]])
print("FAILED", label)
time.sleep(1 / args.rate)
print(f"done: {ok} ok, {failed} failed")
Run it in two steps:
python3 replay.py --since 2026-11-16T10:00Z --until 2026-11-16T14:30Z
python3 replay.py --since 2026-11-16T10:00Z --until 2026-11-16T14:30Z --apply --rate 2
Why each piece is there
Dry run by default. The first run just lists what would be replayed. You'll often spot something surprising, like 4,000 events when you expected 400, before anything happens.
Ordered by received_at. Replaying in roughly the original order reduces surprises, but don't rely on it; webhooks can arrive out of order in the first place. Handlers should already be safe in any order.
One transaction per event. The connection runs in autocommit mode, so each conn.transaction() block is its own transaction. The business logic and the status update commit together. If processing fails, nothing is half-applied, and the row stays failed with the new error.
Rate limit. Handlers often call the provider's API or send emails. Replaying 2,000 events as fast as possible can hit rate limits and create a second incident.
Same function as production. A replay path that differs from the normal path is a replay path that has never been tested.
Replays are only safe if the handler is idempotent and you can see every attempt. delay.run works well for the processing half: one delayed HTTP job per stored event, retried with backoff, with a run log, so a failed event retries on its own instead of waiting for a manual replay.
Before you replay, check idempotency
Replaying assumes that processing an event twice is harmless. Make sure:
- Every handler checks whether it has already applied the event (a processed-events table, or a unique constraint on the business record).
- Side effects like emails are keyed so they don't go out twice.
- "Add" operations (credits, seats, balances) are written as "set" operations, or guarded by the event ID.
If any handler isn't idempotent, replay that event type separately, with a careful look at each row.
Retention
Keep the inbox for at least as long as your longest plausible incident plus investigation time; 30 to 90 days is common. The bodies contain customer data, so apply the same retention and access rules as the rest of your database, and drop old rows on a schedule.
Related reading
- The transactional outbox, explained: how to save to the database and enqueue a job without losing either
- 8 free tools for testing webhooks on localhost, and when to use each
- How to verify HMAC-SHA256 signed webhooks in Node and Python
- Retries and backoff FAQ: 11 questions about retrying failed jobs without making things worse
- GitHub Actions scheduled workflow not running (or running late)? 8 causes and how to fix each
Do you keep raw copies of incoming webhooks? I'm curious how many teams replay from their own inbox versus relying on the provider's resend button.
Top comments (1)
I ran your table and
replay.pyas written against PostgreSQL 16.4 with psycopg 3.3.9, using a toycredit.addedhandler that inserts into aledgertable. With 5 failed rows the dry run listed 5,--applyreplayed 5, and each event got exactly one ledger row. Two cases broke it, though.conn.process_event(row)takes no connection, so a handler that uses the app's own connection or pool commits on its own. I made evt_3 insert its ledger row and then raise (say, an email provider 503). The script marked itfailed, but the credit stayed. After the bug was fixed, running the same replay again credited it a second time:--applyat the same time both replay every row, because the SELECT happens once up front and nothing claims a row. With a 0.2 s handler, both runs printeddone: 5 ok, 0 failed, and every event ended withattempts2 and 2 ledger rows. The same thing can happen with--status pendingif the normal worker is still running.I fixed both by passing
connintoprocess_event(row, conn)and re-reading each row inside the transaction:After that change, evt_3's failed attempt left 0 ledger rows and the retry left 1. The two concurrent runs split the work (2 ok + 3 skipped, and 3 ok + 2 skipped), so each event got 1 row.
One smaller thing:
--since 2026-11-16T10:00without aZis read in the session's TimeZone. On my machine (Asia/Dubai) that window matched 0 rows, when the UTC window matched 5. It's worth requiring the offset in argparse.Do your handlers get the connection passed in, or do they open their own?