delay.run

Posted on Nov 4 Edited on Nov 4

Case study: redesigning a failed-payment retry schedule that retried every card at 00:00 UTC

#payments #scheduling #backend #architecture

This is a hypothetical case study. The company and details are invented to illustrate a common design problem; the patterns and code are real.

The starting point

"Tallyhub" is a small subscription SaaS. When a renewal charge fails (expired card, insufficient funds, a bank's fraud check), it retries. The original implementation was a single nightly job:

# runs at 00:00 UTC via cron
for invoice in Invoice.objects.filter(status="past_due"):
    if invoice.attempts < 4:
        charge(invoice)

It worked well enough until the team looked closely at three complaints.

1. Every retry happened at midnight UTC. For customers in Asia-Pacific that's mid-morning, which is fine. For customers on the US West Coast it's late afternoon. For some European customers it falls just after midnight, before many paydays have cleared. The retry timing was effectively random relative to when customers had money in their accounts.

2. Retries ignored when the failure happened. An invoice that failed at 23:58 was retried two minutes later. One that failed at 00:05 waited almost 24 hours. "Retry 1" meant anything from minutes to a day.

3. Overlapping runs double-charged twice in one quarter. A slow run (the payment API was degraded) was still going when an operator ran the job manually to "catch up". Both processes picked up the same invoices.

The redesign: one scheduled job per retry

The team replaced "scan everything nightly" with one scheduled job per invoice attempt. When a charge fails, the system decides when to retry that invoice and schedules exactly one job for that time.

Step 1: A retry policy relative to the failure

from datetime import timedelta

RETRY_OFFSETS = [timedelta(days=1), timedelta(days=3), timedelta(days=5), timedelta(days=7)]

Retry 1 is now always about a day after the failure, regardless of when it happened.

Step 2: Land retries at a sensible local time

Rather than an arbitrary UTC moment, retries are moved to mid-morning in the customer's billing timezone, with jitter so thousands of invoices don't all fire at 10:00:00.

import random
from datetime import datetime, time
from zoneinfo import ZoneInfo

def next_retry_at(failed_at_utc: datetime, attempt: int, tz_name: str) -> datetime:
    tz = ZoneInfo(tz_name or "UTC")
    target = (failed_at_utc + RETRY_OFFSETS[attempt]).astimezone(tz)
    local = datetime.combine(target.date(), time(10, 0), tzinfo=tz)
    local += timedelta(minutes=random.randint(0, 90))     # spread the load
    return local.astimezone(ZoneInfo("UTC"))

Whether mid-morning local time actually improves recovery is something to measure, not assume. The team logged attempt time and outcome so they could compare later (more on that below).

Step 3: Schedule exactly one job, with an idempotency key

def on_charge_failed(invoice, attempt):
    if attempt >= len(RETRY_OFFSETS):
        mark_uncollectible(invoice)
        return
    run_at = next_retry_at(now_utc(), attempt, invoice.customer.timezone)
    scheduler.schedule(
        run_at=run_at,
        url="https://api.tallyhub.example/internal/retry-charge",
        body={"invoice_id": invoice.id, "attempt": attempt + 1},
        idempotency_key=f"retry:{invoice.id}:{attempt + 1}",
    )

The scheduler can be a database table plus a worker, a job queue with delayed jobs, or an HTTP scheduling service such as delay.run. The important property is the same: one job, one time, one key.

Step 4: Make the handler safe to run twice

Even with a unique key, the handler assumes it might be called twice (retries, redeploys, operator mistakes). It takes a row lock and re-checks state before charging:

@transaction.atomic
def retry_charge(invoice_id: int, attempt: int):
    inv = Invoice.objects.select_for_update().get(id=invoice_id)
    if inv.status != "past_due" or inv.attempts >= attempt:
        return "skipped"                     # paid already, or this attempt already ran
    inv.attempts = attempt
    inv.save(update_fields=["attempts"])
    result = charge(inv, idempotency_key=f"charge:{inv.id}:{attempt}")
    ...

Two layers of protection: the database check stops duplicate work, and the payment provider's idempotency key stops duplicate charges if the same request is sent twice.

Step 5: Cancel retries when they're no longer needed

If the customer updates their card and pays manually, pending retries should disappear. Store the scheduled job's ID on the invoice and cancel it when the invoice is paid. The handler's state check is the backstop if cancellation is missed.

What changed (and what they measured)

Without inventing numbers, here's what the team set out to measure, and what you should measure too if you try this:

Recovery rate per attempt number, before and after.
Recovery rate by local hour of attempt, to test whether the 10:00 choice helps or whether another time is better for their customers.
Duplicate-charge incidents (target: zero).
Load shape on the payment API: one midnight spike versus a spread-out curve.

The design win doesn't depend on the numbers, though. The midnight batch made it impossible to even ask whether timing mattered, because every retry happened at the same moment. Per-invoice scheduling turned retry timing into something you can tune.

Testing the schedule before it touches real cards

A retry schedule is easy to get subtly wrong, so the team tested it without a payment provider in the loop. Three checks did most of the work:

Freeze the clock. The function that computes retry times takes now and the customer's timezone as arguments instead of reading the system clock. Unit tests then cover the awkward cases: a failure at 23:59 local time, a customer in UTC+13, and a retry that lands on a daylight-saving change.
Replay a real month. They exported last month's failed invoices (IDs, timestamps and timezones only, no card data) and ran them through the new scheduler in a dry-run mode that logs planned retry times instead of creating jobs. A histogram of the planned times showed the midnight spike was gone.
Fire every job twice in staging. The handler was called twice for each retry with the same idempotency key. The second call had to be a no-op, with one charge attempt and one log line saying "already handled".

None of this needs special tooling, and it caught two bugs before launch: one off-by-one-day error around month ends, and a missing timezone that defaulted to UTC.

Lessons that generalize

"Scan everything on a schedule" hides per-item timing. If each item has its own clock, give it its own job.
Relative delays beat fixed times for anything triggered by an event.
Every money-moving job needs two idempotency layers: your own state check and the provider's key.
Cancellation is a feature. Design it in, but don't rely on it alone.
Log the inputs you'll want to tune later, such as attempt time, local hour and outcome, from day one.

DEV Community