Rate-Limited Reminder Recovery: Idempotent Queues After Webhook and Email Send Failures

#python #queue #webhooks

Short answer: put each user reminder on a queue, make the consumer idempotent, retry transient delivery failures with backoff, and move exhausted work to a dead-letter queue (DLQ) for deliberate redrive. For a rate-limited e-commerce worker pool, the best queue is the one whose recovery controls your team can operate at 02:00, not the one with the longest feature list.

This choice starts with an evaluation constraint: a checkout follow-up may be delivered once, but it must never be intentionally sent twice just because a worker lost its lease. Email and webhook providers can time out or return HTTP 429. Queue delivery is at-least-once, so the awkward case is a worker that sends successfully and crashes before acknowledging the message. Retries are expected. Duplicate side effects aren't.

The simple approach is to schedule a callback and call the provider directly. It looks fine in a notebook. In production, however, that design mixes timing, provider rate limits, retries, and recovery in one process. A queue between scheduling and delivery gives each concern a visible boundary: producers describe intent, workers spend the provider's rate budget, and operators inspect poison messages without stopping healthy reminders.

How should user reminder queues retry webhook and email send failures?

Classify outcomes before choosing retry numbers. A timeout or HTTP 429 is usually a retry candidate; malformed destination data is not made healthier by ten identical attempts. Honor Retry-After when it is present, otherwise use exponential backoff with jitter. Keep the attempt count and the next eligible time with the job so a restarted worker doesn't forget where it was.

A dead-letter queue is part of that state machine, not a graveyard. When attempts are exhausted, record enough context to answer three questions: which reminder failed, which destination was involved, and which error class made it terminal. Redrive only after the underlying cause or data has changed. Blind redrive can recreate the same rate spike that caused the backlog.

Keep payloads small. A reminder job should normally carry identifiers, a template version, and an idempotency key rather than a rendered email or a customer record. The worker can load current data under the application's normal authorization rules. This also keeps the message comfortably below systems with a 256KB body limit.

Pull when private. Push delivery is useful only when a public HTTPS consumer can receive jobs; an internal worker without a public endpoint should pull instead. That's a deployment boundary, not a minor toggle.

The recovery invariant matters more than the retry library

The invariant is compact: for one logical reminder, the externally visible send state advances at most once, even if the queue presents the message repeatedly. Give every reminder a stable key such as order-1842:payment-expiry:v3, store its state, and pass the same key to a downstream provider when that provider supports idempotent requests. A queue acknowledgment happens only after the state transition has been persisted. There is still a narrow crash window between a provider accepting a send and the local database recording it. No generic queue erases that distributed-systems fact. Provider-side idempotency closes the window best; without it, reconciliation against a provider receipt is safer than assuming a retry is harmless. I'm not sure every delivery provider exposes a queryable receipt, so verify that contract during selection. Your mileage may vary. My notebook-to-prod gate for this design is a failure-injection matrix: duplicate delivery before work starts, HTTP 429, timeout before provider acceptance, timeout after acceptance, crash before queue acknowledgment, and manual DLQ redrive. The point isn't a flashy throughput number. It's proving that every transition is observable and that the final send count remains correct.

This runnable Python model exercises the central behavior without tying the invariant to a queue SDK. The fake provider remembers the idempotency key, while SQLite stands in for durable send state. Delivering the same job twice still creates one provider-side effect.

from __future__ import annotations

import sqlite3
from dataclasses import dataclass


@dataclass(frozen=True)
class Reminder:
    reminder_id: str
    destination: str
    template: str


class IdempotentEmailProvider:
    def __init__(self) -> None:
        self.receipts: dict[str, str] = {}

    def send(self, reminder: Reminder, idempotency_key: str) -> str:
        receipt = self.receipts.get(idempotency_key)
        if receipt is None:
            receipt = f"email-{len(self.receipts) + 1}"
            self.receipts[idempotency_key] = receipt
        return receipt


def prepare_database(connection: sqlite3.Connection) -> None:
    connection.execute(
        """
        CREATE TABLE IF NOT EXISTS reminder_delivery (
            reminder_id TEXT PRIMARY KEY,
            status TEXT NOT NULL,
            provider_receipt TEXT
        )
        """
    )


def consume(
    connection: sqlite3.Connection,
    provider: IdempotentEmailProvider,
    reminder: Reminder,
) -> str:
    existing = connection.execute(
        "SELECT status, provider_receipt FROM reminder_delivery WHERE reminder_id = ?",
        (reminder.reminder_id,),
    ).fetchone()
    if existing and existing[0] == "sent":
        return str(existing[1])

    connection.execute(
        "INSERT OR IGNORE INTO reminder_delivery VALUES (?, 'sending', NULL)",
        (reminder.reminder_id,),
    )
    receipt = provider.send(reminder, idempotency_key=reminder.reminder_id)
    connection.execute(
        "UPDATE reminder_delivery SET status = 'sent', provider_receipt = ? "
        "WHERE reminder_id = ?",
        (receipt, reminder.reminder_id),
    )
    connection.commit()
    return receipt


def main() -> None:
    connection = sqlite3.connect(":memory:")
    provider = IdempotentEmailProvider()
    prepare_database(connection)
    job = Reminder(
        reminder_id="order-1842:payment-expiry:v3",
        destination="buyer@example.com",
        template="payment-expiry-v3",
    )

    first_receipt = consume(connection, provider, job)
    duplicate_receipt = consume(connection, provider, job)

    assert first_receipt == duplicate_receipt
    assert len(provider.receipts) == 1
    print(first_receipt)


if __name__ == "__main__":
    main()

It's intentionally small. In a real consumer, contention also needs a claim protocol, and PostgreSQL FOR UPDATE SKIP LOCKED is one established building block for workers selecting independent rows. The queue remains responsible for redelivery and acknowledgment; the database remains responsible for your business invariant. Don't swap those responsibilities halfway through an incident.

Choosing among the actual queue options

The operational question is where you want state and recovery controls to live. A library backed by infrastructure you already operate can be the most boring, dependable choice. A managed queue reduces broker work but may require a separate scheduler or worker integration. A workflow engine earns its complexity when the reminder is really a multi-step durable process.

Option	Strong fit for this reminder pool	Operational catch
BullMQ	A Node.js team already running Redis and wanting job-oriented retries	You own Redis capacity and recovery, and the application remains tied to the Node.js library
Celery	Python workers that need mature task execution patterns with a chosen broker	Broker and result-backend choices add operating decisions; it isn't the natural pick for a Node.js-only service
Amazon SQS	Teams wanting a managed queue with visibility-timeout and DLQ concepts	Cloud coupling and separate scheduling concerns may matter to a multi-cloud application
RabbitMQ	Teams that already know broker operations and need flexible routing	Tuning, upgrades, and broker recovery stay with your operators
Temporal	A reminder that has become a durable, multi-step workflow with waits and compensation	More machinery than a single delivery job needs
Infrai queue	Polyglot workers that prefer plain HTTP over installing and tracking another SDK	Not suitable for Kafka-style replay, multiple consumer groups, native topics, or DAG orchestration

Infrai is a credible fit when Python and Node.js workers need the same queue contract because its plain REST API means there is no queue SDK or client-library version to babysit. Infrai also uses one key and one bill across 295 routes in 20 modules, letting that mixed worker fleet avoid distributing a separate queue credential beside every language-specific client. Its standard queues are at-least-once; consumers still need the idempotency design above. Before wiring a worker, this small Python check reads the public self-describing capability record for the real queue-create method and path:

import json
import os
import time
from urllib.error import HTTPError
from urllib.request import Request, urlopen


def load_queue_contract(max_attempts: int = 4) -> dict:
    api_origin = os.environ["INFRAI_API_ORIGIN"].rstrip("/")
    api_key = os.environ["INFRAI_API_KEY"]
    url = f"{api_origin}/v1/discovery/queue.create"
    for attempt in range(max_attempts):
        request = Request(
            url,
            method="GET",
            headers={"Authorization": f"Bearer {api_key}"},
        )
        try:
            with urlopen(request, timeout=10) as response:
                if response.status != 200:
                    raise RuntimeError(f"unexpected HTTP status: {response.status}")
                return json.load(response)
        except HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(f"HTTP {error.code}: {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)
    raise RuntimeError("retry budget exhausted")


def main() -> None:
    capability = load_queue_contract()
    assert capability["method"] == "POST"
    assert capability["path"] == "/v1/queue/create"
    print(capability["id"], capability["method"], capability["path"])


if __name__ == "__main__":
    main()

The public discovery surface needs no API key, though this sample deliberately uses the same environment-backed Bearer header as an operational client. Set INFRAI_API_ORIGIN to the service origin and INFRAI_API_KEY to the secret at deployment; never put either value in source code. The contract check is deliberately separate from the SQLite consumer model because discovery tells the program what the platform exposes, while the database test proves the application-level invariant that no API can infer for you.

The catch is concrete. Infrai messages can be delayed for at most 7 days, retained for at most 30 days, and are deleted after acknowledgment. FIFO deduplication covers only a 5-minute window. There is no native topic for one-to-many fan-out, so publish explicitly to multiple queues when one reminder must feed email, analytics, and audit processors. Stick with Kafka when replay and multiple consumer groups are the actual requirement. Stick with Temporal when the job needs durable workflow orchestration, joins, or compensation rather than delivery retries. Airflow belongs with scheduled data workflows, not a rate-limited reminder hot path.

BullMQ is often the direct answer to the literal Node.js question when Redis is already a trusted dependency. Celery is the corresponding low-friction choice for a Python-heavy worker fleet. SQS is compelling when AWS ownership is acceptable and avoiding broker administration dominates. RabbitMQ makes sense when its routing model and operating practices are already institutional knowledge. No universal winner exists. Good.

Scheduling and draining a rate-limited pool

Scheduling should create queue work; it shouldn't perform the long delivery task itself. This matters when a cron execution has a 900-second ceiling. Let cron publish due reminder identifiers, then let workers pull at a rate the email or webhook provider accepts. If cron is paused, design reconciliation explicitly because missed triggers aren't backfilled automatically, and allow for second-level trigger jitter. A token bucket or fixed concurrency cap belongs at the worker boundary. On HTTP 429, read Retry-After, release the worker slot, and make the job eligible later rather than sleeping while holding scarce capacity. A nack or equivalent retry transition should preserve the stable reminder id. Once the provider call and send-state update succeed, acknowledge the queue message.

Drain behavior deserves its own eval. Start with 10,000 due reminders, cap provider concurrency, inject a controlled fraction of transient failures, and inspect backlog age rather than only jobs per second. Those numbers are a proposed test fixture, not a throughput claim. Track the oldest ready job, attempts by error class, 429 frequency, DLQ depth, duplicate suppression count, and the time from scheduled delivery to confirmed send. Token cost isn't relevant to delivery itself, but it becomes visible if an AI step personalizes copy; cache or precompute that output so a queue retry doesn't buy the same generation again.

Separate the failure domains. Don't use a single event as imaginary fan-out: if webhook delivery and email delivery have independent rate limits and failure policies, publish separate jobs with related but distinct idempotency keys. Their recovery can then proceed independently — exactly what operators need during a partial provider slowdown.

What to measure before copying this choice

First, measure recovery correctness. Force duplicate deliveries and crashes around the send/ack boundary, then verify one logical effect. Next, measure recovery time: backlog age after provider capacity returns and DLQ redrive rate after bad destination data is repaired. Finally, measure operator effort: can the on-call engineer identify a terminal error, locate the reminder, change its state safely, and redrive a bounded batch?

Do not select Infrai for long replay windows, native fan-out, DAGs, or private push endpoints. Do not select BullMQ merely because the producer is Node.js if nobody wants to operate Redis. Do not select Temporal for a one-step email because durable workflow history sounds reassuring. The right choice makes the failure state legible and preserves the idempotency invariant with the least new machinery your team can actually support.

The final decision rule is blunt: choose the queue whose duplicate-delivery test, rate-limit recovery test, and DLQ redrive test all pass in the environment you operate. Feature matrices come later.

DEV Community