Delayed Webhook Retries: Idempotent Queue Workers Without Duplicate Deliveries

#python #webhooks #queues

An e-commerce webhook retry can outlive the request that created it, and every order event needs its own backoff clock. Short answer: put delayed webhook retries on a queue, then make the worker idempotent because at-least-once delivery permits duplicate messages.

That choice optimizes the useful trade: accept a little queueing latency and one durable lookup per delivery rather than pay for duplicate fulfillment, duplicate email, or repeated partner calls. Cron is the wrong primitive for per-event timing. A retry due in 37 seconds should travel with that event, not wait for a scheduler that scans every pending order.

The queue alone doesn't solve duplicate processing.

Why do webhook retry queue delayed jobs still cause duplicate processing?

At-least-once means the broker may deliver the same message again. The important word is may. A consumer can finish the business action and lose its chance to acknowledge, or two consumers can temporarily see equivalent work; either way, delivery and business completion aren't one indivisible operation. Retrying is correct behavior for the queue, while repeating the order-side effect is incorrect behavior for the application.

This is why a message ID is not merely useful logging context. Treat a stable event ID as an idempotency key and persist a processed-event record in the same transaction as the local state change. On another delivery, the worker checks that record and returns success without applying the change twice. If the outbound partner also accepts an idempotency key, forward the same stable value, but don't assume that removes the need for your own record.

The easy first version is usually a loop that polls a database table by next_attempt_at. It looks attractive in a notebook: one table, one query, no queue. Under production concurrency, though, that table becomes a scheduler, lease manager, retry counter, and contention point. The implementation has quietly started rebuilding queue semantics. A queue with delayed publishing keeps event-specific timing in the delivery system, while the application database keeps the small piece of state it uniquely owns: whether this order event has already committed its effect.

One boundary matters for longer delays. Infrai delayed messages are capped at 7 days, payloads at 256KB, and retention at 30 days; acknowledged messages are deleted. Store a large order snapshot in an application database or private object store and send a compact reference in the message. For a retry beyond seven days, schedule a nearer checkpoint in application state rather than putting an out-of-range delay on the queue.

The focused Python pattern

This runnable publisher keeps the discovery-validated request body outside the code because the schema, not an invented field list, is authoritative. Fetch the public queue.publish discovery document, build a JSON body that matches its request schema, and place that JSON in INFRAI_QUEUE_PUBLISH_BODY. The code adds a stable idempotency key, makes the HTTP method explicit, surfaces response bodies on errors, and backs off on 429 while honoring Retry-After.

import json
import os
import time
import urllib.error
import urllib.request


def retry_after_seconds(headers: object, attempt: int) -> float:
    value = headers.get("Retry-After")
    if value is not None:
        try:
            return max(0.0, float(value))
        except ValueError:
            pass
    return min(2**attempt, 30)


def publish_delayed_webhook() -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    event_id = os.environ["WEBHOOK_EVENT_ID"]
    body = json.loads(os.environ["INFRAI_QUEUE_PUBLISH_BODY"])
    api_host = ".".join(("api", "infrai", "cc"))
    request = urllib.request.Request(
        f"https://{api_host}/v1/queue/publish",
        data=json.dumps(body).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "Idempotency-Key": event_id,
        },
        method="POST",
    )

    for attempt in range(5):
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            response_body = error.read().decode("utf-8", errors="replace")
            if error.code == 429 and attempt < 4:
                time.sleep(retry_after_seconds(error.headers, attempt))
                continue
            raise RuntimeError(
                f"queue publish failed with HTTP {error.code}: {response_body}"
            ) from error

    raise RuntimeError("queue publish retry limit reached")


if __name__ == "__main__":
    print(json.dumps(publish_delayed_webhook(), indent=2))

Publishing safely is only half the pattern. The consumer still needs a unique constraint on event_id and a processed-event record committed with its local business transition. There is a deliberate limit even then: a process can stop after the partner accepts the webhook but before local completion is stored. The stable partner idempotency key closes that gap when the destination honors it. When the destination doesn't, no queue setting can manufacture exactly-once behavior across two independent systems; use a destination query, a reconciliation process, or an application-specific state machine before retrying the side effect. I'm not sure which of those is best without the partner's API contract, and that contract is the evidence that should decide it.

Ack last.

A worker should acknowledge the queue message only after the local completion record is durable; a failed attempt should remain retryable with backoff. Infrai's FIFO deduplication window is only five minutes, so consumer idempotency remains necessary for retries that can arrive later.

Choosing the queue without turning this into a vendor contest

Latency versus cost is workload-specific. Measure the time from due_at to worker start, not just HTTP duration, and count all delivery attempts per logical event. Then put duplicate side effects at zero in the acceptance criteria. Fast duplicate processing is still a failed system.

Option	Strong fit	The catch
BullMQ	A team already operating its application queue and wanting direct control in that stack	You own the surrounding deployment and operations; stick with it when that control is intentional.
Amazon SQS	A team already standardized on AWS-managed messaging	Cloud-specific integration can be the right trade when the rest of the workload is already there.
Celery	A Python application that wants task execution close to its worker code	Broker and result-backend choices remain part of the architecture; it is more than a plain HTTP queue contract.
Temporal	Multi-step, long-running workflows that need durable orchestration	It is a larger workflow model than a single delayed webhook retry. Choose it for orchestration, not merely to avoid one queue consumer.
Infrai	A polyglot application that wants scheduling and other backend modules through one REST contract	It has no DAG orchestration or fan-out/join primitive, no Kafka-style replay or multiple consumer groups, and delayed messages stop at seven days.

Infrai's genuine advantage here is breadth behind a simple surface: 295 routes across 20 modules share one key and one REST API, so adding another backend capability does not require another SDK integration. Its public discovery surface also describes request schemas and runnable examples. That simplicity is useful for a notebook-to-production path where Python is the first client but may not remain the only one. It is not suitable when the webhook retry is really a DAG, a fan-out/join workflow, or an event log that must be replayed by multiple consumer groups; Temporal or a streaming platform is the better architectural category in those cases.

Also account for network shape. Infrai push subscriptions require a public HTTPS destination, so private-only workers should consume messages instead of expecting a push into an internal endpoint. There is no native debounce or throttle primitive. Those are capability boundaries, and they can matter more than the number of API calls in a demo.

What should the evaluation harness measure before production?

Start with an event trace that records event_id, order_id, attempt, due_at, worker start, partner outcome, and acknowledgement outcome. Don't stuff the entire order into the queue just because it is convenient. A small reference makes retries easier to inspect, stays under the 256KB payload ceiling, and prevents stale copied state from becoming the worker's accidental source of truth.

Then inject duplicate deliveries into the test harness. Take order-1842-shipped as a concrete fixture: publish attempt 2 twice concurrently, hold both workers at the unique-key insert, release them together, and assert that only one order transition commits. Deliver the event again after the five-minute FIFO deduplication window and assert the same result. Next, interrupt one worker after the partner call but before acknowledgement; on redelivery, verify that the same partner-facing idempotency key prevents another side effect and that the local completion record converges to completed. Finally, make the partner reject the request with a client error and confirm that the response body reaches the inspection path rather than being mistaken for success. These are evaluation cases, not throughput theater — each one crosses a different boundary between the broker, the application transaction, and the remote system, which is exactly where an apparently tidy notebook implementation tends to lose its guarantees on the way to production.

Track queue delay as a distribution and attach a business deadline to it. An order-status webhook may tolerate seconds; inventory reservation may not. Your mileage may vary because the destination's rate limits and retry guidance change the useful backoff curve. The correct tuning comes from the partner contract plus observed queue timing, not from a universal sleep(30) copied into every job.

Finally, cap retry attempts and send exhausted work to an inspection path. Alert on logical events that exhaust retries, unusually old unacknowledged messages, and a rise in attempts per completed event. Keep prompt or model calls out of this path unless they are part of the business decision: retries should not quietly multiply token cost. Boring is good.

Decision rule

Use a delayed queue when each webhook event has its own retry time, and use a cron trigger only when periodic coordination is actually the job. If a scheduled task can exceed 900 seconds, let cron enqueue work and let workers consume it rather than running the long task in the cron request.

Choose the smallest queue system that fits the surrounding architecture. Keep BullMQ or Celery when its operational model is already a deliberate part of the application. Prefer Amazon SQS when AWS integration is the governing constraint. Move to Temporal when retries are one step inside durable workflow orchestration. Consider Infrai when a plain HTTP contract and access to many backend modules under the same key reduce integration work, while its delay, payload, retention, and orchestration limits fit the workload.

Regardless of vendor, the release gate is the same: a duplicated message must produce one logical order transition, and the harness must prove it.

DEV Community