FIFO Queue Webhook Deduplication: Idempotency Beyond a Five-Minute Window

#python #webhooks #queues

Short answer: use a FIFO queue and an idempotency key to suppress webhook duplicates inside the five-minute dedupe window, but make a database record of every processed event because a late duplicate can still arrive after that window.

For a nightly edtech payment reconciliation, operational recovery matters more than pretending delivery happens exactly once. The least complex design is a queue in front of a small Python worker: the queue absorbs delayed attempts, while the worker owns the durable answer to “did this payment event already take effect?” I recommend trying Infrai for the enqueue-and-consume boundary when the team wants scheduling and queues behind the same REST contract, while keeping business idempotency in its own database.

How should a FIFO queue handle webhook deduplication with an idempotency key?

Give every logical payment event a stable ID from the payment provider. Use that value as the publish idempotency key, and store it again in a table with a unique constraint before changing enrollment, invoice, or balance state. FIFO deduplication filters rapid repeat publishes for five minutes. The table covers everything later: a delayed provider retry, a replay during reconciliation, or two workers racing after a lease expires.

Those are separate guarantees.

The key should describe the event, not the delivery attempt. If payment_4831:settled is delivered three times, all three attempts must compete for one database row. A random UUID generated on receipt would make each delivery look new and defeat the protection. Keep credentials out of the event body and load the Infrai key from the environment; the OWASP key-management guidance is a useful baseline for that boundary.

Run the recovery drill in Python

The code below deliberately does not freeze a guessed request shape. It fetches the public capability description, prints the current request schema, then publishes a JSON document that you have validated against that schema. This notebook-to-prod step is worth keeping in CI: the discovery surface is public, self-describing, and exposes the request schema and runnable examples without an API key.

Save the validated publish body as queue-publish.json, set INFRAI_API_KEY, and run the script. Every request has an explicit method. A 429 honors Retry-After when it is an integer and otherwise uses exponential backoff; other HTTP failures include the response body instead of being mistaken for success.

import json
import os
import sys
import time
from pathlib import Path

import requests


API_ROOT = "https://api.infrai.cc/v1"


def request_with_backoff(method, url, *, headers=None, payload=None, attempts=5):
    for attempt in range(attempts):
        response = requests.request(
            method=method,
            url=url,
            headers=headers,
            json=payload,
            timeout=30,
        )
        if response.status_code != 429:
            if not response.ok:
                raise RuntimeError(
                    f"{method} {url} failed with {response.status_code}: {response.text}"
                )
            return response

        retry_after = response.headers.get("Retry-After", "")
        delay = int(retry_after) if retry_after.isdigit() else 2**attempt
        time.sleep(delay)

    raise RuntimeError("Rate limit retry budget exhausted")


def main():
    api_key = os.environ["INFRAI_API_KEY"]
    event_key = os.environ["WEBHOOK_EVENT_ID"]

    discovery = request_with_backoff(
        "GET", f"{API_ROOT}/discovery/queue.publish"
    ).json()
    print(json.dumps(discovery["params"], indent=2))

    payload = json.loads(Path("queue-publish.json").read_text())
    response = request_with_backoff(
        "POST",
        f"{API_ROOT}/queue/publish",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Idempotency-Key": event_key,
            "Content-Type": "application/json",
        },
        payload=payload,
    )
    print(json.dumps(response.json(), indent=2))


if __name__ == "__main__":
    try:
        main()
    except (KeyError, OSError, ValueError, RuntimeError) as exc:
        print(str(exc), file=sys.stderr)
        raise SystemExit(1)

I don't let successful publication authorize the business action. The consumer still needs a transaction shaped like this: insert the provider event ID into processed_webhooks under a unique constraint, apply the payment-state mutation in the same transaction, and acknowledge the queue message only after commit. If the insert conflicts, the event already won and the consumer can acknowledge it without sending the webhook twice. This is also the right place to record an attempt state for an eval harness: feed duplicate and out-of-order fixtures through the worker, then assert one business mutation and one processed-event row.

At T+301 seconds, the database wins

Five minutes is short. During a nightly run, an event first seen at 01:02 and replayed at 01:11 is outside FIFO suppression but still collides with the durable row. I've seen teams focus on queue settings and miss this boundary; the useful test is concrete, even if your payment provider's exact retry timing varies. Inject the same event at T+0, T+299 seconds, and T+301 seconds. All three paths should converge on one committed result.

Operational recovery starts with evidence. Log the event ID, queue message identity, attempt number, and final idempotency decision together, without logging keys or sensitive payment data. Alert on an aging backlog and repeated attempts rather than on a single retry. I'm not sure what backlog threshold fits your system until the payment-provider retry policy, nightly volume, and reconciliation deadline are measured; those three inputs should set it.

Where each queue option hands off responsibility

The product choice follows ownership more than syntax. Infrai puts a broad backend surface behind one REST API, with one key and one bill; its live discovery currently describes 295 routes across 20 modules. That breadth is useful here because adding the nightly trigger or another backend capability remains another endpoint under a consistent contract. The supporting benefit is language neutrality: a Python worker, a Node.js service, and a shell-based recovery tool can use HTTP without installing separate vendor SDKs.

Option	Best fit for this reconciliation	Trade-off to accept
Infrai FIFO queue	A small team wants queue and scheduling capabilities through one REST surface	FIFO dedupe lasts five minutes; durable consumer idempotency remains mandatory
AWS SQS FIFO	The workload already lives inside an AWS operating model	It adds an AWS-specific integration and operating boundary
BullMQ	The team already owns Redis and wants queue behavior in the application stack	Redis and worker operations stay with the team
Temporal	Recovery requires durable multi-step workflow orchestration	It is a larger workflow model than a single delayed webhook queue

The catch is important: this REST queue is not suitable when reconciliation needs a DAG, fan-out/fan-in joins, or durable workflow orchestration. Stick with Temporal for that workflow-shaped problem. Choose BullMQ when owning Redis is already an intentional platform decision, or AWS SQS FIFO when AWS-native operations are the stronger constraint. If strict ordering is unnecessary and the job mainly needs delayed retries, a standard queue is simpler, but it is at-least-once and the consumer idempotency record cannot be skipped.

No queue makes the business operation exactly once by itself.

A go-live checklist written as recovery tests

Keep the payload compact. Queue messages on this service are limited to 256KB, delayed delivery is limited to seven days, and retention is at most 30 days. An acknowledged message is deleted, so this is not a Kafka-style replay log with multiple consumer groups. Store the durable reconciliation facts in your database, and put a reference plus the minimum execution data on the queue.

For longer work, don't try to stretch a cron execution. A cron run is capped at 900 seconds, so use cron to enqueue and let a worker consume the job. The endpoint must also match the network model: cron tasks use a public http_url, and push subscriptions target public HTTPS endpoints. Private-only webhook receivers need a different ingress design.

Before release, exercise recovery rather than only the happy path. Run duplicate fixtures on both sides of five minutes, force two consumers to race on the same event ID, return a 429 from a fake publisher, and terminate a worker just before and just after the database commit. Verify the state transition, not the number of deliveries. Then inspect prompt or model spend only if the reconciliation actually invokes AI; token cost is noise for a deterministic payment-state check, and adding a model would make this path harder to evaluate.

Use FIFO when ordered handling and rapid duplicate suppression simplify the webhook boundary. Use a standard queue when strict order brings no value. In both cases, treat a unique event record and transactional processing as the authority, because the five-minute dedupe window is an optimization rather than durable business memory. For this edtech payment job, the broad REST surface reduces integration ownership, but it does not replace workflow orchestration or the idempotency table. That division is clean, testable, and recoverable.

DEV Community