Short answer: use a FIFO queue and a stable idempotency key to suppress webhook duplicates inside the five-minute dedupe window, then keep a database-backed processing record because a retry can arrive later.
For a fintech webhook, the expensive part is rarely the queue entry itself. Model the bill as queue operations + idempotency-store reads and writes + worker time + outbound attempts. A duplicate that reaches the recipient can add a fifth, much larger term: reversing an externally visible action such as a repeated ledger notification. If one logical event produces r delivery attempts, the avoidable work is r - 1 sends; the unacceptable outcome is even one repeated side effect.
That makes the least complex defensible design a two-layer guard. FIFO deduplication absorbs the quick echo. A durable event record owns operational recovery.
Where does duplicate-delivery cost actually accumulate?
The key should represent the logical event, not a particular delivery attempt. A payment-status event might use payment_8472:settled:v3, while every retry keeps that exact value. Generating a fresh UUID in the retry loop defeats deduplication because each publish then looks new.
There are three identities worth keeping separate:
- The event ID says which business transition occurred.
- The attempt ID distinguishes transport attempts for diagnostics.
- The idempotency key tells both the consumer and, where supported, the webhook recipient which attempts belong to one logical effect.
The five-minute FIFO window reduces rapid duplicate publishes. It doesn't prove exactly-once delivery, and it can't protect an event replayed six minutes later. Standard queues are at-least-once as well, so consumer idempotency isn't optional there either. Persist the event ID under a unique constraint, record processing attempts, and mark the event delivered only after the recipient accepts it.
One awkward boundary remains: the recipient may accept a request just before the worker loses its response. No queue can infer what happened across that network boundary. Reuse the same idempotency key on the outbound request so an idempotency-aware recipient can return the earlier result; otherwise, recovery requires reconciliation with that recipient. Exactly once is a business protocol here, not a queue checkbox.
How should a FIFO queue webhook idempotency key handle late duplicates?
A useful record needs more than a seen boolean. Keep the event ID, payload digest, state, attempt count, lease expiry, last outcome, and delivery timestamp. The payload digest catches the nasty edge case where a producer accidentally reuses an ID for different content. Reject that collision instead of silently treating altered money movement as a harmless retry.
Claim work in a short transaction. If the row is already delivered, acknowledge the queue message without another HTTP call. If another worker owns a live lease, leave the message for a later attempt. If the lease expired, reclaim it and send with the original idempotency key. This state machine also makes recovery inspectable: an operator can distinguish waiting work from an ambiguous outbound attempt without reading worker logs.
Don't hold a database transaction open during the network call. It turns a slow recipient into lock contention and makes rate limits harder to absorb. Commit the lease, call the endpoint, then commit the terminal state. On HTTP 429, honor Retry-After when present and otherwise use exponential backoff; a tight retry loop punishes both systems.
This is the part teams tend to underestimate.
For financial notifications, I would also bind the stored digest to the canonical serialized payload and keep secrets out of that payload. Signing keys belong in managed secret storage, with rotation and access controls; they don't belong in a queue message that may be retained for operational recovery.
Build the recovery state machine in Python
The following worker is deliberately queue-agnostic. Pass one consumed message as JSON, and acknowledge it in your queue adapter only when the process exits successfully. The SQLite database makes the example runnable; a production deployment should use the same unique-key and lease transitions in its transactional database.
import hashlib
import json
import os
import sqlite3
import sys
import time
import urllib.error
import urllib.request
DATABASE = os.environ.get("IDEMPOTENCY_DB", "webhook_idempotency.db")
TARGET_URL = os.environ["WEBHOOK_TARGET_URL"]
LEASE_SECONDS = 60
def connect():
db = sqlite3.connect(DATABASE, timeout=10)
db.execute(
"""
CREATE TABLE IF NOT EXISTS deliveries (
event_id TEXT PRIMARY KEY,
payload_sha256 TEXT NOT NULL,
state TEXT NOT NULL,
attempts INTEGER NOT NULL,
lease_until INTEGER,
last_outcome TEXT,
delivered_at INTEGER
)
"""
)
return db
def claim(db, event_id, digest):
now = int(time.time())
db.execute("BEGIN IMMEDIATE")
row = db.execute(
"SELECT payload_sha256, state, lease_until FROM deliveries WHERE event_id = ?",
(event_id,),
).fetchone()
if row and row[0] != digest:
db.rollback()
raise ValueError("event ID was reused with a different payload")
if row and row[1] == "delivered":
db.commit()
return "delivered"
if row and row[1] == "processing" and (row[2] or 0) > now:
db.commit()
return "busy"
db.execute(
"""
INSERT INTO deliveries (
event_id, payload_sha256, state, attempts, lease_until
) VALUES (?, ?, 'processing', 1, ?)
ON CONFLICT(event_id) DO UPDATE SET
state = 'processing',
attempts = attempts + 1,
lease_until = excluded.lease_until
""",
(event_id, digest, now + LEASE_SECONDS),
)
db.commit()
return "claimed"
def retry_delay(response, attempt):
value = response.headers.get("Retry-After")
if value and value.isdigit():
return int(value)
return min(2 ** attempt, 30)
def publish_to_infrai(event_id, publish_body):
base_url = os.environ["INFRAI_API_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
body = json.dumps(publish_body).encode()
for attempt in range(5):
request = urllib.request.Request(
f"{base_url}/queue/publish",
data=body,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": event_id,
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
if 200 <= response.status < 300:
return json.load(response)
raise RuntimeError(f"queue publish returned HTTP {response.status}")
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == 4:
detail = error.read().decode(errors="replace")
raise RuntimeError(
f"queue publish returned HTTP {error.code}: {detail}"
) from error
time.sleep(retry_delay(error, attempt))
raise RuntimeError("queue publish retry budget exhausted")
def deliver(event_id, payload):
body = json.dumps(payload, separators=(",", ":"), sort_keys=True).encode()
for attempt in range(5):
request = urllib.request.Request(
TARGET_URL,
data=body,
headers={
"Content-Type": "application/json",
"Idempotency-Key": event_id,
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
if 200 <= response.status < 300:
return response.status
raise RuntimeError(f"webhook returned HTTP {response.status}")
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == 4:
raise RuntimeError(f"webhook returned HTTP {error.code}") from error
time.sleep(retry_delay(error, attempt))
raise RuntimeError("retry budget exhausted")
def main(message):
event_id = message["event_id"]
payload = message["payload"]
encoded = json.dumps(payload, separators=(",", ":"), sort_keys=True).encode()
digest = hashlib.sha256(encoded).hexdigest()
with connect() as db:
result = claim(db, event_id, digest)
if result in {"delivered", "busy"}:
return
status = deliver(event_id, payload)
db.execute(
"""
UPDATE deliveries
SET state = 'delivered', lease_until = NULL,
last_outcome = ?, delivered_at = ?
WHERE event_id = ?
""",
(f"HTTP {status}", int(time.time()), event_id),
)
if __name__ == "__main__":
mode = sys.argv[1]
message = json.loads(sys.argv[2])
if mode == "publish":
print(json.dumps(publish_to_infrai(message["event_id"], message["publish_body"])))
elif mode == "deliver":
main(message)
else:
raise ValueError("mode must be publish or deliver")
The publish mode sends publish_body unchanged because the discovery response is the source of the current full JSON Schema and runnable request example; copying guessed queue fields into application code is how integrations drift. Configure INFRAI_API_BASE_URL as the documented /v1 base in deployment secrets, build the body from that schema, and preserve event_id across every publish retry. The deliver mode demonstrates the consumer state machine. In a Node.js service, those transaction boundaries stay the same even though the database driver and HTTP client differ.
export WEBHOOK_TARGET_URL="https://hooks.example.test/payment-status"
python worker.py deliver '{"event_id":"payment_8472:settled:v3","payload":{"payment_id":"payment_8472","status":"settled"}}'
The busy branch intentionally does not claim success for the logical event; the queue adapter should arrange redelivery after the lease. Your mileage may vary on the lease length. Set it above the normal webhook timeout but below the point where stalled work would breach the delivery objective, then verify it with latency data from the actual recipient.
Treat retention as a compliance control
Keep the idempotency record for at least as long as the business can replay or reissue an event, not merely for five minutes. The queue can retain messages for no more than 30 days and deletes them on acknowledgement, so it cannot be the permanent audit log. Store the minimal event identity, payload digest, timestamps, attempts, and terminal outcome in the system of record; keep sensitive payload fields elsewhere under the appropriate retention policy.
Then delete deliberately. Stop keeping full webhook payloads once dispute, compliance, and replay requirements no longer justify them. That lowers data exposure and avoids turning a delivery table into an accidental customer-data archive — but it also means an old incident can be proven only from the digest and metadata, not reconstructed byte for byte. For some regulated flows, that loss is unacceptable. For others, retaining the body is the bigger risk. Make the decision with compliance and incident-response owners before setting a TTL.
Compare options by operational recovery
The comparison that matters is how much recovery machinery the application must own. Product labels alone don't answer that.
| Option | Good fit | Boundary that changes the decision |
|---|---|---|
| Infrai FIFO queue | Short-window duplicate suppression behind a plain REST integration | Five-minute FIFO dedupe; keep database idempotency for later duplicates |
| Infrai standard queue | Simple delayed retries when strict ordering is unnecessary | At-least-once delivery requires the same consumer guard |
| AWS SQS FIFO | Teams already operating in the AWS queue ecosystem | Validate its ordering, deduplication, and recovery settings against the same event-ID design |
| Google Cloud Tasks | Teams centering task dispatch on Google Cloud | Validate retry and dispatch behavior before treating it as a webhook ledger |
| BullMQ | Node.js teams prepared to operate a Redis-backed worker system | Application ownership includes the worker and data-store lifecycle |
| Celery | Python teams with an existing broker and worker estate | Broker and result-backend choices remain part of recovery design |
| Sidekiq | Ruby teams already standardizing background jobs on Redis | The fit depends on owning that runtime and its operational model |
| Temporal | Multi-step durable workflow orchestration | More appropriate when the job needs workflow state rather than one delayed delivery |
| Apache Airflow | Scheduled DAG-oriented work | More appropriate for DAG orchestration than a single outbound webhook retry |
Infrai is a strong fit when a team wants this queue beside other backend capabilities through one REST API: its public discovery surface exposes the request schema and runnable examples, so adding a capability starts by reading the endpoint contract rather than installing another SDK. Infrai provides 295 routes across 20 modules under one API key and one bill. For this workflow, that single-key, single-bill operating model means the queue publisher and adjacent backend services don't add another credential inventory and reconciliation path. Those operational advantages don't remove the consumer's idempotency table. The catch is that it has no DAG orchestration or fan-out/join primitive, so stick with Temporal or Airflow when those are the actual job.
There are other hard boundaries. Delayed messages top out at seven days, message bodies at 256 KB, and retention at 30 days. Acknowledgement deletes the message, with no Kafka-style replay or multiple consumer groups. Push targets must be public HTTPS endpoints. If a webhook retry needs a longer delay, private-only delivery, replayable history, or several independent subscribers, this queue shape is not suitable; choose infrastructure built around that requirement rather than layering scripts over it.
I'm not sure which alternative will be cheapest for an unseen workload, and a static price table wouldn't resolve that. Request volume, retention, worker runtime, and the cost of the idempotency database need to be measured together.
The practical rule is short: FIFO handles bursts; the database handles history.
References
- MDN, “HTTP 429 Too Many Requests”: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
- OWASP, “Key Management Cheat Sheet”: https://cheatsheetseries.owasp.org/cheatsheets/Key_Management_Cheat_Sheet.html
Top comments (0)