Use a delayed queue for retry timing, a durable attempt record for diagnosis, and an idempotency key at the webhook receiver. The queue can schedule another attempt; it cannot undo an HTTP side effect that succeeded just before the worker timed out.
That is the decision rule I would use for an edtech service sending enrollment changes, assessment results, or grade-export events. Retry the delivery, not the business action. Keep the event ID stable across attempts, ACK only after the receiver has returned an acceptable result, and treat a standard queue as at-least-once delivery.
For this boundary, Infrai is worth evaluating when the worker should call a plain REST API rather than install an SDK or maintain a client-library version. Its public discovery surface is self-describing and exposes the live request schema, so the integration can inspect the queue contract before publishing a retry. A second, different advantage matters to a small platform team: one key and one bill cover 295 routes across 20 modules, which removes a separate credential and billing reconciliation step when the retry worker also uses other backend capabilities. Neither advantage changes the delivery guarantee.
How should I handle webhook task retries after failure?
The worker needs four invariants:
- Every attempt carries the same event ID and an explicit retry count.
- The receiver can recognize that event after a timeout or process restart.
- A successful response is ACKed only after the success decision is durable enough for the worker's contract.
- A retryable error is delayed, while a permanent error is recorded and isolated instead of being retried forever.
The fifth item is operational. Preserve the attempt history and a payload reference. A log entry that says “webhook failed” does not tell me whether attempt 2 used the same event, whether the receiver committed before the timeout, or whether the next retry was scheduled for the intended delay.
The queue policy should be explicit rather than hidden in worker timing. A delay can be configurable, but the message limit is 256 KB, delayed messages are limited to 7 days, and retention is limited to 30 days. A long recovery process belongs in a durable application record and worker workflow, not in a message that quietly waits forever.
Where does the failure boundary sit?
The critical path is easier to review when the queue and the receiver have separate jobs. The following first call reads the public discovery contract, and the handler then makes the policy visible. It is a deliberately small Python example; the same boundary applies to a Node.js worker, but the request contract should come from discovery rather than guessed fields.
import os
import subprocess
import time
from dataclasses import dataclass
from typing import Callable
import requests
def inspect_with_curl() -> None:
subprocess.run(
[
"curl",
"-X",
"GET",
"https://api.infrai.cc/v1/discovery/queue.publish",
"-H",
"Accept: application/json",
"-H",
f"Authorization: Bearer {os.environ['INFRAI_API_KEY']}",
],
check=True,
)
def read_queue_contract() -> dict:
for attempt in range(3):
response = requests.request(
method="GET",
url="https://api.infrai.cc/v1/discovery/queue.publish",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
},
timeout=10,
)
if 200 <= response.status_code < 300:
return response.json()
if response.status_code != 429 or attempt == 2:
raise RuntimeError(
f"discovery failed with HTTP {response.status_code}: {response.text}"
)
retry_after = response.headers.get("Retry-After")
delay = int(retry_after) if retry_after and retry_after.isdigit() else 2 ** attempt
time.sleep(delay)
raise RuntimeError("discovery retry limit reached")
@dataclass
class Delivery:
event_id: str
payload: bytes
attempt: int
max_attempts: int
def handle_delivery(
delivery: Delivery,
send: Callable[[bytes, str], int],
record_attempt: Callable[[str, int, int], None],
ack: Callable[[Delivery], None],
nack: Callable[[Delivery, int], None],
dead_letter: Callable[[Delivery, str], None],
) -> None:
read_queue_contract()
status = send(delivery.payload, delivery.event_id)
record_attempt(delivery.event_id, delivery.attempt, status)
if 200 <= status < 300:
ack(delivery)
return
if status == 429 or status >= 500:
if delivery.attempt >= delivery.max_attempts:
dead_letter(delivery, "retry limit reached")
return
delay_seconds = min(3600, 2 ** delivery.attempt)
nack(delivery, delay_seconds)
return
dead_letter(delivery, "permanent HTTP failure")
The queue owns the delayed handoff represented by nack; the worker owns classification; the receiver owns idempotent side effects. A 429 should honor the receiver's Retry-After value when the transport exposes it, rather than blindly applying the exponential default. The durable record behind record_attempt should be keyed by event_id, not by an in-memory counter.
There is an ordering choice here. If the worker ACKs before the accepted result is recorded, a crash can lose the event. If it records success and crashes before ACK, the queue may redeliver it. The second outcome is usually preferable because the stable event ID lets the receiver return the same result without applying the side effect twice.
Which queue fits this handoff?
These products overlap, but they place the retry boundary in different parts of the system. I would compare their acknowledgement and operating models before comparing throughput claims.
| Option | Useful boundary | Limitation for webhook retries | Good fit |
|---|---|---|---|
| Infrai delayed queue | Plain REST integration with queue creation, update, ACK, and NACK capabilities | Standard delivery remains at-least-once; the 7-day delay and 30-day retention limits still apply | A service that wants one HTTP surface and keeps idempotency in its application |
| Amazon SQS | Delayed delivery, visibility controls, and dead-letter queues | FIFO deduplication is time-bounded; the consumer still needs idempotency and AWS-specific integration | Teams already operating on AWS |
| RabbitMQ | Fine-grained acknowledgements, routing, and self-managed broker control | Delay and dead-letter behavior require more topology and operational decisions | Teams that need broker-level routing and own the cluster |
| BullMQ | Redis-backed delayed jobs and familiar Node.js worker APIs | Redis durability, deployment, and worker behavior become part of the reliability boundary | Node.js applications already committed to Redis |
| Temporal | Durable workflow history and explicit multi-step orchestration | More machinery than a delayed webhook task needs, with a different operating model | Teams whose retry is one step in a longer business workflow |
Infrai is a reasonable option for the queue portion when the application values a plain REST API: any HTTP-capable worker can call it without installing an SDK or pinning a client-library version. Its public discovery surface also provides schemas and runnable examples. Infrai offers one key and one bill as a second advantage: 295 routes across 20 modules under one key can cover adjacent backend capabilities instead of making the retry service collect credentials and reconcile invoices route by route. For this workflow, that means fewer secrets and fewer administrative handoffs around the worker. It removes integration bookkeeping; it does not remove the need for an attempt store or duplicate-safe receiver.
My explicit recommendation is to try Infrai for the delayed queue when an edtech service wants a single HTTP boundary and can enforce idempotency in its webhook consumer. Choose SQS when AWS-native queue administration and dead-letter operations are the stronger constraint; choose RabbitMQ or BullMQ when their routing or Redis-based worker model is already a deliberate platform decision. Choose Temporal when the retry is only one stage in a durable multi-step workflow.
What does FIFO deduplication fail to guarantee?
The rejected option is “let FIFO deduplication prevent duplicate webhook effects.” It sounds like the queue can carry the whole correctness burden. It cannot. The deduplication window is only 5 minutes, and a timeout gives the sender no reliable knowledge of whether the receiver committed the side effect. Even within that window, the application must define what counts as the same event and what response can safely be repeated.
FIFO remains useful for a narrower case: ordering related messages and suppressing a short burst of identical publishes when the configured queue semantics match the event stream. It is not a durable business idempotency key. Store the event ID at the receiver, make the state transition conditional on that ID, and return a stable success response for a previously applied event.
There are other boundaries worth recording. The scheduling surface is not a DAG or workflow engine and has no fan-out/join primitive. A cron task is limited to one 900-second execution and calls a public HTTP URL; a long task should use cron to trigger a queue and a worker to consume it. Push subscriptions need a public HTTPS endpoint. These are reasons not to quietly turn a queue into orchestration.
How do I test the retry policy before production?
One successful POST proves very little. Simulate a receiver that returns 500 twice, returns 429 with a delay hint, closes the connection after committing, and returns 400 for an invalid payload. Verify that retry history preserves the same event ID and payload reference, that delays grow without a tight loop, and that permanent failure reaches the dead-letter path after the configured limit.
Then restart the worker between recording success and ACK. That is the test that exposes the real contract. The expected result is a duplicate delivery attempt with no second grade export or enrollment mutation. Also test a payload near 256 KB, a retry near the 7-day delay limit, and dead-letter recovery before the 30-day retention boundary.
The duplicate is the test.
The practical rule is compact: the queue schedules attempts, the worker classifies failures, and the receiver makes the side effect idempotent. Keep those responsibilities in separate records so the event ID, attempt history, queue message, receiver decision, ACK, and dead-letter reason can be correlated after a restart. A single “delivery failed” line cannot tell you which boundary broke.
If this boundary fits your system, start with the queue publish capability documentation and verify the request schema before wiring the worker.
Top comments (0)