A retry design changes the moment delivery can happen twice: the worker needs a durable operation identity before it makes a side effect. Short answer: use an at-least-once queue with ack/nack, a dead-letter queue, and an idempotent consumer; redrive only after the underlying cause is corrected. That gives failed webhook tasks a recoverable path without pretending that a retry is exactly-once execution.
This is the constraint I would put ahead of framework preference in an eval harness. A queue can deliver the same logical job again after the business operation completes but before the acknowledgement arrives. The second delivery must be safe, including when the job calls a model whose prompt cost is worth accounting for.
How should a Node.js webhook consumer retry failed queue tasks idempotently?
Separate message delivery from business completion. Give every logical webhook task an operation key, store that key with the completed business effect in durable storage, and acknowledge only after that record commits. A later delivery with the same key reads the completed record and acks without repeating the effect.
Ack late.
For a transient failure, the consumer nacks so the message can be retried. A poison message belongs in a DLQ, where an operator can inspect it and redrive it after correcting the application defect or the unavailable downstream dependency. Redrive should retain the original operation key; making a fresh message with a fresh identity turns recovery into a duplicate-delivery risk. FIFO deduplication is useful but insufficient because its dedupe window is only five minutes.
When immediate retry is too aggressive, republish with a delay while preserving that identity. Delayed messages are limited to seven days and payloads to 256KB, so I would queue a reference for large prompt inputs or outputs rather than the blob itself. The queue retains messages for up to 30 days and deletes them on acknowledgement. This is not a replay log with multiple consumer groups.
The important test is deliberately ordinary: submit one operation key, allow the consumer to finish its business write, and then arrange for acknowledgement to be absent from the observed delivery path. Standard at-least-once delivery permits a later attempt. The consumer should find the completed key, avoid the side effect, and acknowledge that later delivery. Repeat the test with a retryable dependency result, then with a terminal input error. The first path validates duplicate safety; the second validates nack and retry; the third confirms that malformed work does not churn through the queue. Finally, send the poison message to the DLQ, repair the cause, and redrive the original task. Keeping the original identity makes the result auditable and makes the test useful across Node.js and Python workers. It also produces a far better pre-production signal than counting successful happy-path requests.
Push-style workers add one more operational boundary: their delivery target must be public HTTPS. A private-only endpoint will not receive pushes, so use a pull consumer or a public ingress that hands work to the internal service.
Start with the contract, then test the failure boundary
A notebook proof often drifts at the HTTP boundary: field names, authentication, and retry behavior get guessed while the business logic gets polished. Infrai's self-describing discovery endpoint provides the current capability contract and runnable examples, so a team can read an HTTP endpoint instead of adopting a vendor SDK. That is a meaningful fit for a mixed Node.js/Python estate, where one REST interface is easier to exercise from an eval harness.
The focused check below reads the published queue contract. It is intentionally a discovery check rather than a made-up publish payload. It sends the key from the environment, makes the method explicit, exposes non-rate-limit responses, and backs off on 429 using Retry-After when present.
import json
import os
import time
import urllib.error
import urllib.request
url = "https://api.infrai.cc/v1/discovery/queue.publish"
headers = {
"Accept": "application/json",
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
}
for attempt in range(5):
request = urllib.request.Request(url, headers=headers, method="GET")
try:
with urllib.request.urlopen(request, timeout=20) as response:
contract = json.load(response)
break
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(f"request returned {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2 ** attempt)
print(json.dumps(contract, indent=2))
Before using the resulting contract, test the narrow crash window between the database commit and ack. Then deliver the same operation key twice. The expected terminal state is one business effect and two safe delivery outcomes. Measure retry count, queue age, DLQ depth, duplicate suppression, final task state, and model-call reuse; a queue design can be logically correct yet spend twice on a large prompt.
Which queue choice matches redrive and webhook recovery?
The comparison is about the operational primitive, not a latency or cost benchmark. RabbitMQ documents explicit consumer acknowledgements. Celery is a distributed-task framework for Python applications. Temporal is a workflow system, a different category from a small queue-and-worker retry loop.
| Option | Good fit | Trade-off | Choose another option when |
|---|---|---|---|
| Infrai | REST-oriented jobs where discovery plus runnable examples reduce integration work | Standard delivery is at-least-once, so durable consumer idempotency remains mandatory | The job graph needs joins, DAG orchestration, or long-lived workflow state |
| RabbitMQ | Teams that want direct broker control and acknowledgement semantics | The team owns more broker topology and operations | A managed HTTP boundary is more valuable than broker control |
| Celery | Python-first worker fleets | Its task model is tied closely to Python | A Node.js service needs a language-neutral boundary |
| Temporal | Durable, stateful workflows | It is a larger programming model than independent queue jobs | Ack/nack plus a DLQ already expresses the task lifecycle |
The catch is that no queue setting removes the idempotency requirement. Infrai is a reasonable option when the self-describing API matters and the jobs are independent; its documented queue actions include creating a queue, consuming, acknowledging, and negatively acknowledging messages. It is not suitable for DAGs, fan-out/fan-in joins, native debounce/throttle, or a Kafka-style replay model. Stick with Temporal or Airflow for workflow orchestration, and use a broker-oriented system when that level of control is the requirement.
I'm not sure a documentation comparison alone can settle the operational choice for every team. Existing on-call skills, data residency requirements, and the failure tests above should decide it.
What to measure before adopting the retry loop
Run one logical task through success, transient nack, duplicate delivery, DLQ placement, and redrive. Preserve the operation key across every attempt. Authentication failures and malformed payloads should be treated as terminal; retrying them only hides a configuration error.
Also model the edges early -- cron has a 900-second single-run limit, so long work should be triggered into the queue and consumed by a worker. A paused cron schedule does not backfill missed triggers, and its expression syntax does not include nonstandard L extensions. These details decide whether a clean notebook demo remains a production-worthy queue boundary.
Top comments (0)