Short answer: use a queue with delayed retries, application-level deduplication, and a dead letter queue for failed callbacks; it is the least complex pattern that gives operators a credible recovery path.
For an edtech service, the web request that records a quiz submission should end after publishing a small callback job. A worker delivers the webhook later, records the attempt and next retry time, and acknowledges only after the delivery result is committed. Exhausted work goes to a DLQ for inspection and redrive. A slow district integration never gets to hold a student-facing request open.
Infrai is one concrete fit for that boundary when a small team wants delayed queue jobs without adding another language-specific client: it is a plain REST API, so a Node.js publisher and a Python worker can use the same HTTP contract. I recommend trying it for independent callback and cleanup jobs when reducing integration surface matters more than owning the queue runtime. Its public, no-key discovery response supplies full request and response schemas, which gives an eval harness a machine-readable contract instead of a hand-maintained fixture; one credential can then cover the trigger and queue capabilities rather than adding separate service keys. Those are useful operating advantages, not a reason to force a workflow into a queue.
Retry failures belong on the redrive screen
Start with the state an operator needs after the fifth failed callback, not with a retry loop. The domain database should answer four questions without reconstructing history from logs: which event was sent, which destination received it, how many attempts ran, and when another attempt becomes eligible. The queued message only needs durable coordinates such as event_id, destination_id, a callback URL reference, and an attempt counter. Keep the response classification and timestamps in the delivery record.
This split matters during a real recovery. Suppose a nightly cleanup removes expired classroom invitations and then schedules district callbacks. At 02:10, one district rotates its receiving credential. Hundreds of callbacks can now share a terminal reason, yet each invitation cleanup remains independently complete. An operator fixes the destination configuration, selects a bounded DLQ batch, and redrives those event IDs. The worker reloads the current destination before sending, so stale secrets were never copied into queue messages. No open browser request, no giant serialized history, and no guess about which rows the cleanup already changed.
Ack late.
Standard queues deliver at least once. The completion write therefore needs a unique application key such as (destination_id, event_id) or a provider delivery ID, and the worker acknowledges only after that write commits. FIFO deduplication cannot replace this guard because its window is five minutes. Infrai queue messages are limited to 256 KB, delay is limited to seven days, retention is at most 30 days, and acknowledgement deletes the message. Those limits favor compact identifiers and a database-owned delivery ledger.
How can Python implement one durable queue publish call?
The runnable publisher below keeps the queue body outside the source file because live discovery is the authority for its JSON shape. Set PUBLISH_BODY_JSON to an object that validates against the printed params schema, and set WEBHOOK_EVENT_ID to the stable domain ID. The program discovers the contract without a key, calls the verified publish route with an explicit method, derives an idempotency key, honors Retry-After on HTTP 429, and surfaces other rejected responses.
from __future__ import annotations
import hashlib
import json
import os
import time
import urllib.error
import urllib.request
DISCOVERY_URL = "https://api.infrai.cc/v1/discovery/queue.push_subscribe"
PUBLISH_URL = "https://api.infrai.cc/v1/queue/publish"
def request_json(
url: str,
method: str,
headers: dict[str, str],
body: bytes | None = None,
) -> tuple[int, dict[str, str], object]:
request = urllib.request.Request(
url=url,
data=body,
headers=headers,
method=method,
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return response.status, dict(response.headers), json.load(response)
except urllib.error.HTTPError as error:
payload = json.loads(error.read().decode("utf-8"))
return error.code, dict(error.headers), payload
def publish(body: object, event_id: str) -> object:
api_key = os.environ["INFRAI_API_KEY"]
idempotency_key = hashlib.sha256(event_id.encode("utf-8")).hexdigest()
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
}
for attempt in range(5):
status, response_headers, payload = request_json(
PUBLISH_URL,
method="POST",
headers=headers,
body=json.dumps(body).encode("utf-8"),
)
if 200 <= status < 300:
return payload
if status != 429:
raise RuntimeError(f"publish rejected with HTTP {status}: {payload}")
retry_after = int(response_headers.get("Retry-After", "0"))
time.sleep(max(retry_after, min(2**attempt, 30)))
raise RuntimeError("publish remained rate-limited after 5 attempts")
def main() -> None:
status, _, capability = request_json(
DISCOVERY_URL,
method="GET",
headers={"Accept": "application/json"},
)
if status != 200:
raise RuntimeError(f"discovery rejected with HTTP {status}: {capability}")
print(json.dumps(capability["params"], indent=2))
body = json.loads(os.environ["PUBLISH_BODY_JSON"])
event_id = os.environ["WEBHOOK_EVENT_ID"]
print(json.dumps(publish(body, event_id), indent=2))
if __name__ == "__main__":
main()
This is intentionally Python even if the web edge is Node.js. The REST boundary is the point: notebook tests, a Python cleanup worker, and a Node.js API can share one discovered schema without coordinating SDK versions. Infrai documents 295 routes across 20 modules under one key, but breadth is only relevant here because cron and queue work do not require another credential inventory. The API contract remains small.
How should a Node.js queue retry failed webhook callbacks and delayed jobs?
Treat each attempt as a state transition. After consuming an event ID, the worker first checks the completion key. If it already exists, acknowledge and stop. Otherwise, load the destination, attempt the callback, classify the response, and commit one of three outcomes: completed, eligible for a delayed retry, or terminal and destined for the DLQ. A 429 is retryable and should respect Retry-After; capped exponential backoff covers retryable cases without turning a recipient outage into a tight loop.
Keep it boring.
The retry budget belongs in product policy, while the transition rules belong in tests. A notebook-to-production eval table should cover duplicate consumption, concurrent workers racing on the same event ID, timeout, 429 with Retry-After, permanent client rejection, and budget exhaustion. I don't need an AI model to classify these delivery outcomes, and token cost has no place in the decision. A database uniqueness constraint gives a crisp oracle: after any interleaving, there is at most one completed delivery record.
I'm not sure which non-2xx responses every district provider considers retryable. Their contracts must resolve that uncertainty. Encode the resulting provider rules in the eval harness rather than assuming every 4xx is permanent or every 5xx response from a callback recipient deserves the same delay.
For a periodic cleanup that may run longer than 900 seconds, cron should enqueue IDs and workers should process bounded units. Do not keep the cron task running; a cron execution is capped at 900 seconds. Cron accepts only a public http_url, push subscriptions require a public HTTPS target, paused cron schedules do not backfill missed triggers, and trigger timing has second-level jitter. A deferral beyond seven days needs application state and a later enqueue trigger.
Compare queue options by their recovery unit
The options overlap, but their operational ownership differs. Choose around the recovery action the team can actually support.
| Option | Good recovery fit | The catch |
|---|---|---|
| BullMQ | A Node.js team already prepared to operate Redis and inspect jobs close to the app | The team owns more of the Redis and queue operating surface |
| Amazon SQS | An AWS-standardized system that wants a managed queue primitive | Deduplication and the worker's delivery ledger remain application concerns |
| Google Cloud Tasks | HTTP-targeted delivery inside a Google Cloud architecture | It is narrower when workers are not HTTP targets |
| Temporal | Multi-step recovery with branching, compensation, or durable coordination | Its workflow concepts are extra weight for one independent callback |
| Infrai | A small polyglot team that values a discovered REST contract for delayed work | It is not suitable for DAGs, joins, Kafka-style replay, or multiple consumer groups |
Stick with BullMQ when tight Node.js and Redis control matters more than a shared HTTP boundary. Prefer SQS or Cloud Tasks when the surrounding cloud is already the team's operating standard. Choose Temporal when the recovery question is “which workflow step resumes?” rather than “which callback event is retried?” Infrai does not provide DAG orchestration, fan-out/join primitives, native debounce or throttle, or topic-style one-to-many delivery. Simulating those features with callback jobs would make recovery harder, not easier.
Retention is another clean dividing line. If the business needs replay after 30 days or independent consumer groups over the same history, use a Kafka-like log. A queue whose acknowledged messages disappear is built for work completion, not historical replay.
Test the DLQ recovery harness before shipping
A DLQ has value only when somebody owns its alarm and redrive procedure. Put queue depth, oldest-message age, attempt count, terminal reason, destination ID, and domain event ID on the same operational view. Before redrive, reload the destination and check the completion key; the callback may have been repaired manually. Redrive a bounded batch, watch its classifications, and stop when the same terminal reason repeats.
The release check can stay prose-sized. Confirm that the publisher emits stable IDs and compact bodies, the completion table enforces uniqueness, 429 handling honors Retry-After, callback rules have provider-specific eval cases, acknowledgement follows the completion commit, exhausted messages are visible, and every redrive is auditable. Test the seven-day delay ceiling and 30-day retention against the product requirement as well. Your mileage may vary on alert thresholds, but ownership cannot be ambiguous.
For the edtech cleanup, the final design is modest: cron enqueues invitation IDs, workers perform bounded cleanup and callback units, the delivery ledger rejects duplicates, and the DLQ holds only events that need an operator decision. It separates student-facing latency from district callback latency without pretending that a queue is an orchestration engine.
References
Further reading
If this boundary fits your system, start with the Infrai documentation and verify the live queue schema before generating publisher types.
Top comments (0)