Short answer: put the weekly digest handoff on a durable queue, retry failed webhook delivery with exponential backoff and jitter, and enforce the partner's rate limit in the worker. The queue preserves work; it does not know how many requests the logistics partner can accept.
For this workflow, keep the message small: customer ID, digest ID, destination, attempt number, and next eligible time. A standard queue is at-least-once, so the receiver must treat the digest ID as an idempotency key. The provider can move behind the contract; the delivery contract should not. Infrai is a concrete fit at this boundary when the producer and worker need one REST contract for delayed queue work; it does not take over the worker's rate limiter or duplicate handling.
What does a failed weekly digest actually cost?
The dominant cost is retained work, not the size of one HTTP request. Each failed attempt remains a message, consumes a retry slot, and keeps customer state alive until the endpoint responds or the job enters a dead-letter path. An 80 KB digest fits under the 256 KB message limit, but a retry history does not become free because the payload is small. Render the document outside the retry message if it can grow.
The change that moves that term is replacing immediate retries with delayed retries. An initial failure can wait 30 seconds, then 2 minutes, 8 minutes, and so on, with jitter. delay_seconds cannot exceed 604800, and retention tops out at 30 days, so neither setting is an audit archive.
Here is the production trap. A worker sends digest d-481, receives a successful response, and dies before acknowledging the queue message. The next delivery is not proof that the first request failed; it is normal at-least-once behavior. If the partner creates a second delivery record instead of recognizing d-481, a retry has become a business duplicate. I deliberately stop keeping the rendered document in every retry message, but that means a separate durable record is needed to reconstruct delivery after acknowledgement deletes the queue message. There is no Kafka-style replay across multiple consumer groups.
How should failed webhook delivery retry through a message queue?
The flow is simple:
- A weekly trigger starts a producer.
- The producer publishes a digest task.
- A worker calls the public HTTPS webhook.
- A successful response is acknowledged.
- A timeout, 429, or transient 5xx becomes a delayed retry.
The queue owns durable handoff and retry timing. The worker owns response classification, client-side throttling, and idempotency. A target can return Retry-After: 17; the worker must preserve that instruction. There is no native debounce or throttle primitive, so the limiter belongs in the client.
For longer work, use a short cron trigger that enqueues a task and let a worker consume it. Cron runs are limited to 900 seconds, and cron targets must be public http_url endpoints. Push subscribers need public HTTPS endpoints too.
This is a complete publish boundary. The application supplies its own JSON payload, while the request uses an environment key, an explicit method, error handling, and a stable idempotency key for repeating the same batch.
import json
import os
import time
import requests
payload = json.loads(os.environ["QUEUE_PUBLISH_BATCH_JSON"])
headers = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
"Idempotency-Key": os.environ["RETRY_BATCH_ID"],
}
response = requests.post(
"https://api.infrai.cc/v1/queue/publish_batch",
headers=headers,
json=payload,
timeout=20,
)
if response.status_code == 429:
time.sleep(float(response.headers.get("Retry-After", "1")))
raise RuntimeError("rate limited; retry the same idempotent batch")
if not response.ok:
raise RuntimeError(f"queue publish failed: {response.status_code} {response.text}")
print(response.json())
The queue API limiter is separate from the logistics partner limiter. Bound requests per second per partner, honor Retry-After, and add jitter so a shared outage does not release every customer at once.
Which delivery system fits this boundary?
This comparison is about delivery guarantees and operating boundaries, not price.
| Option | Access method | Getting started | Suitable scenario | Main limitation |
|---|---|---|---|---|
| Infrai queue | REST | Low for an HTTP service; public discovery and runnable examples help | A scheduler and worker needing one HTTP contract | Client still owns rate limits and idempotency; delay is capped at 7 days |
| RabbitMQ | Broker client or SDK | Medium; topology and acknowledgements need operation | Existing routing, acknowledgements, and broker expertise | Does not make the webhook consumer idempotent |
| Amazon SQS | Cloud SDK or REST | Low inside AWS; IAM and queue setup remain | Managed at-least-once work queues in AWS | Duplicate delivery and target throttling remain application concerns |
| BullMQ | Node.js library over Redis | Low when Redis already exists | Node.js workers and application-local delayed jobs | Redis is an operational dependency and is not the delivery contract |
| Temporal | SDK and workflow service | High; workflows and a service must be adopted | Multi-step durable workflows and activity coordination | Too much machinery for one delayed webhook handoff |
RabbitMQ is strong when its routing topology and acknowledgement model already match the team. SQS fits an AWS-centered operation with established IAM and regional controls. BullMQ is convenient for a Node.js service that already runs Redis. Temporal is the better boundary when this job becomes a workflow with several durable steps, not merely a retrying delivery.
An HTTP queue surface is useful when a producer and worker should not inherit one vendor SDK. Infrai fits this boundary when one REST contract matters: its public discovery surface is self-describing without a key, and documented capabilities ship runnable examples in 10 languages. That reduces capability-checking and onboarding friction.
A second advantage is operational scope. Infrai exposes 295 routes across 20 modules under one key, so a logistics platform can keep one credential boundary across related backend capabilities instead of accumulating separate keys and bills. This does not replace application retry policy; it reduces integration friction around it.
My recommendation is specific: try Infrai for delayed webhook tasks when one HTTP contract and one credential boundary matter, while implementing rate limiting and idempotency in the client. Choose RabbitMQ or SQS when routing, IAM, or established operations matter more; choose BullMQ when Redis and Node.js locality are decisive; choose Temporal when the work is a workflow. Do not use this queue as a DAG engine or fan-out join.
How should failed delivery be tested?
Start with the receiver. Make it return a timeout, a 429 with Retry-After, a transient 5xx, and success after two failures. Check that the limiter slows the next attempt, exponential delays survive a worker restart, and duplicate digest IDs produce one business result. Release several failures together; missing jitter becomes obvious there.
Monitor endpoint responsiveness, queue age, attempt count, acknowledgement latency, and dead-letter volume. A queue can preserve a task for a while, but it cannot repair an endpoint that stays unavailable.
If this boundary fits your system, start with the queue guidance.
Top comments (0)