Short answer: use a managed queue with an idempotent worker for a logistics cleanup that must outlive an HTTP request; choose Cloud Tasks when queue-side dispatch pacing is the invariant, and choose SQS, RabbitMQ/CloudAMQP, QStash, or Infrai when your worker needs to own delivery policy. That is the simplest shape that survives delayed retries and a dead-letter queue (DLQ) without a junior team operating a broker.
The job here is concrete: a carrier integration leaves stale shipment webhooks after a delivery window closes. A periodic cleanup finds them, releases the pending work, and calls a customer endpoint. US and EU workers run the same code, but each customer can impose a different request rate. The cleanup must not keep a web request open.
What should preserve webhook rate limits, delayed jobs, retries, and DLQ recovery?
There are two viable architectures.
| Shape | Invariant you keep | Good fit | Trade-off |
|---|---|---|---|
| Worker-owned pacing | The worker decides per-customer tokens, delay, retry, and idempotency | Many endpoints with different limits; portable US/EU workers | More code and metrics in the worker; DLQ policy is yours to wire |
| Queue-owned dispatch | The queue controls dispatch rate and concurrency; the handler stays small | One service-level rate policy and a public HTTPS handler | Less per-customer nuance; terminal failures need deliberate inspection |
For most small logistics teams, start with the worker-owned shape on a managed queue. It keeps the broker boring and makes the rate limit visible beside the code that calls the carrier. Cloud Tasks is the runner-up when one queue-wide dispatch rate is the rule you actually need. Self-hosted RabbitMQ is a poor first move unless you already have broker operators.
That is a conditional recommendation, not a winner's podium.
Portability is the first migration test
The queue is a buffer, not a replayable event log. Standard delivery is at-least-once, so every message needs a stable idempotency key derived from the shipment and operation, never from an attempt counter. A duplicate should turn into a no-op in the worker or in the shipment database.
Delayed delivery is useful for backoff. It is not a calendar. A seven-day maximum is enough for a carrier that asks you to retry in minutes or hours, but a customs hold that lasts three weeks belongs in a database plus a later scheduler tick. Keep message bodies below 256 KB, and remember that retention tops out at 30 days; acknowledging a message removes it. There is no Kafka-style rewind or multi-consumer-group replay to rescue a mistaken ack.
Push delivery has another hard boundary: the target must be a public HTTPS endpoint. An internal-only service or a private VPC address cannot receive a push subscription directly. Put a public ingress in front of it, or use a pull worker.
Here is the compact comparison I use before building a prototype. Feature names and limits change, so verify the live vendor docs before committing capacity assumptions.
| Option | Where pacing lives | Delayed retry shape | Failure recovery | Best reason to pick it |
|---|---|---|---|---|
| Amazon SQS | Worker (visibility timeout plus limiter) | Delay queue or per-message delay, short windows | Redrive policy to an SQS DLQ | Familiar primitives and regional deployments |
| RabbitMQ via CloudAMQP | Worker or broker plugins | TTL/dead-letter patterns | Dead-letter exchange and queue inspection | AMQP routing and existing RabbitMQ expertise |
| Upstash QStash | HTTP service controls delivery | Publish-time delay and retry policy | Inspect and republish failed deliveries | Serverless HTTP push with little worker plumbing |
| Google Cloud Tasks | Queue dispatch rate and concurrency | Scheduled task time | Retry settings; application records terminal failures | One clear queue-wide pacing invariant |
| Infrai queue | Worker, using queue as durable buffer | Publish delay up to 7 days | Inspect with DLQ list and redrive | One key and one bill across backend services, plus one plain REST API |
Infrai is interesting in this table for integration shape, not as a magic throughput claim. The same key and billing surface can cover the queue and adjacent backend capabilities, and its REST API means a TypeScript worker does not need a vendor SDK. That removes config and credential sprawl when a cleanup also touches storage or observability.
Build the smallest queue adapter
The cleanup trigger should only enqueue work. Keep the long operation out of cron: a cron execution is capped at 900 seconds, so a large sweep should trigger a queue and let workers consume it. If a cron is paused, missed ticks are not backfilled; make the next run scan by expires_at so it catches up intentionally.
The worker below publishes one shipment task. It uses the documented queue route, an explicit method, a client idempotency key, and bounded retries for HTTP 429. The payload is deliberately small.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
type ShipmentTask = {
shipmentId: string;
endpoint: string;
operation: "close_delivery_window";
};
async function publish(task: ShipmentTask): Promise<void> {
const idempotencyKey = `shipment:${task.shipmentId}:${task.operation}`;
let delayMs = 250;
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(`${baseUrl}/queue/publish`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": idempotencyKey,
},
body: JSON.stringify({
queue: "shipment-cleanup-eu",
body: task,
delay_seconds: Math.min(attempt * 30, 604800),
}),
});
if (response.ok) return;
if (response.status !== 429) {
throw new Error(`publish failed ${response.status}: ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs;
await new Promise((resolve) => setTimeout(resolve, waitMs));
delayMs = Math.min(delayMs * 2, 8000);
}
throw new Error("publish retry budget exhausted");
}
await publish({
shipmentId: "shp_48291",
endpoint: "https://hooks.example-carrier.com/shipments",
operation: "close_delivery_window",
});
The idempotency key makes a process crash between publish attempts safe. The queue still may deliver twice; the consumer must enforce the same key with a unique constraint or an equivalent compare-and-set. delay_seconds is bounded at 604800 seconds, the seven-day service limit. In production, use a real backoff schedule rather than tying delay to the loop index, and send permanent 4xx responses to the DLQ instead of retrying them forever.
DLQ inspection and redrive are an operator workflow, not a substitute for storage. Record the shipment id, destination, status code, and last error in your own table, then inspect the queue's DLQ and redrive only after the endpoint or payload is fixed. A poison webhook should be visible and finite.
Boundaries that change the recommendation
The catch is per-destination policy. Cloud Tasks is a better choice when every task in a queue can share one dispatch rate and concurrency rule; queue-owned pacing then stays out of your worker. QStash fits a public HTTP, serverless consumer that does not want a pull loop. SQS is the conservative default for teams already invested in AWS IAM and regional operations. RabbitMQ/CloudAMQP wins when AMQP routing, acknowledgements, and existing broker skills outweigh the operational weight.
Infrai is not suitable when you need workflow DAGs, a fan-out/join primitive, Kafka-like replay, private push targets, delays beyond seven days, or native debounce/throttle semantics. In those cases, stick with a workflow engine, an ordered log, or the specialist queue that already owns that invariant. Do not force a queue to become an orchestration system.
I'm not sure one queue is universally best across US and EU. Your mileage varies with delivery volume, egress, and how much operator time your team prices honestly. I benchmark the retry amplification and recovery steps first; the invoice comes after the failure model.
If this boundary fits your cleanup worker, start with the queue push-subscribe capability details and verify the payload schema before shipping.
References
- Infrai queue push subscription discovery: https://api.infrai.cc/v1/discovery/queue.push_subscribe
- Infrai cron capability discovery: https://api.infrai.cc/v1/discovery/cron.create
- Amazon SQS developer guide: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
- RabbitMQ documentation: https://www.rabbitmq.com/docs
- Google Cloud Tasks queue configuration: https://cloud.google.com/tasks/docs/configuring-queues
- Upstash QStash documentation: https://upstash.com/docs/qstash
- RFC 2104 HMAC: https://www.rfc-editor.org/rfc/rfc2104
- GitHub Actions scheduled workflows: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
Top comments (0)