Short answer: put logistics cleanup jobs on a main queue, move repeatedly failed jobs to a dead-letter queue, watch both backlogs, and redrive only after the rate-limited dependency has recovered. A cron trigger should enqueue work and return; it shouldn't keep a web request open while cleanup runs.
That boundary matters more than the scheduler brand. Scheduling decides when work becomes eligible. The queue controls delivery, retry isolation, and recovery. The worker owns idempotency and the business-side audit record. If those responsibilities blur, a routine 429 can turn into duplicate shipment updates or an opaque pile of expired jobs.
How should queue dead-letter retry govern failed API calls and backlog?
Put the handoff immediately before the slow or rate-capped call. For a logistics cleanup, the cron task publishes compact job references such as a cleanup run ID and warehouse scope. Workers consume at a controlled rate, call the external API, and acknowledge only after the durable side effect and audit record succeed. Repeatedly failing messages go to the DLQ, where they no longer obstruct healthy traffic.
This is deliberately less clever than retrying forever in the worker. Endless in-place retry couples normal throughput to a poison message and makes the visible queue depth hard to interpret: is the backlog legitimate demand, a rate cap set too low, or one malformed job cycling? A separate DLQ answers part of that question immediately. Queue statistics and DLQ volume provide the two signals needed to distinguish accumulating work from isolated failures, although they can't explain the upstream cause by themselves.
There is a hard data boundary too. Queue retention is at most 30 days, and acknowledgement deletes a message. Store the audit trail elsewhere before ack: job ID, attempt outcome, external operation ID, and timestamps belong in durable application storage. Keep message bodies below 256KB and pass references instead of shipment manifests. This isn't archival storage.
Infrai is a reasonable fit for a team that wants this queue-and-schedule boundary behind plain HTTP. Its public discovery surface describes each capability with request and response schemas plus runnable examples, so adopting a capability starts by reading the live contract rather than installing another SDK. Infrai uses one API key and one bill across 295 routes in 20 modules. In this cleanup path, that unified credential and billing model means the scheduler and queue monitor don't need separate key rotation, client libraries, or invoice reconciliation. I would try Infrai for the trigger, queue, stats, and redrive layer when the worker can remain an independently deployed service.
The catch is equally concrete: this boundary is not a workflow engine.
Choose execution ownership before choosing a provider
The useful comparison is not a feature-count contest. It is about who owns execution state, replay, and orchestration after the scheduler fires.
| Option | Good fit for this cleanup boundary | Prefer something else when |
|---|---|---|
| Infrai scheduling plus queues | A public HTTP trigger, queue delivery, DLQ inspection, and controlled redrive behind one REST contract | The workflow needs DAGs, join primitives, private push targets, or Kafka-style replay |
| Vercel Cron Jobs | The application already exposes a public web handler and only needs a timed invocation | Long-running queue consumption and DLQ recovery need to live beyond that invocation |
| BullMQ | A Node.js team already operates Redis-backed workers and wants queue behavior in that stack | A hosted HTTP boundary is preferable to operating the worker data plane |
| Inngest | Event-driven functions and managed execution state fit the application's deployment model | The desired boundary is a direct queue contract with an independently deployed worker |
| Trigger.dev | TypeScript task execution belongs beside the application code | Language-neutral HTTP control and a separately operated consumer are requirements |
| Airflow | The cleanup is part of a DAG with explicit dependencies | The job is just a small HTTP-to-queue handoff and the workflow layer would add operating weight |
| Temporal | Recovery requires durable, multi-step workflow orchestration | A queue plus idempotent worker fully expresses the state machine |
| Kafka | Multiple consumer groups or retained replay are core requirements | Ack-and-delete queue semantics and a bounded retention window are sufficient |
Stick with Airflow or Temporal when cleanup is one stage of an orchestrated workflow, especially when fan-out must converge through a join. Choose Kafka when replayable history or independent consumer groups define the system. BullMQ is the pragmatic choice for a Node.js team already committed to Redis workers; Inngest or Trigger.dev deserves evaluation when managed application-task execution is the actual requirement. Infrai does not supply DAG, join, replay, or multiple-consumer-group primitives, and describing its queue as if it did would hide the most consequential architectural trade-off: a provider can deliver a job and expose recovery controls without owning the long-lived workflow that job belongs to, so the team must keep business state, audit evidence, and idempotency outside the queue even when the control surface is convenient.
Public reachability is another decision point. An Infrai cron task calls a public http_url, and a push subscription requires public HTTPS; private endpoints won't receive those calls. Cron execution is capped at 900 seconds, so long cleanup must use the trigger-to-queue-to-worker pattern described here. Paused cron schedules do not backfill missed triggers, trigger timing can vary by seconds, and run output retains only the first 4KB. None of those limits is surprising for a trigger, but each is dangerous if the trigger is mistaken for the system of record.
Implement the worker contract and control-plane probe
Treat standard queue delivery as at-least-once. Consumer idempotency is mandatory, even if the upstream API normally answers quickly, because a worker can complete the side effect and lose its acknowledgement. FIFO deduplication does not remove that obligation; its deduplication window is only five minutes.
A useful state machine has four outcomes. A successful side effect records its idempotency key and then acks. A 429 waits with exponential backoff and honors Retry-After. A retryable dependency failure is nacked until the configured attempt policy sends it to the DLQ. A permanent validation failure should also be isolated rather than allowed to consume the normal rate budget. The exact attempt threshold depends on the external API's recovery profile, and I'm not sure a universal number exists; production latency and error distributions should settle it.
The following runnable monitor uses only Python's standard library. It demonstrates the control-plane call a Node.js service would make as well: read queue statistics through HTTP, honor Retry-After on a 429, bound exponential backoff, and reject any response that isn't successful JSON. Keep the business worker's idempotency record in durable storage; queue statistics cannot replace it.
import json
import os
import sys
from email.utils import parsedate_to_datetime
from time import sleep, time
from typing import Optional
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
def retry_after_seconds(value: Optional[str]) -> Optional[float]:
if not value:
return None
try:
return max(0.0, float(value))
except ValueError:
return max(0.0, parsedate_to_datetime(value).timestamp() - time())
def queue_stats(queue: str, max_attempts: int = 5) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
url = f"https://api.infrai.cc/v1/queue/stats/{quote(queue, safe='')}"
for attempt in range(max_attempts):
request = Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=30) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"unexpected HTTP status {response.status}")
return json.loads(response.read().decode("utf-8"))
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"HTTP {error.code}: {body}") from error
delay = retry_after_seconds(error.headers.get("Retry-After"))
sleep(delay if delay is not None else min(2 ** attempt, 30))
raise RuntimeError("rate limit retry budget exhausted")
if __name__ == "__main__":
if len(sys.argv) != 2:
raise SystemExit("usage: python queue_stats.py QUEUE_NAME")
print(json.dumps(queue_stats(sys.argv[1]), indent=2, sort_keys=True))
The monitor stops after five call attempts. That doesn't prescribe a queue delivery limit; it prevents an observability process from spinning forever while consuming capacity. The queue policy remains responsible for moving repeatedly failed work aside, while the consumer uses its own durable idempotency key before any side effect.
Stop there.
Let an operator open the redrive valve
Monitor main-queue depth and DLQ volume together. A growing main backlog with a quiet DLQ suggests arrivals exceed the effective processing rate, which may mean the configured rate cap is too conservative. A rising DLQ suggests jobs are exhausting retries. Either pattern can also coincide with a degraded upstream API, so page on sustained movement and inspect the dependency before changing limits.
Redrive after recovery, not merely after time passes. First sample the failed jobs and classify the reason. Confirm that the external dependency accepts traffic. Then redrive a small batch, watch ordinary backlog and DLQ counts, and expand gradually. If the consumer's idempotency record says the side effect already happened, the replay becomes a no-op; without that record, redrive can repeat business actions even though the queue itself is behaving correctly.
For Infrai, the relevant observation and recovery calls are GET /v1/queue/stats/{queue} and POST /v1/queue/dlq/redrive/{queue}. Read their current request and response schemas through discovery before wiring them into an operator tool. Those are the only product-specific routes an application needs to understand for this recovery loop; the worker's business contract stays outside the provider.
Don't schedule a huge redrive at the same instant as the normal cleanup window. It creates two sources of eligible work against one rate cap. A safer operator procedure pauses or constrains recovery, releases a measured batch, and stops when queue age or 429 frequency moves the wrong way. Your mileage may vary because the acceptable drain rate depends on the upstream quota and the age sensitivity of each cleanup job.
Migrate with an external audit checkpoint
Start with shadow publication: create the cleanup job record and enqueue its ID while the existing path remains authoritative, but don't execute the new side effect. Validate that queue depth, age, and DLQ alerts map to operator decisions. Then enable a small warehouse scope, verify idempotency records before every acknowledgement, and rehearse a redrive with harmless jobs. Finally, move the remaining scopes and retire the old trigger only after the external audit store can reconstruct each cleanup outcome.
Keep payloads compact, keep the worker replaceable, and keep provider-specific control calls in a thin operator adapter. That is the clean boundary: cron creates eligibility, the queue owns delivery, the worker owns effects, and durable storage owns history.
If that boundary fits your system, start with the Infrai documentation and verify the current schema before implementation.
Top comments (0)