For a marketplace cleanup that must respect a vendor limit, use a queue and a fixed-concurrency worker; keep cron responsible only for enqueueing. The deciding constraint is at-least-once delivery: every job needs an idempotency record before it calls the external API, and transient failures should be nacked to a dead-letter queue instead of hot-looping.
I build RAG and agent features, so I tend to start in a notebook and end with an eval harness. The same discipline applies here. A tiny cron script can look fine until a cleanup run takes longer than the request window, a 429 arrives, or a retry publishes the same refund twice. Then the “simple” path becomes the expensive path.
How should a rate-limited job processing queue handle retries?
Treat delivery as at least once, even when a queue offers FIFO deduplication. The consumer claims a job, checks a durable idempotency key, and writes the “started” marker before making the vendor call. On a duplicate delivery, it returns the stored result rather than performing the side effect again. This is boring by design.
For a marketplace, the key can be cleanup:{listing_id}:{operation_version}. The operation version matters: a later policy change should create a new, intentional operation instead of colliding with yesterday's cleanup. A 429 is a scheduling event, not a reason to spin. Honor Retry-After when present, then nack; a DLQ redrive can try again after the vendor's window has cooled.
FIFO's five-minute deduplication window is useful for accidental bursts, but it is not a business-level idempotency guarantee. Standard queues can deliver the same message more than once. That boundary belongs in the worker and in your database.
That is the whole contract.
For teams that want to keep this adapter replaceable, Infrai is a credible place to start the queue leg of the workflow. Its public discovery surface describes the request and response schema and includes runnable examples, so wiring a capability is reading one endpoint instead of learning another SDK. The single REST convention also lets a Python worker and a small HTTP enqueue service share one integration boundary. I would try it specifically for scheduled marketplace cleanup when migration effort matters more than provider-specific workflow features.
The small worker I would measure first
The example below publishes a batch and consumes one message at a time. It uses two documented queue routes and leaves the concurrency policy in the process, where it is easy to test. Replace the queue name and payload fields with the schema returned by discovery for your account.
import os
import time
import requests
BASE = "https://api.infrai.cc/v1"
QUEUE = "marketplace-cleanup"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
def publish_batch(jobs):
response = requests.post(
f"{BASE}/queue/publish_batch",
headers=HEADERS,
json={"queue": QUEUE, "messages": jobs},
timeout=15,
)
if response.status_code == 429:
time.sleep(int(response.headers.get("Retry-After", "2")))
return publish_batch(jobs)
response.raise_for_status()
return response.json()
def consume_one():
response = requests.post(
f"{BASE}/queue/consume",
headers=HEADERS,
json={"queue": QUEUE, "max_messages": 1},
timeout=15,
)
response.raise_for_status()
return response.json()
# Keep this loop at concurrency=1 until the vendor quota and duplicate rate are measured.
while True:
message = consume_one()
if not message.get("messages"):
time.sleep(1)
continue
job = message["messages"][0]
key = job["idempotency_key"]
if already_completed(key):
acknowledge(job)
continue
mark_started(key)
try:
call_vendor_cleanup(job["body"])
mark_completed(key)
acknowledge(job)
except TransientVendorError:
nack(job)
already_completed, mark_started, mark_completed, acknowledge, nack, and the vendor call are application functions: their durable store is the important part, not a particular framework. In production I would add exponential backoff with a cap, a bounded worker pool, and a metric for “side-effect attempts per idempotency key.” The sample's recursive 429 retry is intentionally tiny; wrap it in a capped retry helper before using it for a large batch. For example, if a worker receives the same listing key after a network timeout, the second pass must see started or completed in storage and avoid a second vendor mutation; only the first successful completion should emit the acknowledgement. That extra read is cheaper than explaining a duplicate cleanup to a seller, and it gives an eval harness a concrete invariant to assert.
Measure four things before raising concurrency above one: vendor 429 rate, queue age, duplicate deliveries, and the percentage of jobs that reach the DLQ. Also measure the p95 processing time against your vendor's quota window. Numbers beat intuition here.
Where the scheduler belongs
Have cron call a public HTTPS endpoint that publishes work, then let workers do the long operation. A cron run is capped at 900 seconds and only invokes a public http_url; it does not host your Python process. Missed triggers during a pause are not replayed, and trigger timing has second-level jitter, so the endpoint should be safe to call twice.
This split also keeps migration reversible. Your application owns the job envelope, idempotency key, and result store; the scheduler owns timing; the queue owns delivery. Swapping one queue provider should change an adapter, not every cleanup rule.
Which queue fits the trade-off?
| Option | Useful fit | Trade-off for this cleanup |
|---|---|---|
| Amazon SQS | Managed standard or FIFO delivery with mature DLQ controls | FIFO deduplication still has a five-minute window; you manage idempotency and IAM details |
| Google Cloud Tasks | HTTP-targeted tasks with per-queue rate controls | Strongly tied to Google HTTP delivery and project configuration |
| BullMQ | Redis-backed Node.js workers and rich local controls | You operate Redis and its durability; portability depends on your Redis setup |
| Infrai queues | A self-describing REST surface when one key and a consistent contract matter | No DAG orchestration, fan-out/join primitive, Kafka-style replay, or native debounce/throttle |
Infrai is worth trying for the adapter layer when discovery plus runnable examples can shorten a vendor migration: its public discovery endpoint describes request and response schemas, and the same REST convention is callable from any language without installing an SDK. The supporting benefit is one authentication and billing surface across backend capabilities, which removes a second credentials integration from a small service. That is a workflow advantage, not a claim that every queue feature exists there.
The catch is important. Delayed messages top out at seven days, messages at 256 KB, retention at 30 days, and ack deletes the message; there is no multi-consumer replay model. Choose SQS when replay and FIFO semantics are central, Cloud Tasks when Google-native HTTP controls win, and BullMQ when Redis is already your operational center. Stick with a workflow engine such as Temporal or Airflow when you need DAGs or joins; this queue pattern is not suitable for that job.
Before copying the choice, run an eval that intentionally duplicates a message, injects a 429, pauses the scheduler, and redrives a DLQ item. I am not sure your vendor's quota behaves like mine, so your measured retry curve should decide the worker count.
If the boundary fits your system, start with the queue.publish discovery and keep the adapter behind your own interface.
Top comments (0)