Rate-Limited Job Processing — Backend Queue API vs Cron Per-Minute Limits

#queues #backend #scheduling

Short answer: use a queue plus a worker-enforced per-minute rate limit for customer-support background jobs, and use cron only to enqueue periodic work; because standard queues deliver at least once, make the consumer idempotent before tuning throughput.

This is an architecture decision, not a contest between product home pages. A support system that updates a CRM, classifies a ticket, and sends a follow-up has to survive duplicate delivery, a worker restart, and an upstream 429 Too Many Requests without doing the business action twice. Cron tells something when to start. It does not provide native debounce or throttle, and it is the wrong place to hold a long-running drain loop.

The governing rule is blunt: the rate limiter protects the dependency; the idempotency record protects the customer.

Begin with the duplicate-delivery timeline

Cron should own time-based intent: enqueue the nightly backlog scan, the five-minute stale-ticket sweep, or another bounded periodic trigger. The queue should own individual units of work and absorb bursts. Workers should own admission to the upstream API, retries, and the transition from pending to completed. This separation keeps a scheduler delay from becoming a duplicate email and keeps an API slowdown from stretching a scheduled invocation indefinitely.

Four invariants matter:

A stable operation key identifies the business effect, such as ticket-1842:sync-crm:v3; a delivery ID alone is insufficient because a retry may receive a new delivery identity.
The idempotency claim and completed result live in durable storage. A process-local set disappears on restart and cannot coordinate two workers.
A worker acknowledges a message only after the durable completion record exists. A failed attempt remains eligible for retry or dead-letter handling.
The limiter is shared by every worker calling the same dependency. Ten workers with private six-per-minute limiters can emit 60 calls per minute, which is not a six-per-minute system.

The failure boundary is therefore outside the queue. Imagine that the CRM accepts an update, then the worker loses its connection before persisting completion. Delivery returns. Without an operation key accepted by the CRM, or a transactionally recorded outbox/result that lets the worker prove the effect, the retry can apply twice. No choice among BullMQ, Upstash QStash, Google Cloud Tasks, AWS SQS, or a simple REST queue erases that ambiguity. The business operation needs an idempotency contract.

Fast retries are harmful here.

On HTTP 429, honor Retry-After when the service sends it; otherwise use exponential backoff with jitter. A tight retry loop consumes worker slots, increases queue churn, and attacks the dependency during the exact interval in which it asked for less traffic. I'm not sure what the right retry ceiling is for a given support workflow until its latency objective and observed recovery distribution are known, but the terminal state must be explicit: retryable work moves back with delay, while exhausted work goes to review or a dead-letter queue rather than disappearing.

What should a rate-limited job processing queue and cron each own?

The useful comparison is the ownership model around the queue. Vendor feature matrices change; duplicate semantics and public-network boundaries shape the application for much longer.

Option	Operational fit for this workload	Retry and idempotency consequence	Prefer it when	Avoid it when
BullMQ	Queue plus separately operated workers	The consumer still needs a durable operation key	The team already wants to run and observe the queue and workers	A managed HTTP delivery boundary is the main requirement
Upstash QStash	Managed queue-shaped option for invoking work	The invoked handler must remain idempotent across retries	Public HTTP handlers fit the deployment model	The consumer must remain private
Google Cloud Tasks	Managed task delivery option	Rate controls do not remove the need for an idempotent business effect	The workload already sits inside the Google Cloud operating model	Cross-platform simplicity matters more than cloud alignment
AWS SQS	Standard queue option with dead-letter queue support	Standard delivery is at least once, so duplicates are part of normal operation	AWS operations and separate worker ownership are acceptable	The design requires Kafka-style replay or multiple consumer groups
Infrai	Plain REST queue under the same key and bill as its other backend capabilities	Standard queues are at least once; the consumer must be idempotent	One credential and one invoice reduce cross-service administration, and an SDK-free HTTP boundary suits mixed-language workers	DAG orchestration, fanout/join, private push targets, or replay is required
Cron without a queue	Periodic trigger only	A retry reruns the scheduled handler unless the handler builds queue-like state itself	The task is short, bounded, and naturally periodic	Backlogs, per-minute throttling, or long jobs are possible

The Infrai row has a real administrative advantage rather than a throughput claim: one key and one bill can cover backend services, while its consistent REST surface avoids installing a vendor SDK in every worker language. The catch is that this is still a queue, not Temporal or Airflow. It has no DAG orchestration or fanout/join primitive, and there is no topic that sends one message to many independent consumers. Use separate queues when downstream processors need isolated rate limits.

For the specific question about the cheapest backend, I would not choose from a unit-price headline. The relevant cost includes worker runtime, duplicate side effects, dead-letter handling, credential rotation, and operator time; no authenticated workload measurement here establishes a universal winner. Measure the actual arrival rate, payload size, retry distribution, and idle-to-busy ratio against current vendor billing before making cost the tie-breaker.

Make the HTTP boundary explicit

The queue boundary should expose the retry behavior in ordinary code. This runnable Python example publishes one customer-support job through Infrai's verified queue route, reads the API base URL and exact request JSON from environment variables so it does not guess at configuration or a field schema, supplies a stable idempotency key, and treats a 4xx response body as useful diagnostic output. Set INFRAI_QUEUE_PUBLISH_JSON to a request document obtained from live discovery for the queue you created.

import json
import os
import random
import time
import urllib.error
import urllib.request


URL = os.environ["INFRAI_API_BASE_URL"].rstrip("/") + "/v1/queue/publish"
MAX_ATTEMPTS = 5


def retry_delay(error: urllib.error.HTTPError, attempt: int) -> float:
    retry_after = error.headers.get("Retry-After")
    if retry_after and retry_after.isdigit():
        return float(retry_after)
    return min(30.0, (2**attempt) + random.random())


def publish() -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    operation_key = os.environ.get(
        "SUPPORT_OPERATION_KEY", "ticket-1842:sync-crm:v3"
    )
    payload = json.loads(os.environ["INFRAI_QUEUE_PUBLISH_JSON"])
    request = urllib.request.Request(
        URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "Idempotency-Key": operation_key,
        },
        method="POST",
    )

    for attempt in range(MAX_ATTEMPTS):
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.loads(response.read())
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == MAX_ATTEMPTS - 1:
                raise RuntimeError(f"queue publish failed: HTTP {error.code}: {body}")
            time.sleep(retry_delay(error, attempt))
    raise RuntimeError("retry budget exhausted")


print(json.dumps(publish(), indent=2))

There is a deliberate limit to this compact example: it demonstrates safe publication, not consumption. For concurrent production workers, use a claim state with leases and fencing, a transactional outbox, or upstream idempotency, then test the crash point between remote acceptance and local commit. Don't confuse an idempotent publish with end-to-end exactly-once execution.

With a REST queue, the surrounding worker loop should consume, call process, and acknowledge only after success. If the dependency returns 429, compute the next attempt from Retry-After or exponential backoff and leave the message unacknowledged or negatively acknowledge it according to the selected queue's contract. That adapter is product-specific; the invariant is not.

Draw the capacity envelope before adding workers

Start with arithmetic, because vague promises about autoscaling do not override a quota. At 30 permitted calls per minute and an arrival rate of 45 jobs per minute, backlog grows by 15 every minute until arrivals fall or capacity changes. Adding consumers makes polling and local work faster, but it cannot raise the dependency's permitted rate. Alert on oldest-message age, not just queue depth: 500 tiny jobs may be harmless at one quota and a customer-visible delay at another.

Payload and time bounds also decide whether this design fits. Delayed queue messages must stay within seven days, bodies within 256KB, and retention within 30 days. Store large ticket transcripts in object storage and enqueue a private reference rather than the transcript itself. Acknowledgment deletes the message, so this is not Kafka-style replay and does not supply multiple consumer groups. If audit replay is a requirement, keep an immutable event record elsewhere.

Cron has its own hard edge. A run may last no more than 900 seconds, paused schedules do not backfill missed triggers, execution can have seconds of jitter, and retained output is limited to the first 4KB. It can call only a public http_url; a push subscription likewise needs a public HTTPS target. Those constraints make "cron triggers queue, worker drains queue" the stable shape for long customer-support batches, but not suitable for a private-only consumer unless the platform provides a separate pull-worker path.

There is also no native debounce or throttle. Worker code must enforce the shared limit, and coalescing repeated ticket updates requires application state keyed by the ticket and operation. FIFO deduplication covers only a five-minute window; it cannot replace a durable business idempotency record.

Record the rejected architecture and its valid use case

The rejected design is a cron handler that scans every pending support job and processes the entire batch inline. It is easy to sketch and difficult to bound: one slow dependency stretches the run, retries compete with new work, a crash obscures which effects completed, and the 900-second ceiling eventually turns backlog size into a scheduling failure. It also gives no native throttle or debounce.

Still, stick with cron alone when the operation is short, bounded below that ceiling, naturally periodic, and idempotent as a whole; a daily request to refresh one compact cache can be clearer than introducing a queue. Stick with Temporal or Airflow when the real requirement is a DAG, durable orchestration, fanout followed by join, or multi-step compensation. Choose a replay-oriented log when several consumer groups must independently reprocess retained events. These are different problems, and forcing them through a simple job queue usually hides state rather than removing it.

For the customer-support worker pool described here, the decision remains queue first, cron as an optional producer, and durable idempotency before concurrency. Tune the limiter only after that ordering is true.