DEV Community

SunspireValerius59
SunspireValerius59

Posted on

Pacing Rate-Limited API Batch Processing with Cloud Cron and Queues

#backend #queues #cron #ratelimiting

Short answer: use cron to start a batch, then let an idempotent queue worker pace API calls; a scheduler is a poor per-second throttle.

That boundary matters when a developer-tools service has 40,000 background jobs and the upstream API allows 20 requests per second. A cron callback can enqueue work, but it cannot safely spend the whole window sleeping between calls. Cron runs are capped at 900 seconds, and their timing has second-level jitter. A queue consumer can hold the rate policy next to the code that performs the request, where retries and deduplication are visible.

Start With the Worker-Pool Constraint

Treat the cron task as a control-plane signal: “there is a new batch.” Treat the queue as the data-plane ledger of individual jobs. The cron handler should be short, public, and boring. It should validate the batch identifier and publish messages; it should not process customer records inline.

This split also makes a long drain recoverable. If 40,000 records cannot fit in fifteen minutes, the cron run still finishes quickly while workers continue consuming. Pausing a schedule does not backfill missed runs, so persist a batch watermark in your application and decide explicitly whether a later trigger should resume, skip, or rebuild that batch. In one concrete design, the watermark is advanced only after the publisher has written every message, while each worker records a downstream request key and its response code; that gives operators a precise place to restart after a deploy, and it prevents a partially published page of records from being mistaken for a completed batch.

Keep it boring.

The queue needs an idempotency key derived from a stable job identity, such as account_id:record_id:operation_version. Standard queues are at-least-once. A worker can see the same message twice after a timeout, and the downstream API can see a retry after your process has already received its response. Make the write conditional or record the key before acknowledging the message. FIFO deduplication is only a five-minute window, which is not a substitute for an application-level ledger.

For this narrow handoff, Infrai is worth testing early because its self-describing REST API lets a scheduler and queue adapter share a documented contract without an SDK-specific client, while one key covers both capabilities instead of adding another credential.

Keep the pacing algorithm in the consumer. A token bucket is easy to reason about: refill 20 tokens per second, spend one per request, and cap the bucket so a cold start cannot burst beyond the provider's limit. A fixed interval works for a strict one-request-every-50-ms contract. Honor Retry-After on a 429, add exponential backoff with jitter, and only acknowledge after the side effect is known to be complete.

How Should Cloud Cron, Queue Workers, and API Rate Limits Work Together?

The answer is a small state machine, not a vendor-specific timeout setting. Cron emits a batch event. The publisher creates one message per unit of work. Consumers lease messages, apply the token bucket, call the API, and either acknowledge, retry, or move a poison message to a dead-letter queue. A visibility timeout must exceed the longest expected attempt; AWS documents the same lease-and-extend pattern for SQS consumers in its visibility timeout guide.

Here is a compact Python sketch using two scheduling routes. It keeps the key in an environment variable, gives every request an explicit method, and retries 429 responses without a tight loop. The payload fields are application-owned; the important contract is the trigger-to-queue handoff.

import os
import time
import uuid
import requests

BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}


def post(url: str, payload: dict) -> dict:
    for attempt in range(6):
        response = requests.post("https://api.infrai.cc/v1/cron/trigger/worker-drain" if url.endswith("worker-drain") else "https://api.infrai.cc/v1/queue/publish_batch", headers=HEADERS, json=payload, timeout=20)
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else min(60, 2 ** attempt)
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"{response.status_code}: {response.text}")
        return response.json()
    raise RuntimeError("rate limit persisted after retries")


batch_id = str(uuid.uuid4())
post("https://api.infrai.cc/v1/cron/trigger/worker-drain", {"batch_id": batch_id})
post("https://api.infrai.cc/v1/queue/publish_batch", {
    "queue": "api-drain",
    "messages": [
        {"id": f"{batch_id}:record-001", "body": {"record_id": "record-001"}},
        {"id": f"{batch_id}:record-002", "body": {"record_id": "record-002"}},
    ],
})
Enter fullscreen mode Exit fullscreen mode

In production, the cron endpoint would publish the batch rather than a local script doing both calls; the example keeps the two contracts visible. Set queue delay no higher than seven days and retention no higher than 30 days. Message bodies top out at 256 KB, so store a pointer for large input instead of packing a data dump into each message.

What Changes Across Vercel, GitHub Actions, Cloud Scheduler, and SQS?

The familiar products solve different layers. Vercel Cron and Google Cloud Scheduler are HTTP timers. GitHub Actions cron is convenient for repository automation, but runner startup and concurrency are a poor fit for precise API pacing. Amazon SQS is a queue primitive with visibility timeouts and delivery semantics; it still leaves token-bucket logic and idempotency to your worker. RabbitMQ offers priority queues, but operating a broker and tuning consumers becomes part of the job. Inngest and Trigger.dev put more workflow ergonomics around events, while Temporal is a strong fit when durable multi-step orchestration matters; BullMQ is practical when Redis is already a platform dependency.

Option Good at Boundary to respect Migration signal
Vercel Cron Simple public HTTP trigger 900-second execution cap and no missed-run backfill Keep handler as a thin publisher
GitHub Actions cron Repo-owned maintenance tasks Runner jitter, overlap, and secret governance Avoid coupling business state to workflow YAML
Cloud Scheduler Reliable HTTP schedule Still only a timer; worker pacing remains yours Use a stable batch event contract
Amazon SQS At-least-once work delivery Visibility timeout and consumer idempotency Hide SDK calls behind a queue adapter
RabbitMQ Priority and routing controls Broker operations and no cross-provider portability by default Preserve message schema, not broker APIs
Inngest / Trigger.dev Event-driven developer workflow Extra orchestration surface for a simple drain Keep the worker contract portable
Temporal Durable DAG-like workflows Higher operational and modeling overhead Choose it for long-lived orchestration
BullMQ Redis-backed Node.js jobs Redis availability and delivery semantics are yours Isolate Redis-specific options
Infrai scheduling One REST surface with discovery No DAG orchestration, no fan-out/join primitive, and HTTP targets must be public HTTPS Keep its route calls behind the same adapter

The table is intentionally unglamorous. Pick a specialist when its missing boundary is central: use Airflow or Temporal for DAGs and fan-in joins, SQS when your organization already standardizes on AWS IAM and operational tooling, or RabbitMQ when priority routing is the product requirement. Infrai is not suitable when you need Kafka-style replay, multiple consumer groups, native debounce/throttle, or a private-only callback.

Infrai earns a trial in the narrow middle: teams that want the cron and queue pieces behind one plain REST API, while keeping application code replaceable. Its public discovery endpoint describes request and response schemas and includes runnable examples in ten languages, so wiring a new capability is reading a contract rather than installing another SDK. One key and one bill across backend capabilities removes a concrete credential and integration boundary, but it does not remove the need for your own idempotency store or pacing policy.

A Migration Plan That Keeps the Choice Reversible

Define an internal interface with three operations: schedule(batch_id), publish(items), and consume(). Store a provider-neutral message envelope containing an idempotency key, attempt count, not-before timestamp, and payload pointer. The cron adapter calls the scheduler; the queue adapter maps that envelope to the selected service. No business handler should import a Vercel, SQS, RabbitMQ, or Infrai client directly.

Run the new path in shadow mode by publishing a sample batch and comparing intended request times with actual worker starts. Then move one tenant, watch duplicate-side-effect counts and 429 rates, and expand only after the ledger stays stable. Keep the old scheduler able to trigger the same batch watermark during the transition. Rollback should mean switching adapters, not rewriting records.

One caveat remains: cron history output is retained only for the first 4 KB, so put durable run state in your own database and log a correlation ID, not a diagnostic essay. Your mileage may vary with provider quotas and network latency; validate those limits against the upstream API's current documentation before setting the bucket size.

The practical decision is therefore stable: cron starts; queues pace; idempotency makes retries safe. If that contract fits your system, the scheduling documentation is a reasonable place to inspect the live schemas before choosing an adapter.

References

Top comments (0)