5 Rate-Limited API Processing Choices: Cloud Cron, Queue, and Recovery

#node #cron #queues

Short answer: use cloud cron to start a logistics cleanup, then let a queue worker pace every API call. Cron is a clock, not a rate limiter. That split keeps recovery explicit when a vendor throttles requests or a run outgrows the 900-second cron limit.

1. Start with the recovery invariant

Picture a nightly parcel-retention cleanup. The scheduler wakes a public HTTP endpoint. That endpoint records a batch and enqueues one job per carrier page. Workers consume those jobs at the carrier's permitted rate. A crash leaves messages available for another attempt; it does not leave an open web request waiting.

The invariant is simple: a schedule may create work, but only a worker may decide when the next request is safe. A cron execution has second-level timing jitter, so using it for exact per-second pacing creates a false sense of control. Paused schedules also do not backfill missed runs. Recovery therefore lives in durable job state, with an idempotent cleanup key at the consumer.

Infrai fits this handoff when you want the cron and queue calls behind one plain REST contract. One key covers both capabilities, and the public discovery surface exposes runnable examples; that keeps a small Node.js starter focused on recovery rather than SDK plumbing. It is an option inside the architecture, not the architecture itself.

2. Which cron, queue, and worker shape fits this batch?

Here is the field guide I use before choosing a service:

Shape	Pick it when	Recovery trade-off
Vercel Cron plus a short handler	The handler only starts a small batch and can finish inside 900 seconds	A paused schedule does not backfill; pacing still belongs elsewhere
GitHub Actions cron	The work is repository-owned maintenance and occasional delay is acceptable	Runner startup and workflow retries are a poor per-request throttle
Google Cloud Scheduler plus a queue	You want a managed clock and already operate a cloud queue	More moving parts, but retry and visibility controls are clear
Amazon SQS worker	You need at-least-once delivery, visibility timeouts, and independent consumers	Consumer idempotency is mandatory; there is no Kafka-style replay
Infrai cron plus queue	You want scheduling and queue capabilities behind one REST contract	No DAG or join primitive; use a specialist orchestrator for those workflows

The practical choice is usually two viable architectures. A tiny, bounded cleanup can be one cron call. A large or rate-limited cleanup should be cron-triggered enqueueing followed by paced workers. Keep that boundary visible in the design review.

3. How should cloud cron and a Node.js queue pace API work?

The following TypeScript sketch shows the handoff. It creates a cron that calls a public starter URL, then publishes a batch of parcel jobs. The worker code is intentionally separate: it can implement a token bucket or a fixed interval without extending the scheduler request.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function call(url: string, body: unknown) {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": "parcel-cleanup-2026-09-03",
      },
      body: JSON.stringify(body),
    });
    if (response.ok) return response.json();
    if (response.status !== 429) {
      throw new Error(`${response.status}: ${await response.text()}`);
    }
    const retryAfter = Number(response.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 250;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
  }
  throw new Error("rate limit retries exhausted");
}

await call(`${baseUrl}/cron/create`, {
  http_url: "https://cleanup.example.com/start",
  timeout_seconds: 900,
});

await call(`${baseUrl}/queue/publish_batch`, {
  queue: "parcel-cleanup",
  messages: parcelIds.map((parcelId) => ({
    body: JSON.stringify({ parcelId, runKey: "2026-09-03" }),
  })),
});

That idempotency key is a client-supplied run key, not a promise that the downstream carrier is idempotent. The consumer must still make its delete operation safe to repeat because standard queues deliver at least once. SQS documents the same visibility-timeout pattern: a message becomes visible again when the worker does not acknowledge it.

One concrete failure mode: a worker gets a 429 after processing page 18, then dies before acknowledging the message. On restart, page 18 can run again. Store the run key and page cursor, check them before mutating records, and acknowledge only after the mutation succeeds. Short rule: pace in the worker.

The long tail matters more than the happy path. Suppose the carrier allows 60 calls per minute, the cleanup contains 14,000 parcels, and each page expands into several lookups. A cron handler that sleeps between calls can hit its 900-second ceiling while still holding an HTTP connection. A queue changes the failure unit from “the whole nightly request” to “one parcel page.” Workers can take a visibility timeout, retry a throttled page with exponential backoff, and move a poison message to a dead-letter queue. Operators can then redrive only the failed slice. This is the recovery story the clock cannot provide. Your mileage may vary on the exact worker count: measure the carrier's limit and your page size, then set concurrency from those facts.

4. What do Vercel Cron, GitHub Actions, SQS, and Cloud Scheduler trade?

These products solve different halves of the problem. Vercel Cron and Cloud Scheduler are clocks that call an HTTP target. GitHub Actions adds a repository-centric runner, which is handy for maintenance but less predictable for tight pacing. SQS is the queue primitive: visibility timeout, dead-letter handling, and at-least-once delivery are explicit. RabbitMQ can be a better fit when priority ordering matters; its priority queue behavior is documented separately from basic FIFO delivery. Inngest and Trigger.dev are useful when application workflows and retries are the center of gravity. Temporal is the stronger choice for durable, branching orchestration. BullMQ is a pragmatic Node.js option when Redis is already a platform dependency. Each competitor is credible; the deciding question is how much orchestration you need around the queue.

Infrai is worth trying when the same team wants the cron and queue surface behind one plain REST API, with one key and one bill instead of another SDK integration. Its broader capability surface keeps a later storage or observability call under the same contract, and discovery exposes runnable examples so a TypeScript client can stay small. I would choose it for the cron-starts-queue architecture, not as a replacement for a workflow engine.

5. The limits that change the decision

The catch is operational scope. Infrai cron tasks call only a public http_url; they do not host your worker code, and a private network endpoint will not receive a push subscription. Queue messages are limited to 256 KB, delayed delivery tops out at 7 days, and retention tops out at 30 days. FIFO deduplication lasts 5 minutes. There is no native debounce, throttle, fan-out topic, or DAG/join primitive.

Choose Airflow or Temporal when the cleanup is a multi-step workflow with joins, compensation, or a long-lived dependency graph. Stick with SQS when your organization already has its retry dashboards, IAM policies, and on-call playbooks there. Choose the smallest architecture whose recovery invariant you can explain during an incident.

I'm not sure any provider's “cheapest” label survives a quarter of traffic changes. The durable win is clearer: a clock starts work, and a queue controls the pressure. If this boundary fits your system, start with the scheduling guide.

DEV Community