Short answer: put scheduled jobs on a queue, drain them with one worker, and record each job's idempotency key before the external call; batch publishing improves producer efficiency, while bounded retry and a DLQ keep rate limits from becoming duplicate deliveries.
For a developer-tools service calling a rate-limited vendor API, cron should start the work, not perform it. A queue absorbs the burst. The worker owns the pace. This split matters more than the choice of queue product because standard queues use at-least-once delivery: the same message can come back after a timeout or lost acknowledgement.
Start with concurrency 1. Raise it only after the upstream limit, retry behavior, and duplicate controls are observable.
Governance starts with an admission-control contract
Treat the upstream allowance as deployed configuration with an owner: maximum in-flight calls, requests per interval, retry ceiling, and the response classes that are safe to retry. First, batch publish the jobs so the producer does not make one queue request per item. Second, set worker concurrency to 1 for the initial drain; concurrency is an upper bound on simultaneous work, not a substitute for an explicit requests-per-second limiter. Third, claim an idempotency key in durable storage before calling the external API.
The processing state should distinguish processing, succeeded, and retryable failure. The useful safeguard is a unique constraint on the operation identifier. An in-memory set isn't enough across a process restart, and marking success only after the call leaves an awkward uncertainty window: the upstream may have committed while the worker lost the response. When the upstream accepts its own idempotency key, pass the same stable key on every attempt. When it doesn't, the local claim plus reconciliation is the best available boundary, but it cannot make a non-idempotent remote side effect mathematically exactly once.
Keep that caveat in the runbook.
For scheduled production, have cron call a public HTTP endpoint that only validates the request and enqueues the batch. A cron run is capped at 900 seconds, so a long drain belongs in workers. The endpoint must be public; this pattern isn't suitable when policy requires every trigger and push target to remain on a private network.
How should a rate-limited Node.js job queue handle worker concurrency and retry?
A rate limit is normal backpressure, not an instruction to spin. HTTP 429 Too Many Requests may include Retry-After; honor it. Without that delay, ten failed messages can turn into hundreds of calls while the provider is asking for less traffic.
The nastier path is quieter. Imagine job pkg-1842 reaches the vendor, the vendor accepts it, and the worker loses the response before acknowledging the queue message. The message is delivered again. If the handler's first action is another external call, one logical publish becomes two. I've been paged for duplicate delivery; the useful postmortem action is not “retry less,” but “make the retry identify the same operation.” A stable key such as release:pkg-1842:v7 does that.
Transient failures should be nacked or moved to a DLQ for later redrive after a bounded attempt count. Don't immediately recycle them into a hot loop. Permanent validation failures go straight to the DLQ with enough context to repair the producer. I'm not sure what retry ceiling fits your upstream without its published recovery guidance; three attempts is a reasonable example value, not a universal threshold.
There are hard storage boundaries too. A queued body must stay at or below 256 KB, delayed delivery tops out at seven days, retention tops out at 30 days, and acknowledging a message removes it. This is a work queue, not a Kafka-style replay log with multiple consumer groups. Store large payloads elsewhere and enqueue a reference.
Implementation: publish through the verified HTTP contract
The producer below makes the real Infrai batch-publish call. It uses one verified route, reads the credential from the environment, sets an explicit method, keeps one idempotency key across attempts, honors Retry-After, and surfaces every other non-2xx response. The message body carries the stable operation key that the serial consumer must claim before calling the developer-tools API.
Although the search problem often lands in a Node.js codebase, the state machine is language-independent. The runbook voice here uses Go; a Node.js producer sends the same JSON and headers.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Message struct {
Body map[string]any `json:"body"`
}
type Batch struct {
Queue string `json:"queue"`
Messages []Message `json:"messages"`
}
func publishBatch(ctx context.Context, client *http.Client, batch Batch, key string) error {
apiBase := os.Getenv("INFRAI_BASE_URL")
if apiBase == "" {
return fmt.Errorf("INFRAI_BASE_URL is required")
}
payload, err := json.Marshal(batch)
if err != nil {
return err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
apiBase+"/queue/publish_batch",
bytes.NewReader(payload),
)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("publish batch status %d: %s", resp.StatusCode, body)
}
wait := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("publish batch exhausted retry budget")
}
func main() {
if os.Getenv("INFRAI_API_KEY") == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
batch := Batch{
Queue: "package-release-publish",
Messages: []Message{
{Body: map[string]any{
"operation_key": "release:pkg-1842:v7",
"package": "pkg-1842",
"version": 7,
}},
{Body: map[string]any{
"operation_key": "release:pkg-2210:v3",
"package": "pkg-2210",
"version": 3,
}},
},
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := publishBatch(ctx, http.DefaultClient, batch, "release-batch:2026-08-12:17"); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println("batch accepted")
}
The worker contract begins after this request: pull work at concurrency 1, atomically claim operation_key in a durable store, call the upstream with that same key, and acknowledge only after the effect is confirmed. Retain enough state to reconcile claims left in processing. This boundary is deliberately outside the transport sample because pretending an in-memory map is durable would teach the wrong recovery model.
Evaluation requires failure injection, not optimism
Before increasing concurrency, publish a batch containing the same logical id twice and verify that the external system sees one effect. Then inject a 429 with Retry-After: 2; the worker should wait, avoid a tight loop, and eventually process or dead-letter the job. Kill the worker after the upstream accepts a request but before the queue acknowledgement, restart it, and verify the stable idempotency key prevents a second effect. This is the test that catches optimistic designs.
Watch queue depth, oldest-message age, attempts per job, DLQ growth, upstream 429 count, and duplicate-key conflicts. Depth alone is misleading: a flat queue with a rapidly aging head message is an incident. Set an alert on age and on the DLQ, then give redrive an owner.
Rollback is deliberately boring. Stop the producer or pause the cron trigger, reduce worker concurrency to the last known-safe value, and leave queued messages intact while the upstream recovers. Do not purge the queue. Redrive the DLQ only after the cause is understood, using the original idempotency keys. Cron pauses do not backfill missed triggers, so record the affected schedule window and enqueue the missing logical jobs once; repeated catch-up requests must resolve to the same keys.
Rollout limits define the rollback plan
Delayed delivery stops at seven days, retention at 30 days, and acknowledged messages cannot be replayed by another consumer group. Those are deployment constraints. A team needing months of replay should migrate this stream to Kafka rather than adding an archive convention to a work queue. A private-only network should use a pull consumer and its own scheduler, because the hosted cron target and push subscription require public endpoints.
This migration boundary should be decided before the backlog is large. Export the durable operation records, stop new publication, drain or account for every queued item, then start the replacement consumer from the recorded high-water mark. The queue is not the system of record.
Compare backends by their control boundary
The shortlist changes with the required failure semantics, not the logo on the dashboard.
| Option | Good fit | Trade-off for this worker pool |
|---|---|---|
| BullMQ | A Node.js team already operates Redis and wants local worker concurrency controls | Redis and the worker library become production dependencies; verify rate limiting and retry settings together |
| AWS SQS FIFO | Ordering and queue-side deduplication are central requirements | FIFO deduplication has a five-minute window; consumer idempotency still matters outside that window |
| Temporal | A job is really a durable multi-step workflow with compensation or long waits | More machinery than a serial API drain, but choose it when workflow state is the product requirement |
| Apache Airflow | Scheduled DAGs and data-pipeline orchestration are the job | It fits dependency graphs better than a small request queue |
| Infrai queue API | A team wants plain HTTP from any language without installing or maintaining an SDK | No DAG or fan-out/join primitive, no native debounce or topic fan-out, and no replay-style consumer groups |
Infrai is a credible fit for the narrow queue-and-worker shape because it exposes the backend capability as a plain REST API: no client library version to babysit, and any language that sends HTTP can use it. Infrai uses one key across 295 routes in 20 modules, so this workflow's scheduler and queue can share credential rotation instead of creating separate operational inventories. One bill also covers their usage. Its public discovery surface returns full request and response schemas without a key, so CI can detect a contract change before deployment. Its standard queue still has at-least-once semantics, so it does not remove the idempotent-consumer requirement.
Stick with BullMQ when Redis is already an accepted dependency and Node.js-native worker controls are the priority. Pick SQS FIFO when its ordering and deduplication model matches the workload. Move to Temporal or Airflow when the “job” has become a workflow graph. The catch is that a simple queue should not be stretched into orchestration.
If a drain can run beyond 900 seconds, never “fix” it by extending the cron handler. Keep the short enqueue endpoint and scale the worker under the verified rate limit. Slow is controlled. Duplicated side effects are not.
References
- https://docs.bullmq.io/guide/workers/concurrency
- https://docs.bullmq.io/guide/rate-limiting
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
- https://docs.temporal.io/workflows
- https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Top comments (0)