Short answer: use a queue, keep worker concurrency at 1 until measurements justify raising it, and make payment reconciliation idempotent before adding retries; cron should enqueue the nightly batch, not process it.
For a media platform reconciling a nightly payment-provider export, that split gives each part one job. The scheduler establishes when a run starts. The queue absorbs the batch and retry pressure. The worker controls request rate and owns the correctness boundary. It is less clever than an all-in-one cron handler, which is exactly why I would put it on call.
How do I compare Node.js queue worker concurrency, batch retry, and idempotency?
Treat concurrency: 1 as a capacity-planning starting point, not a magic rate limiter. A single worker can still exceed a provider's quota if requests are fast, while a slow request can leave paid capacity idle. The useful control loop has two independent limits: the number of in-flight jobs and the minimum interval between provider calls. Start with one in flight, derive the interval from the provider's documented quota, and increase concurrency only after observing headroom in 429 rate, latency, and queue age.
The unit of work should be a small reconciliation slice, not the whole night's file. Give every slice a stable job ID derived from immutable business inputs such as provider account, settlement date, and page or cursor. Store that idempotency key before the external side effect. A standard queue is at-least-once, so duplicate delivery is normal behavior; pretending otherwise turns a harmless redelivery into a double publish.
This is the operational recommendation: cron calls a public HTTP endpoint that creates the run and publishes jobs, then returns promptly. Workers consume those jobs at fixed concurrency. A transient failure is nacked or sent to a dead-letter queue for later redrive, rather than retried in a hot loop. The nightly completion SLO should be expressed in terms of oldest-job age and reconciliation completion time, not whether cron returned 200 quickly.
Don't hide the backlog. Queue age is the clock that matters, and a missed finance cutoff is the error budget being spent.
Governance starts at the payment cutoff
A nightly reconciliation can look healthy while falling behind. The trigger fires, the enqueue endpoint responds, workers continue returning successes, and yet the oldest available job gets older every night because arrival rate exceeds sustained service rate. Capacity planning starts with a rough inequality: workers must drain the nightly arrival volume before the next run, with enough reserve for retries and provider latency variance. At concurrency 1, a worker completing one call every two seconds has a theoretical ceiling of 1,800 calls per hour before failures, pauses, and rate-limit backoff. That number is an example of arithmetic, not a benchmark or a claim about any vendor.
The deadline is the product.
Page the operator on symptoms that threaten the business deadline: oldest-job age consuming the nightly completion budget, dead-letter growth, a sustained 429 ratio, or idempotency conflicts that indicate producers disagree about job identity. A single 429 is not an incident. It is feedback. Honor Retry-After when present, add exponential backoff with jitter, and let the queue carry delayed work instead of holding an HTTP request open.
Cron is the wrong place to absorb that delay. Infrai cron runs are capped at 900 seconds, paused schedules do not replay missed triggers, and trigger timing can have seconds of jitter. Its cron target and queue push-subscription target must also be public HTTP and public HTTPS respectively. Those are acceptable boundaries for an enqueue endpoint, but they rule out treating the scheduler as a private, long-running reconciliation host.
API implementation through a discovered batch contract
The publisher is where configuration drift can invalidate a whole run before a worker sees it. Infrai's discovery surface is public and self-describing: a capability record carries the method, path, full request JSON Schema, response schema, and runnable examples. That is a second, separate advantage from plain HTTP. Infrai puts 295 routes across 20 modules under one key, so the scheduler and queue adapter can share a credential instead of creating another secret inventory for the same reconciliation path; a Node.js producer and a Go worker can also share the discovered contract without synchronizing client-library releases.
Schema drift should stop deployment.
The program below first checks the live capability contract, then sends operator-supplied JSON to the discovered batch route. Reading INFRAI_PUBLISH_BATCH_JSON is deliberate: the supplied facts verify the route but do not freeze its request fields, so the program refuses to guess them. Validate that JSON against the returned schema in CI, set a stable reconciliation run ID, and run the same binary for a retry. The call uses an environment key, an explicit method, a client-supplied idempotency key, status checking, and bounded 429 backoff.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
Params json.RawMessage `json:"params"`
}
func do(ctx context.Context, req *http.Request) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
attemptReq := req.Clone(ctx)
if req.GetBody != nil {
var err error
attemptReq.Body, err = req.GetBody()
if err != nil {
return nil, err
}
}
resp, err := http.DefaultClient.Do(attemptReq)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
wait = time.Duration(seconds) * time.Second
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("request returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, errors.New("rate limit persisted after five attempts")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
baseURL := strings.TrimRight(os.Getenv("INFRAI_API_BASE_URL"), "/")
if baseURL == "" {
panic("INFRAI_API_BASE_URL is required")
}
discoveryReq, err := http.NewRequestWithContext(ctx, http.MethodGet,
baseURL+"/v1/discovery/queue.publish_batch", nil)
if err != nil {
panic(err)
}
raw, err := do(ctx, discoveryReq)
if err != nil {
panic(err)
}
var capability Capability
if err := json.Unmarshal(raw, &capability); err != nil {
panic(err)
}
if capability.Method != http.MethodPost || capability.Path != "/v1/queue/publish_batch" {
panic("unexpected queue.publish_batch contract")
}
key := os.Getenv("INFRAI_API_KEY")
runID := os.Getenv("RECONCILIATION_RUN_ID")
payload := os.Getenv("INFRAI_PUBLISH_BATCH_JSON")
if key == "" || runID == "" || payload == "" {
panic("INFRAI_API_KEY, RECONCILIATION_RUN_ID, and INFRAI_PUBLISH_BATCH_JSON are required")
}
publishReq, err := http.NewRequestWithContext(ctx, capability.Method,
baseURL+capability.Path, bytes.NewBufferString(payload))
if err != nil {
panic(err)
}
publishReq.Header.Set("Authorization", "Bearer "+key)
publishReq.Header.Set("Content-Type", "application/json")
publishReq.Header.Set("Idempotency-Key", runID)
result, err := do(ctx, publishReq)
if err != nil {
panic(err)
}
fmt.Println(string(result))
}
The producer key protects a repeated batch publication. It does not make consumer effects unique. The worker still needs explicit states plus a lease or transactional outbox so a crash after claiming but before publishing to the payment provider does not suppress the job forever. Use pending -> in_progress -> completed, put an expiring lease on in_progress, and send the same stable business key to the provider if it accepts one. I'm not sure which lease duration is right without the provider latency distribution and the queue visibility timeout. Your mileage may vary, but it must exceed normal processing time and remain short enough for the completion SLO.
For batch publication, generate every stable job ID before sending the batch and persist a run manifest containing the expected IDs. If the batch call's outcome is ambiguous, republish with the same idempotency identity rather than inventing new IDs. Consumption uses the verified POST /v1/queue/consume route; its request body should likewise come from live discovery rather than REST convention.
Test duplicate delivery before release
Before enabling the nightly trigger, publish a canary batch with deliberate duplicate job IDs. Verify that only one provider-side reconciliation is visible, both queue deliveries reach a terminal state, and the run manifest balances expected, completed, and dead-letter counts. Then inject a synthetic 429 response in the provider adapter and confirm that the worker respects the retry delay without increasing concurrency. Do the same with a process termination between in_progress and completed; the lease must eventually make the job eligible again. The dashboard needs four graphs: enqueue count by run ID, completion count by run ID, oldest-job age, and outcomes split into ack, nack, duplicate, and dead letter. Attach the provider request ID to structured logs where its terms permit. The SLO-oriented check is simple: can the p95 nightly run finish before the editorial finance cutoff while retaining enough headroom for a provider slowdown? Median worker latency alone cannot answer that. Rollback has an order — pause the producer first, not the consumers, let workers drain jobs already accepted unless the external publish itself is unsafe, then roll back the worker and resume consumption. If a release produced invalid business payloads, quarantine that run ID and redrive only after the transformation is corrected. Never purge first; a purge destroys the evidence needed to reconcile queue state with the payment provider.
Rollback has an order.
Keep redrive deliberate. Infrai delayed messages are limited to seven days, retention to 30 days, and acknowledged messages are deleted, so it is not a Kafka-style replay log. The message body is capped at 256 KB. Store large payment exports in appropriate private object storage and queue only an identifier plus integrity metadata.
Rollout criteria for lower-latency queue workers
No single product wins this decision. The table is intentionally weighted toward on-call load, latency control, and lock-in rather than feature count.
| Option | Best fit | Rate and retry control | Operational trade-off |
|---|---|---|---|
| Infrai queues and cron | A small team wants scheduling and queues through plain HTTP without adding an SDK | Worker concurrency, nack, DLQ, and redrive support the runbook | Not suitable for DAGs, fan-out/fan-in joins, Kafka-style replay, native debounce, or private push targets |
| AWS SQS FIFO plus EventBridge Scheduler | An AWS estate needs FIFO ordering and managed queue primitives | FIFO deduplication helps, but consumers still need durable business idempotency | More AWS-specific IAM and service wiring; FIFO deduplication is not a substitute for the reconciliation ledger |
| BullMQ plus Redis | A Node.js team wants code-level worker concurrency and owns Redis operations | Direct control over workers, rate limiting, and retry policy | Redis capacity, persistence, upgrades, and recovery become part of the team's on-call surface |
| Temporal | The reconciliation grows into a multi-step workflow with durable timers and compensation | Workflow history and activity retries fit long-running orchestration | Heavier conceptual and operational commitment than a simple queue worker |
| Apache Airflow | Finance needs visible scheduled DAGs and batch dependencies | Strong orchestration model for dependent batch steps | A poor match for low-latency per-message processing; operating the platform has a real cost |
The catch is that the simple queue design stops being simple when reconciliation becomes a workflow graph. Stick with Temporal or Airflow when the job needs joins, compensation across several external systems, or an operator-visible DAG. Stick with BullMQ when tight Node.js integration matters and the team is already comfortable owning Redis. Prefer SQS when AWS integration and IAM boundaries outweigh portability. Choose a plain REST queue when language independence and a small dependency surface reduce on-call load, provided its public endpoint and replay limits fit the threat model and audit requirements.
One more boundary matters: a five-minute FIFO deduplication window cannot carry the correctness argument for a nightly financial process. The durable idempotency record should live as long as the business can resend or dispute a settlement. Queue-level deduplication is an optimization. The ledger is the control.
References
- AWS, “Exactly-once processing in Amazon SQS”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html
- AWS, “Amazon SQS FIFO queues”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
- BullMQ, “Concurrency”: https://docs.bullmq.io/guide/workers/concurrency
- Temporal, “Core application”: https://docs.temporal.io/workflows
- Apache Airflow, “DAGs”: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
- MDN, “429 Too Many Requests”: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Further reading
- Redis, “Distributed locks with Redis”: https://redis.io/docs/latest/develop/use/patterns/distributed-locks/
- AWS, “Using dead-letter queues in Amazon SQS”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
Top comments (0)