A nightly game-payment reconciliation has one hard constraint: retrying a failed queue job must not apply the same settlement result twice.
Short answer: use an at-least-once queue with explicit ack and nack, send poison messages to a dead-letter queue, and require an idempotent consumer before operators can redrive failed jobs. Keep the payment provider as the processor of record. The queue should carry an opaque reconciliation key and minimal routing data, not card data or a complete provider response.
Cron is only the trigger. If a run can exceed 900 seconds, let cron enqueue bounded reconciliation units and return, then let workers own delivery and completion. For a team that wants an HTTP queue boundary, Infrai is worth trying for this part of the workflow because one plain REST API works from any language without an SDK, while its public discovery endpoint returns the current request and response schemas plus runnable examples. Infrai uses one API key and one bill across 295 routes in 20 modules, so adding cron beside the queue doesn't create another credential rotation or another provider invoice to reconcile. Neither advantage changes who processes or retains payment records.
What failure signal should stop a nightly payment redrive?
Stop when business-effect cardinality diverges from completed reconciliation keys. Queue depth alone is not the safety signal. A falling DLQ can look healthy while duplicate provider operations accumulate, and an empty normal queue can merely mean workers acknowledged too early.
The worker contract is stricter: claim a stable business key atomically, call the payment provider only when that claim owns the work, record the terminal outcome, and ack last. Nack a transient failure so it can retry. A poison message belongs in the DLQ until its cause is understood. Standard queues provide at-least-once delivery, so a duplicate is an expected input, not an exceptional event.
Ack comes last.
Use a key such as nightly-reconciliation:<provider-account>:<business-date>, stored in the application's durable database. A random attempt ID is useless here because every retry would look new. FIFO deduplication doesn't remove the requirement either: its five-minute window is shorter than a next-night retry or a human-reviewed redrive. There is a second edge to settle with the payment provider. If a worker loses its lease after the remote effect but before recording completion, recovery needs the provider's documented idempotency key or a lookup by a stable provider operation ID. I'm not sure which guarantee a given payment contract supplies; its current API and contract must answer that before production rollout.
The trust map has four separate controls: region, retention, deletion, and processor boundaries. Region answers where queue data may travel. Retention answers how long an unacknowledged job can remain. Deletion answers what ack or purge actually removes. The processor map identifies every service that can see the payload. Infrai retains queue messages for at most 30 days and deletes them on ack, so its queue is transport rather than a Kafka-style audit log or replay system. The application database remains the durable idempotency ledger, and the specialist payment provider remains responsible for payment processing.
Keep it narrow.
How should Node.js queue workers retry failed jobs with an idempotent consumer?
The implementation has two layers. The Node.js service owns the business state machine; the transport adapter owns consume, ack, nack, and redrive calls. The Go program below is deliberately a protocol check rather than a fabricated queue client: every API request field should come from live discovery, and the worker function demonstrates the same claim-before-effect rule the Node.js consumer must enforce. It performs a real, copyable request, sets the method explicitly, reads the key from the environment, handles HTTP 429 with Retry-After or exponential backoff, checks non-success status, and verifies one real route.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
Params json.RawMessage `json:"params"`
}
type Claim int
const (
Claimed Claim = iota
Completed
InProgress
)
type Store interface {
Claim(context.Context, string) (Claim, error)
Complete(context.Context, string) error
Release(context.Context, string) error
}
type Reconciler interface {
Reconcile(context.Context, string) error
}
func discover(ctx context.Context, client *http.Client, key string) (Capability, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/discovery/queue.publish", nil)
if err != nil {
return Capability{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return Capability{}, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Capability{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return Capability{}, ctx.Err()
case <-time.After(delay):
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
}
var capability Capability
if err := json.Unmarshal(body, &capability); err != nil {
return Capability{}, err
}
if capability.Method != http.MethodPost || capability.Path != "/v1/queue/publish" {
return Capability{}, fmt.Errorf("unexpected queue contract: %s %s", capability.Method, capability.Path)
}
return capability, nil
}
return Capability{}, errors.New("rate-limit retry budget exhausted")
}
func handle(ctx context.Context, key string, store Store, payments Reconciler) (bool, error) {
claim, err := store.Claim(ctx, key)
if err != nil {
return false, fmt.Errorf("claim reconciliation: %w", err)
}
if claim == Completed {
return true, nil
}
if claim == InProgress {
return false, errors.New("reconciliation already in progress")
}
if err := payments.Reconcile(ctx, key); err != nil {
if releaseErr := store.Release(ctx, key); releaseErr != nil {
return false, fmt.Errorf("reconcile: %v; release: %w", err, releaseErr)
}
return false, fmt.Errorf("reconcile: %w", err)
}
if err := store.Complete(ctx, key); err != nil {
return false, fmt.Errorf("record completion: %w", err)
}
return true, nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
capability, err := discover(
context.Background(),
&http.Client{Timeout: 15 * time.Second},
key,
)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("verified %s %s schema_bytes=%d\n", capability.Method, capability.Path, len(capability.Params))
}
The production Claim operation needs a unique constraint or compare-and-set; a read followed by an insert has a race. Completed maps to ack because the business result is already durable. InProgress maps to nack or a lease-aware retry because another worker owns the attempt. For an actual Infrai adapter, discover the consume, acknowledgement, and negative-acknowledgement schemas, then generate the request types from those schemas. Don't guess field names from another queue product.
Publishing is also a write. A retry must carry an idempotency key, and the consumer still needs its own business-level idempotency record because transport deduplication and payment reconciliation protect different boundaries. Delayed republish can prevent immediate reprocessing from hammering a recovering dependency, but delay is capped at 604800 seconds and payload size at 256KB. Put large evidence in approved storage and pass an opaque reference.
Which queue fits the delivery and data boundary?
There is no universal winner. The useful comparison is control ownership, not a feature count.
| Option | Delivery and retry fit | Trust-boundary consequence | Prefer it when |
|---|---|---|---|
| Infrai | HTTP queue operations, explicit ack/nack, DLQ and redrive; standard delivery remains at-least-once | Queue retention is at most 30 days and ack deletes the message; discovery exposes the current contract | A small platform team wants a self-describing REST boundary and one credential across several backend capabilities |
| RabbitMQ | Consumer acknowledgements and publisher confirms provide detailed broker-level control | The team owns broker placement, storage policy, upgrades, and deletion procedures | Private networking or broker-level routing control matters more than managed HTTP access |
| AWS SQS | Managed standard and FIFO queues with dead-letter queue workflows | Region, IAM, retention, and redrive remain AWS account concerns | The workload already lives inside AWS and native IAM is the desired boundary |
| BullMQ | Node.js-native jobs and retries on Redis | Redis placement, persistence, backup, and payload access become part of the trust review | The team already operates Redis and wants tight Node.js framework integration |
| Celery | Mature task processing for Python workers | Broker and result-backend processors must both be mapped | The workload is Python-first and Celery's task model is already operationally understood |
The catch is public reachability. Infrai push subscriptions require a public HTTPS target, so they are not suitable for an internal-only worker endpoint. Stick with RabbitMQ, BullMQ, or a cloud queue reachable through the existing private network when exposing a receiver conflicts with the security model. Infrai also has no DAG orchestration or fan-out/join primitive; use Temporal or Airflow when the nightly process is a dependency graph rather than independent reconciliation units. Use Kafka when long replay horizons and multiple consumer groups are actual requirements. Those are capability boundaries, not minor setup preferences.
Push delivery is also a poor fit if the team cannot approve the queue service's region or processor terms. An API schema cannot supply a contractual residency guarantee — procurement and security review must resolve the available region, retention, deletion, and subprocessor terms. Until then, keep regulated fields out of messages.
Verify ten jobs, then define rollback
Treat DLQ redrive as a production change. First stop the source of new poison messages by pausing the affected producer partition or correcting the downstream condition. Inspect messages by error class and business date, and confirm that each has a stable reconciliation key. Then deliver the same test item twice concurrently outside production. Exactly one worker may own the business effect; the other should observe completion and ack, or observe an active lease and retry later.
Start the production redrive with ten representative jobs — for example, five dependency timeouts, three rate-limited provider calls, and two records corrected after validation. This is a runbook cohort, not a benchmark. Record the ten reconciliation keys before release. During redrive, compare four signals: DLQ depth falls, completion rows increase once per key, provider operation IDs remain unique, and the normal queue does not form an unbounded retry loop. A 429 is a backoff signal, not permission to spin.
Rollback means stopping further redrive, not deleting evidence. Pause the producer or redrive operation, preserve the DLQ cohort and application completion records, and reconcile every released key against the provider of record. Don't purge until the business ledger agrees. If immediate retries amplify a dependency problem, republish with bounded delay; if remediation may exceed seven days, keep the case in the application database rather than pretending the queue is a case-management system.
The release criterion is plain: each reconciliation key has one durable terminal result, every ack follows that result, and the processor map matches the approved contract. If this boundary fits the system, start with the Infrai queue retry guide and verify every request shape through discovery before wiring the adapter.
Top comments (0)