Short answer: for a B2B SaaS shipment update that must reach many subscribers, use the scheduler only to create a durable delivery batch, then let a background job queue own per-subscriber retries and dead-letter decisions. The queue is not the guarantee. An idempotency key, an append-only delivery record, and a deliberate acknowledgment boundary are the guarantee.
This is the architecture decision I would record for a scheduled shipment fan-out. The scheduler establishes intent. A publisher creates one command per subscriber. A worker applies the command, writes the result to an audit trail, and acknowledges only after that durable state is settled. The transport may deliver a command more than once; the business effect must remain safe to repeat.
That distinction is especially important in payment and ledger systems, where “exactly once” is usually an effect we construct from at-least-once delivery, idempotent writes, and reconciliation. It is not a property to assume because a queue has a reassuring name.
A duplicate shipment update is a normal outcome
Start by naming the invariant. For shipment shp_1842, subscriber warehouse-eu, and schedule run 2026-08-10T02:00Z, the command identity can be shipment:shp_1842:warehouse-eu:2026-08-10T02:00Z. A retry reuses that identity. A redrive reuses it too. Creating a fresh identifier during redrive turns a recovery action into a second business event.
The application-owned delivery record needs at least these transitions:
-
planned: the schedule selected the shipment and subscriber. -
published: a command with the stable identity was accepted by the queue. -
applied: the subscriber endpoint accepted the update, or the local side effect committed. -
retryable: the attempt failed in a way that may change. -
dead_lettered: bounded attempts ended, or policy classified the message as poison. -
reconciled: an operator or automated process recorded the final disposition.
The state machine matters more than the library. A timeout after an HTTP request is not proof that the subscriber did not receive the update. A lost acknowledgment is not proof that the worker did not commit it. In both cases, the next attempt must consult the delivery record or a subscriber-side idempotency contract before doing anything irreversible.
Keep the shipment payload small and immutable. Put the shipment ID, subscriber ID, event version, schedule run ID, and policy version in the command; keep the authoritative payload in storage if it can change or contain sensitive fields. That gives operators a useful audit trail without making a dead-letter queue the system of record. Compliance retention is a policy decision based on jurisdiction and data class, so the delivery record should retain the policy version that governed its lifecycle rather than hard-code a universal period.
The ledger records intent, publication, and effect
There are four boundaries worth testing separately.
First, the scheduler can select work and crash before publishing all commands. A run table must make the selection reproducible and expose the difference between “not selected,” “selected but not published,” and “published.” A periodic reconciliation pass can safely republish the original command identity.
Second, publication can succeed while the publisher loses its response. A publish operation therefore needs a deterministic idempotency key, and the queue adapter must define what a repeated publish means. If the adapter cannot provide that contract, the application needs a durable outbox with a uniqueness constraint.
Third, the worker can commit a subscriber update and lose its acknowledgment. This is the classic duplicate-delivery case. The worker should make the subscriber operation idempotent, or record a compare-and-set version such as shipment_version = 18 so a later delivery of version 18 becomes a no-op. Don't use an in-memory mutex as the only protection; it vanishes on restart and does not coordinate workers.
Fourth, a subscriber can return an error that has the wrong operational meaning. A temporary timeout deserves bounded retry. A malformed destination or an authorization failure may be a poison message. A 429 is a signal to respect the receiver's rate policy, not an invitation to spin. The classification should be explicit, observable, and covered by tests.
Three words: classify before retry.
For a concrete fan-out, suppose a tenant has 12,000 subscribers and one shipment update carries event version 18. The scheduler should create a batch record and partition it into commands; it should not hold a 12,000-item in-memory list until every delivery finishes. A worker may process a bounded slice, persist attempt number and next eligible time, then release the message. If the process dies between the external request and the local write, reconciliation compares the subscriber's recorded version with the command version. That's slower than a cheerful fire-and-forget loop, but it gives a financial operator an answer that can be defended.
What should a Node.js background job queue guarantee for scheduled shipment retries?
A small setup is not the one with the fewest moving parts. It's the one whose failure boundaries the team can inspect during an incident.
| Control plane | Good fit | Boundary to accept |
|---|---|---|
| Scheduler plus application outbox | The schedule selects bounded batches and the application owns auditability | You operate the outbox poller and reconciliation job |
| Queue plus worker pool | Each subscriber delivery can be retried independently | You must define visibility, acknowledgment, and dead-letter policy |
| Workflow engine | Delivery includes long waits, branching, or compensation | More orchestration state and operational concepts than a simple fan-out needs |
| Stream platform | Replay and several independent consumer groups are first-class requirements | Acknowledgment alone does not express business completion |
| Repository scheduler | Maintenance is repository-scoped and occasional | Per-subscriber delivery state and triage become awkward |
For this scenario, I would choose a scheduler, an outbox, a queue, and workers, with a delivery ledger beside them. The outbox closes the scheduler-to-queue gap; the ledger closes the queue-to-business-effect gap. Those are separate responsibilities, even if a single deployment owns both.
The rejected option is a scheduled handler that loops over every subscriber and retries inline. It is valid for a bounded, transactional maintenance task that can be rerun as one unit and does not need per-recipient visibility. It is unsuitable when one slow or invalid subscriber must not delay the rest, when operators need selective redrive, or when the run exceeds the execution window of the scheduler. Fewer components are useful only when they do not erase the recovery boundary.
Compare control planes by operational ownership
The following code isolates the business contract from a Node.js queue client or any particular broker. A Node.js service can implement the same interfaces around its queue adapter. The important sequence is claim, apply, record, and acknowledge; the adapter must acknowledge only when Handle returns nil.
package main
import (
"context"
"errors"
"fmt"
)
var ErrApplied = errors.New("delivery already applied")
type Delivery struct {
ID string
ShipmentID string
SubscriberID string
EventVersion int64
ScheduleRunID string
}
type DeliveryResult struct {
Status string
}
type DeliveryStore interface {
ClaimAndApply(context.Context, Delivery) (DeliveryResult, error)
RecordFailure(context.Context, string, string) error
}
type Worker struct {
Store DeliveryStore
}
func (w Worker) Handle(ctx context.Context, d Delivery) error {
if d.ID == "" || d.ShipmentID == "" || d.SubscriberID == "" {
return errors.New("invalid delivery command")
}
result, err := w.Store.ClaimAndApply(ctx, d)
if errors.Is(err, ErrApplied) {
return nil
}
if err != nil {
if auditErr := w.Store.RecordFailure(ctx, d.ID, err.Error()); auditErr != nil {
return fmt.Errorf("delivery %s failed: %v; audit failed: %w", d.ID, err, auditErr)
}
return fmt.Errorf("delivery %s: %w", d.ID, err)
}
fmt.Printf("delivery=%s status=%s\n", d.ID, result.Status)
return nil
}
func main() {
fmt.Println("acknowledge only after Worker.Handle returns nil")
}
In a real implementation, ClaimAndApply would use a database transaction or an equivalent compare-and-set protocol. The uniqueness key should cover the business identity, not merely the queue message ID. A message can be duplicated with a different transport identifier, while two legitimate schedule runs can carry the same shipment and subscriber with different event versions.
Do not put secrets in the command or in logs. For outbound HTTP, validate destinations against an allowlist or a controlled tenant mapping; accepting an arbitrary URL supplied by a message creates a server-side request forgery risk. The OWASP guidance on SSRF prevention is a useful security review input here, particularly because fan-out systems turn stored destinations into repeated network requests.
Observability should expose counts by state, attempt age, subscriber class, event version, and schedule run. It should not expose the shipment payload by default. Alert on planned commands that never publish, published commands that never reach a terminal state, and dead-letter growth. A green scheduler metric can't prove delivery.
How can a worker preserve delivery evidence?
A dead-letter queue is a quarantine and review surface. It should preserve the original command, failure class, attempt history, and the reason it stopped retrying. It should not be a second queue that receives everything after an arbitrary number of attempts.
A retry policy can be expressed as a small table:
| Failure | Default action | Required evidence |
|---|---|---|
| Timeout or connection reset | Retry with bounded backoff | Attempt timestamps and receiver identity |
| Rate limit response | Retry after the receiver's stated delay | Response class and next eligible time |
| Invalid command | Dead-letter immediately | Validation error and schema version |
| Unknown outcome after timeout | Reconcile before redrive | Subscriber version or request receipt |
| Repeated authorization failure | Dead-letter and alert | Tenant and credential policy reference |
Redrive is a state transition, not a copy button. An operator should confirm that the subscriber is still entitled to receive the shipment update, inspect whether event version 18 was already accepted, and then release the original command identity. If the shipment has been superseded, reconciliation may close the command as obsolete instead.
Your mileage may vary on the retry count. There is no honest universal number without knowing subscriber latency, rate limits, business urgency, and the cost of a duplicate. What should be universal is the evidence: every attempt needs a reason, and every terminal state needs an owner.
A scheduled cleanup of expired delivery records can use the same architecture, but it should be treated as maintenance of the ledger, not as permission to delete evidence still required by a retention policy. The cleanup job needs its own idempotency key and audit record.
Where this setup stops fitting
Use the queue for isolation, the outbox for publication recovery, and the ledger for business truth. Keep the schedule thin. Make delivery idempotent. Separate retryable failures from poison commands, and redrive only after reconciliation.
Choose a direct scheduled handler when the operation is one bounded transaction and whole-run recovery is acceptable. This queue-and-ledger setup is not suitable when the operation is a single atomic database change with no per-recipient state; in that case, stick with the transaction and its database retry policy. Choose a workflow or stream model when the requirement has become compensation, long waits, replay, or multiple independent consumers. The right answer is determined by the failure boundary, not by whether the setup looks small in a tutorial.
Top comments (0)