DEV Community

ArthurFinley2291
ArthurFinley2291

Posted on

Logistics Digest Node.js Background Job Queue: At-Least-Once Duplicate Processing

Short answer: for a weekly logistics digest, accept at-least-once delivery, give each customer-period digest a stable idempotency key, and commit the send decision before acknowledging the job. Retries then repeat delivery attempts without repeating the business effect.

This is a delivery-guarantee problem, not a queue-selection slogan. A worker can finish sending a digest and disappear before its acknowledgement reaches the broker. The next delivery is expected. The dangerous design is the one that treats the second delivery as exceptional and has no durable record with which to decide what happened.

Start with the business effect, not the delivery ID

A logistics digest might summarize shipments active at the end of a week: delayed consignments, customs holds, and parcels awaiting handoff. The producer should create one command for the intended recipient and period, for example digest:customer-1842:2026-W32, and carry that key through every retry. A queue delivery identifier is not suitable for this job because a redelivery normally receives a new identifier while the customer-period intent remains the same.

I write the invariant down before writing the consumer: one customer, one digest period, one accepted send decision. The message may be seen twice. The digest decision may be recorded once.

Three words matter.

The idempotency key must describe the business command, not the worker attempt. If the digest contents are allowed to change after a retry, store a payload hash as well. A reused key with a different hash should be rejected or routed for review; silently sending a different report under the same identity makes reconciliation ambiguous.

How can a Node.js background job queue make duplicate processing idempotent?

Make the database transaction the authority. In one transaction, insert the idempotency key and payload hash under a unique constraint, create the outbox event or durable send record, and mark the command accepted. Only after commit should the consumer acknowledge the queue message. A duplicate key reads the existing record, records the repeated delivery as an attempt, and acknowledges without creating another business command.

There is an important boundary around the external email or notification provider. A local transaction cannot make a remote side effect exactly once by itself. Use a provider-side idempotency key when the provider supports one; otherwise, persist a send intent and reconcile provider responses before retrying. The strongest honest claim is exactly-once business effect where both sides honor the same idempotency contract, while the queue itself remains at-least-once.

The following Go example shows the state transition in a small in-memory model. The mutex stands in for database transaction serialization and the map stands in for a unique index; a production Node.js worker needs the equivalent constraint in its database, even though the surrounding application is written in JavaScript or TypeScript.

package main

import "fmt"

type DigestStore struct {
    accepted map[string]string
}

func (s *DigestStore) Accept(key, payloadHash string) (duplicate bool, conflict bool) {
    if previous, exists := s.accepted[key]; exists {
        return previous == payloadHash, previous != payloadHash
    }
    s.accepted[key] = payloadHash
    return false, false
}

func main() {
    store := DigestStore{accepted: make(map[string]string)}
    key := "digest:customer-1842:2026-W32"

    duplicate, conflict := store.Accept(key, "sha256:weekly-summary-v1")
    fmt.Printf("first duplicate=%t conflict=%t\n", duplicate, conflict)

    duplicate, conflict = store.Accept(key, "sha256:weekly-summary-v1")
    fmt.Printf("retry duplicate=%t conflict=%t\n", duplicate, conflict)
}
Enter fullscreen mode Exit fullscreen mode

This example deliberately omits the notification call. Putting a remote send inside a database transaction does not extend the transaction's atomicity; it only makes the failure window harder to reason about. An outbox worker can claim the intent, call the provider with the same key, record the response, and retry according to the response class. I am not sure every provider preserves idempotency for every notification operation, so that capability must be verified in the provider's contract rather than inferred from the word “retry.”

Retry policy is a scheduling contract

Transient failure and permanent failure need different paths. A rate-limit response such as HTTP 429 should be retried after the server's Retry-After guidance when present; otherwise use bounded exponential backoff with jitter. The attempt record should retain the delivery ID, idempotency key, payload hash, attempt number, status, timestamps, and response classification. That data lets an operator distinguish a slow dependency from a duplicate business effect.

Do not retry malformed customer data forever. After a bounded policy, move the message to a dead-letter queue with its failure reason and correlation identifiers. A dead-letter queue is a holding and inspection boundary, not proof that the underlying command is safe to replay. Redrive only after the payload, handler, or dependency condition has been corrected, and preserve the original idempotency key during that redrive.

The operational alert should focus on the invariant. A rise in duplicate deliveries can be normal during a broker or worker disturbance; a rise in second sends for one customer-period is a correctness incident. Those are different signals and should not share one counter.

Delivery guarantees are application boundaries

For this workload, at-least-once delivery plus idempotent effects is usually the practical baseline. At-most-once delivery can lose a customer's report when a worker fails after receipt. An exactly-once queue claim does not remove the need to coordinate the queue, the database, and the external notification system. The useful decision is therefore to choose the weakest transport guarantee that meets loss tolerance, then make the business effect stricter at the application boundary.

Requirement Design response Trade-off
A digest must not be silently lost Durable queue, acknowledgement after commit, dead-letter handling Recovery and storage state must be operated
A customer must not receive two copies Stable customer-period key and unique constraint Old keys need retention and privacy rules
Digest contents must be auditable Store payload hash, decision, attempt history, and provider reference More records and data-governance work
A dependency is rate limited Honor Retry-After, then bounded backoff Delivery may arrive later than the schedule
Reports must be replayed for many consumers Use an event-log design or explicit fan-out More infrastructure than a single work queue

The catch is that this pattern is not suitable when the product requires immutable replay for several independent consumers, long-running workflow joins, or arbitrary historical reconstruction from the transport. Use an event log or workflow system when those are first-class requirements. Stick with a simpler work queue when the job is one bounded command and the durable application record is the source of truth.

Prove the rollout with reconciliation

Deploy the unique constraint and idempotency record before increasing concurrency or retry limits. Test the edges that ordinary happy-path tests skip: stop the worker after the send intent commits but before acknowledgement, deliver the same key to two workers, inject a 429, and redrive a dead-lettered message. The expected result is one customer-period decision, an explainable attempt history, and no second business effect.

Then reconcile by joining four facts: the producer command, the accepted idempotency record, the domain data used to build the digest, and the notification provider's result. An empty queue proves only that messages left the queue. It does not prove that every active customer received exactly the intended report.

The failure test deserves more detail than a green unit-test badge. Suppose the weekly scheduler emits the command, the consumer renders a report from shipment rows, and the provider accepts the send. If the process dies at that exact boundary, the broker will redeliver. The second worker must find the same key and the same payload hash, see that the send intent already has a provider reference, and finish as a duplicate without sending again. If the process dies before the intent commits, the second worker must be allowed to create it. If the shipment data changes between those attempts, the hash comparison must make the difference visible rather than quietly mixing two weekly snapshots. A test that only invokes the handler twice in a row misses all three boundaries because it contains no crash, no concurrent transaction, and no remote response ambiguity. I use explicit fault injection here, including a forced 429 and a process termination after the local commit, because a retry policy is only as correct as the state transition it can recover. The exact provider behavior still needs contract testing; a queue acknowledgement cannot answer a question that belongs to the notification provider.

Keep the reconciliation query boring. Boring is good in a ledger-oriented system. It should find missing records, hash conflicts, repeated provider references, and commands whose final status exceeds the weekly delivery window. Your mileage may vary on the retention period because legal, privacy, and customer-support requirements differ; document that decision instead of hiding it inside queue settings.

References

Top comments (0)