Use a durable delivery-claim ledger keyed by the logical password-reset challenge, channel, and message purpose; commit that claim before dispatch, reuse the same idempotency key on every retry, and record an ambiguous outcome as unknown rather than pretending the notification was sent exactly once. For a gaming account reset with a short expiry, compliance evidence is the deciding constraint: the system must explain why an email or SMS was attempted, which logical message it represented, and what the transport acknowledged without storing the reset secret itself.
Exactly-once is the wrong external promise. A worker can crash after a transport accepts a message but before the acknowledgement reaches the backend. No local flag can distinguish that case from a request the transport never accepted. The defensible goal is narrower: one durable intent, one stable deduplication identity, bounded retry behavior, and an audit trail that preserves uncertainty.
It sounds fussy. It isn't.
Minimize reset data before deduplicating it
The ledger needs an opaque challenge identity, not the secret that authorizes the password change. That privacy boundary comes first; deduplication cannot justify copying credentials into a longer-lived evidence store.
Start with the unit of intent, not the queue delivery. A queue message ID identifies one delivery attempt by the broker; a password-reset challenge identifies the user-visible action. If a handler derives its idempotency key from the queue message ID, a redelivery with a new envelope can pass the guard and send the same reset twice. Derive the stable key from immutable business inputs such as challenge_id, channel, and message_purpose. Keep template_version in the recorded payload hash when content changes matter to the evidence, but don't casually put a mutable template version into the deduplication key: doing so may turn a rendering update into permission to notify again.
For a workflow that deliberately sends both channels, email and SMS need separate claims because they are separate intended effects. For a fallback workflow, the workflow state must authorize only one active channel at a time; two independent consumers racing over the same account event can each be internally idempotent and still produce two messages. Deduplication is local to the identity you define — a badly scoped identity gives a perfectly consistent wrong answer.
A Node.js backend should put the unique constraint and state transition in the database, even though the example below is Python. The language isn't the concurrency boundary. The database is. An in-process map, a request-scoped flag, or a cache entry created after dispatch leaves a gap in which two workers can both decide they are first.
The minimum invariants are deliberately small:
- One row exists for each logical message and channel.
- The unique claim is committed before any external side effect begins.
- Every transport retry reuses the same idempotency key and content hash.
- Expired challenges are suppressed before dispatch and recorded with a reason.
- An acknowledgement records the transport request identifier; a timeout records an unknown outcome, not a fabricated success or failure.
- Logs contain an opaque challenge identifier and payload hash, never the reset token or raw message body.
That last distinction matters. accepted, rejected, and unknown are three different facts. Collapsing unknown into rejected makes an automatic retry look safe when it may duplicate an already accepted SMS. Collapsing it into accepted makes the audit record claim evidence the system never received. I'm not sure every transport exposes enough lookup data to resolve an unknown result; that capability has to be verified for the chosen integration before the retry policy is approved.
What should compliance evidence prove about Node.js email and SMS retries?
The ledger should model a state machine, not a Boolean sent column. A practical sequence is claimed to dispatching to accepted, with terminal expired and rejected states plus an unknown state for an interrupted acknowledgement. Each transition should be append-only in the audit history even if a current-state column exists for fast reads.
Consider a ten-minute reset challenge. The API creates challenge rst_7f31, records its expiry, and publishes a notification intent. Worker A claims the email message and begins dispatch. The transport accepts it, but Worker A loses its connection before receiving the response. Worker B later sees the queue redelivery. If it checks only status != accepted, it sends again. If it checks only that a row exists, it may suppress a message that was never accepted. The correct next action depends on transport semantics: reuse the original idempotency key when that key is honored across retries, query by the stored request identity when reconciliation is available, or quarantine the unknown outcome for an explicit policy decision. The ledger doesn't erase uncertainty; it stops the application from laundering uncertainty into a confident but false event.
Short expiry adds another boundary. The worker must compare the authoritative expiry with the current database time before dispatch, not merely trust a delayed job's original schedule. Once the challenge is expired, mark the intent expired and stop. Sending an unusable reset link after expiry is noisy for the player and creates misleading evidence because the delivery succeeded while the security action could not.
For compliance review, retain event identifiers, transition timestamps, worker identity, channel, content hash, policy version, idempotency key, and transport acknowledgement identifier according to the organization's retention policy. Keep the token out. A complete rendered-body hash can show that repeated attempts used identical content, but a hash is evidence of equality, not evidence that a recipient read or even received the message. Precise labels beat expansive claims.
Storage writes and retention are the real cost
| Approach | Duplicate control | Crash after acceptance | Evidence quality | Suitable use |
|---|---|---|---|---|
| Process memory or expiring cache | Best-effort within a narrow window | State may vanish or race | Weak; expiry can erase the decision | Low-risk, replaceable notifications |
| Transactional outbox only | Prevents lost publication from the business transaction | Consumer can still repeat the external effect | Good intent history, incomplete delivery history | Events whose consumers are independently idempotent |
| Durable claim ledger plus stable transport key | Enforces one logical claim and repeat identity | Preserves and may reconcile an unknown result | Strongest of these options if transitions are retained | Security messages requiring reviewable retry decisions |
The third design has a catch: it costs extra writes, needs retention and privacy rules, and requires operational handling for records that remain unknown. It is not suitable when the message is disposable and duplication has no meaningful impact; a short-lived cache may be enough for a transient game-presence update. Stick with a transactional outbox without a dedicated delivery ledger when downstream delivery is already idempotent and the outbox record supplies all evidence the organization requires.
A replaceable transport boundary in Python
The following sketch keeps transport and database APIs generic. db.transaction() must provide a real database transaction, insert_delivery_claim() must be backed by a unique constraint on dedupe_key, and transport.send() must receive the same key on each permitted retry. Those are contracts, not comments to wave away during implementation.
from dataclasses import dataclass
from datetime import datetime, timezone
from hashlib import sha256
@dataclass(frozen=True)
class ResetNotice:
challenge_id: str
channel: str
destination_ref: str
rendered_body: bytes
expires_at: datetime
def make_dedupe_key(notice: ResetNotice) -> str:
identity = f"password-reset:{notice.challenge_id}:{notice.channel}"
return sha256(identity.encode("utf-8")).hexdigest()
def claim_notice(db, notice: ResetNotice) -> dict:
now = datetime.now(timezone.utc)
dedupe_key = make_dedupe_key(notice)
body_hash = sha256(notice.rendered_body).hexdigest()
with db.transaction():
existing = db.get_delivery_claim_for_update(dedupe_key)
if existing is not None:
return existing
state = "expired" if now >= notice.expires_at else "claimed"
return db.insert_delivery_claim(
dedupe_key=dedupe_key,
challenge_id=notice.challenge_id,
channel=notice.channel,
destination_ref=notice.destination_ref,
body_hash=body_hash,
state=state,
expires_at=notice.expires_at,
claimed_at=now,
)
def dispatch_claim(db, transport, notice: ResetNotice) -> str:
claim = claim_notice(db, notice)
if claim["state"] in {"accepted", "expired", "rejected"}:
return claim["state"]
if claim["state"] == "unknown":
return "reconciliation_required"
with db.transaction():
current = db.get_delivery_claim_for_update(claim["dedupe_key"])
now = datetime.now(timezone.utc)
if now >= current["expires_at"]:
db.append_transition(current["dedupe_key"], "expired", now)
return "expired"
if current["state"] != "claimed":
return current["state"]
db.append_transition(current["dedupe_key"], "dispatching", now)
try:
receipt = transport.send(
channel=notice.channel,
destination_ref=notice.destination_ref,
body=notice.rendered_body,
idempotency_key=claim["dedupe_key"],
)
except TimeoutError:
db.append_transition(
claim["dedupe_key"], "unknown", datetime.now(timezone.utc)
)
return "reconciliation_required"
db.append_transition(
claim["dedupe_key"],
"accepted",
datetime.now(timezone.utc),
transport_request_id=receipt.request_id,
)
return "accepted"
There is an intentional hard stop on unknown. A production reconciler may resolve it from a transport lookup or safely repeat the request under the same transport-enforced key, but blindly moving it back to claimed would discard the one fact the system knows: dispatch started and its result was not observed. Your mileage may vary on retention periods and escalation ownership, because those depend on policy and jurisdiction; the state meanings should not vary.
Failure injection is the release gate
Testing needs the ugly interleavings, not just two identical HTTP requests. Run two workers against the same key and assert that the unique constraint produces one claim. Terminate a worker after dispatching and before the acknowledgement write. Advance the clock past expiry while a job waits. Change the rendered body while keeping the logical identity and assert that policy blocks the mismatch. For fallback delivery, race email and SMS authorization and verify the workflow permits only the intended channel. These tests belong at the database and transport-adapter boundary because a mocked service function won't reproduce transaction contention.
Operationally, count claims by state and age, alert on old dispatching and unknown records, and sample payload-hash mismatches. Don't use a high retry count as a success metric. A retry is evidence that an earlier attempt did not produce a usable local outcome; it says nothing by itself about recipient delivery.
Reject cache-only guards at the review gate
A cache-only guard is attractive because set-if-absent is fast and easy to add around a worker. It was rejected here because the compliance question outlives a cache TTL, eviction can remove the only decision record, and a write performed after dispatch retains the crash window. Writing the cache before dispatch flips the failure: a crash can leave a key that suppresses a message that was never attempted. Shortening the TTL only changes which failure is more likely. It doesn't establish evidence.
The cache option still has a valid use case. Use it as a rate-control layer in front of the durable claim, or as the only guard for low-consequence, rapidly obsolete messages where an occasional duplicate or omission is explicitly acceptable. That isn't the password-reset case: a player may request several challenges, each challenge expires quickly, and reviewers need to distinguish a new authorized reset from a duplicate delivery attempt for the same one.
The decision rule is plain. If the team must later explain one security notification, preserve its logical identity and every state transition in durable storage; if the transport result becomes ambiguous, preserve that ambiguity until a documented reconciliation policy resolves it. Do not label a queue as exactly-once and assume the external world agreed.
Top comments (0)