EastonPierce8265

Posted on Sep 21

Duplicate Event Notifications Explained: 2026 Node.js Email and SMS Retry Keys

#node #email #backend

Short answer: A Node.js backend should deduplicate each seller order event into one immutable email or SMS intent, then record retries as separate attempts instead of claiming exactly-once transport.

A marketplace should let the team that owns the seller experience own the notification template, while the delivery system owns retry state. Mixing those responsibilities is the quiet cause of many duplicate and inconsistent new-order alerts. The practical design is to persist one notification intent per order and channel under a unique key, freeze the rendered content, and record every delivery attempt separately. Retries then reuse the same intent instead of creating another email or SMS. This ownership split also keeps a copy edit, locale update, or seller-profile change from altering content halfway through a retry sequence.

This distinction matters because a worker can lose its database connection after a provider accepts a request. The worker cannot infer from its timeout whether nothing happened or a message is already in flight. Retrying is necessary; creating a fresh intent is not.

How can retries prevent duplicate event notifications without claiming exactly once?

An idempotency key answers which business action is being repeated. It does not answer which subject line, locale, SMS text, or destination should be used on the repeat. If a retry fetches the current template and current seller profile, the second attempt can differ from the first even though both carry the same key. That makes investigation harder and can turn a harmless transport retry into two distinct communications. Consider a worker that renders revision 7, sends it, times out, and returns the job to the queue. While it waits, the template owner publishes revision 8 and the seller changes a forwarding address. A worker that renders again has created a new communication under an old operation key. Freezing the payload at intent creation prevents that ambiguity.

For a marketplace order, define the intent before contacting either channel. A useful identity tuple is marketplace_id + order_id + event_kind + channel + template_revision. The tuple is more informative than a random request ID because it states the scope of uniqueness. A database unique constraint should enforce it. A hash of that tuple can be exposed as an opaque idempotency key when a downstream interface accepts one.

The template owner decides the words and the revision. The notification service snapshots the rendered subject and body, destination, locale, and consent decision into the intent record. The delivery worker may read that snapshot, but it must not reinterpret the order or silently upgrade the template during a retry. This is the ownership boundary.

There is a deliberate trade-off here. Including template_revision in the identity permits a corrected template to become a new intent; excluding it guarantees at most one seller alert for the order event regardless of content changes. For a routine new-order notice, exclude the revision from the unique constraint but retain it as immutable metadata. A correction then requires an explicit new event kind, which is easier to audit than an accidental resend.

The two ledgers have different retention jobs

One table represents notification intents. Another represents attempts. The intent row is compact and authoritative: business key, channel, frozen payload digest, template revision, state, and timestamps. The attempt row is operational evidence: attempt number, start and finish time, outcome class, downstream request identifier when one exists, and a bounded error category.

Do not put full response bodies, seller addresses, or rendered message content into every attempt row. If a 6 KB payload is copied into four attempts for 8 million monthly intents, the duplicated payload alone is about 192 GB before indexes and storage replication. The number is arithmetic, not a benchmark: 6 KB x 4 x 8,000,000. Store the immutable payload once, then let attempt records point to it.

Retention should follow the question each dataset answers. Keep intents long enough to resolve seller disputes and prove which business action was represented. Keep verbose attempt diagnostics for the shorter period needed to investigate delivery failures. Aggregate counters can live longer than high-cardinality traces.

Three clocks, three purposes.

Cardinality grows faster than many teams expect. Labels such as channel=email and outcome=accepted are bounded. order_id, seller_id, destination, and downstream request ID are effectively unbounded; using them as metric labels creates a time series for each value. Put those identifiers in searchable, access-controlled logs or trace attributes, and keep metrics grouped by stable dimensions such as channel, template revision, outcome class, and deployment version.

A retry-safe Node.js contract

The ingest path should perform one atomic database transaction: insert the intent with a uniqueness constraint and insert an outbox record that refers to it. If the unique key already exists, return the existing intent identifier. PostgreSQL documents ON CONFLICT as the mechanism for choosing an alternative action when a uniqueness conflict occurs. This closes the race between two consumers that receive the same order event at nearly the same time.

The outbox publisher may enqueue the same intent more than once. That is acceptable. Every worker first claims or reads the existing intent; it never constructs a second one from the queue message. Before a send, it records an attempt. After the downstream call, it records the observed result.

A minimal black-box test can stay at the HTTP boundary. Assume the local fixture exposes an illustrative order-event endpoint and an intent inspection endpoint:

body='{"event_id":"evt-order-8042","event_kind":"order.created","order_id":"8042","seller_id":"seller-19","channel":"email","template_revision":"order-created-v7"}'
curl --fail-with-body --silent --show-error \
  --request POST http://localhost:3000/test/order-events \
  --header 'content-type: application/json' \
  --header 'idempotency-key: marketplace:8042:order.created:email' \
  --data "$body"
curl --fail-with-body --silent --show-error \
  --request POST http://localhost:3000/test/order-events \
  --header 'content-type: application/json' \
  --header 'idempotency-key: marketplace:8042:order.created:email' \
  --data "$body"
curl --fail-with-body --silent --show-error \
  http://localhost:3000/test/notification-intents/marketplace%3A8042%3Aorder.created%3Aemail

The assertion is not merely that both POST requests return success. The inspection response should show one intent, one frozen payload digest, and however many attempts the fixture deliberately caused. Then change one field while reusing the key. The service should reject the mismatch rather than treating the same key as permission to send different content.

HTTP defines idempotent methods in terms of the intended effect of multiple identical requests, but POST is not inherently idempotent. An application-level key and stored result supply the missing identity for this operation. The server must also compare a request fingerprint because accepting the same key with a different order, channel, or payload hides a caller bug.

No database transaction can atomically cover a local row and an unrelated delivery provider. There remains an ambiguous interval after the provider accepts a request and before the attempt is marked accepted. A downstream idempotency facility can narrow that interval when available, but the local model must retain an unknown outcome rather than converting every timeout into failed.

Unknown is awkward. It is also honest.

Observability without paying per order forever

Start with four counters: intents created, uniqueness conflicts, attempts started, and attempts by outcome class. Add a histogram for attempt latency. These measurements can answer whether duplicate input is rising and whether downstream behavior changed without using an order ID as a metric label.

Logs serve the individual investigation. Emit one structured event when an intent is created or reused and one for each attempt transition. Include the opaque intent ID, attempt ID, channel, template revision, outcome class, and trace ID. Redact destinations and message bodies. Sampling deserves care: routine accepted attempts may be sampled, but duplicate-key conflicts, payload mismatches, unknown outcomes, and terminal failures should be retained because they are precisely the rare paths needed during an audit.

Retention math should be reviewed before adding fields. Suppose an attempt event is 1.2 KB, the service processes 8 million intents a month, and the mean is 1.15 attempts per intent. Raw attempt logs are about 11.04 GB per month before indexing, replicas, and metadata. Adding a 4 KB rendered body to each event would raise the raw monthly volume by another 36.8 GB. That extra copy contributes no new evidence if the payload digest already links to the immutable intent.

Tracing has a related trap. Recording order_id as an attribute can help a targeted search, but exporting every successful trace at full rate is usually a poor default for a high-volume notification path. Head sampling reduces volume early but may discard a trace that later fails. Tail sampling can retain failures after observing an outcome, although it requires buffering and more collector work. The choice belongs in the cost model, with an explicit retention target and measured event size.

Roll out the ownership boundary in four steps

First, introduce the intent unique key in observe-only logic and measure collisions without changing send behavior. Second, snapshot template revision and payload digest while the existing path still delivers. Third, switch workers to consume intent IDs and reuse frozen content; inject timeouts before and after the downstream call to test ambiguous outcomes. Finally, enable the uniqueness constraint as the authority and shorten verbose attempt retention only after audit queries have been exercised.

Success means duplicate order events converge on one immutable seller-notification intent, while retries remain visible as separate attempts. That model does not pretend the network offers exactly-once delivery. It gives template owners a stable contract, operators defensible evidence, and the observability budget a bounded set of dimensions.

DEV Community