Short answer: treat delivery as an auditable state machine, not a single retryable function. Give each health-report notification a stable idempotency key, claim it transactionally, and record the provider result before a worker can try again. That makes duplicate sends detectable and usually preventable, while being honest about the one gap no email or SMS system can close: a process can die after the provider accepts a message but before your database records it.
The concrete job here is small: a backend generates a PDF health report and sends it as an email attachment, with an SMS link as a fallback. The integration-effort constraint changes the design. A complicated delivery SDK, a second queue protocol, and six configuration files create more places to lose the correlation ID. I want one narrow contract that any transport can implement and a ledger a reviewer can inspect.
Why exactly-once is a claim you should not make
A retry loop can provide at-least-once delivery. It cannot prove exactly-once side effects across two systems. Consider the timeline: the worker uploads the attachment, calls the email provider, gets a network response, and crashes before committing sent in its database. The next worker sees pending and sends again. The provider may have accepted the first request. Your database does not know that. In a healthtech workflow, that gap is not an abstract distributed-systems footnote: a duplicate report can trigger a second patient notification, a second support ticket, and an audit question about which copy is authoritative. The event record, attachment checksum, request key, and provider response must therefore be connected before anyone retries. That is extra bookkeeping, but it is the smallest explanation that survives a post-incident review.
No magic.
This is the classic two-generals problem in a less dramatic costume. An idempotency key lets the receiving API collapse repeated requests only when that API stores and enforces the key. Your own ledger still matters because it gives support and compliance staff a durable explanation of what happened.
I once started with a boolean called email_sent. It looked clean until SMS became a fallback. Then a timeout after the email call left email_sent = false, and the fallback path sent an SMS for an event that had already reached the inbox. The fix was not a longer timeout. It was a per-channel attempt record with the same event identity and a different delivery identity.
How should a Node.js backend model email and SMS deduplication?
Use three gates. Each gate answers a different question, so combining them into one sent flag loses useful evidence.
- Event gate: has this business event already produced a notification intent?
- Attempt gate: has this channel and intent already been claimed by a worker?
- Provider gate: did the transport receive the same idempotency key before?
The event key should be derived from immutable business data, such as reportId:recipientId:reportVersion. Do not use a timestamp or a random UUID generated inside the retry loop. The channel attempt key can append :email or :sms; a manual resend should create a new, explicitly linked intent instead of quietly reusing the old one.
Here is the smallest TypeScript shape I use around a SQL-backed ledger. The repository methods are deliberately generic: the important part is the uniqueness constraint and the order of operations.
type Channel = "email" | "sms";
type DeliveryState = "pending" | "claimed" | "accepted" | "failed";
type NotificationIntent = {
eventKey: string;
reportId: string;
recipientId: string;
channel: Channel;
attachmentUrl?: string;
};
interface DeliveryLedger {
insertIfMissing(intent: NotificationIntent): Promise<boolean>;
claim(eventKey: string, channel: Channel, workerId: string): Promise<boolean>;
recordAccepted(eventKey: string, channel: Channel, providerId: string): Promise<void>;
recordFailure(eventKey: string, channel: Channel, reason: string): Promise<void>;
}
async function dispatch(
ledger: DeliveryLedger,
intent: NotificationIntent,
send: (key: string, intent: NotificationIntent) => Promise<{ id: string }>,
workerId: string,
): Promise<DeliveryState> {
const created = await ledger.insertIfMissing(intent);
if (!created) return "accepted";
const claimed = await ledger.claim(intent.eventKey, intent.channel, workerId);
if (!claimed) return "accepted";
const attemptKey = `${intent.eventKey}:${intent.channel}`;
try {
const result = await send(attemptKey, intent);
await ledger.recordAccepted(intent.eventKey, intent.channel, result.id);
return "accepted";
} catch (error) {
await ledger.recordFailure(
intent.eventKey,
intent.channel,
error instanceof Error ? error.message : "unknown transport error",
);
throw error;
}
}
insertIfMissing must be backed by a unique index on (event_key, channel). Checking first and inserting later is a race. Two workers can both observe no row. The database has to arbitrate. claim should be a conditional update with a lease or version number, so a stalled worker does not own the row forever.
The attachment itself needs a stable reference too. Generate the report once, store it under a content-addressed or immutable key, and put that reference in the intent. Regenerating a PDF during a retry can produce a different checksum and turn one notification into two semantically different messages.
What does a retry audit need to prove?
An audit row is useful only if someone can reconstruct the decision without reading worker logs. I keep the event key, channel, report checksum, attempt count, lease owner, timestamps, provider request key, provider message ID, and the final classification. Keep recipient address data out of general-purpose logs; store a reference to the protected record instead.
A practical classification has three buckets: retryable transport failure, permanent input failure, and ambiguous outcome. A DNS timeout is retryable. An invalid destination is not. A timeout after request transmission is ambiguous: retry with the same provider key if the provider supports idempotency, then flag the row for reconciliation if the outcome remains unknown.
Do not turn every exception into a retry. Backoff with jitter, cap the attempt count, and send the exhausted row to a review queue. The reviewer should see the original event and the exact reason the system stopped.
The metrics I care about are boring and revealing: duplicate-key rejects, ambiguous outcomes, median time from intent to acceptance, attachment checksum mismatches, and the percentage of rows reconciled manually. A single aggregate “delivery success” metric hides the failure mode this design is meant to expose.
The smallest test that catches the expensive bug
Unit tests for the sender are not enough. Run an integration test against the real transaction boundary and force a crash at each edge: before claim, after claim, after the provider accepts, and after recordAccepted. Then replay the same event. The expected result is one provider request key, one ledger row, and a visible ambiguous state when the crash happens in the last window.
it("replays an accepted email with the same key", async () => {
const calls: string[] = [];
const send = async (key: string) => {
calls.push(key);
return { id: "provider-msg-7" };
};
await dispatch(ledger, intent, send, "worker-a");
await dispatch(ledger, intent, send, "worker-b");
expect(calls).toEqual(["report-42:patient-9:email"]);
});
That test proves your local gate. It does not prove the provider's behavior. Check the provider contract separately: does it retain keys, for how long, and what response identifies a replay? Your adapter should preserve those semantics instead of pretending every HTTP 2xx means the same thing.
Trade-offs and the scale-up decision
The ledger adds a table, a transaction, and a reconciliation job. That is real integration effort. It also gives you a defensible answer when a patient asks why a report arrived twice. For a low-risk internal alert, a queue with at-least-once delivery may be enough. Do not pay the audit cost where the consequence is trivial.
The catch is that no design can guarantee exactly one human-visible message when the provider and your database commit independently. If you need that guarantee, choose a transport with durable idempotency and a documented retention window, and still keep the ambiguous state. Stick with a simpler outbox plus dedupe ledger when you control the worker and need portable email and SMS adapters; choose a managed workflow when your team cannot operate leases, replay tooling, and reconciliation.
At scale, I would move report generation into an immutable artifact service, partition the ledger by tenant and month, and expose a read-only audit endpoint. I would not add a framework just to hide three SQL statements. The fastest integration is the one a new engineer can trace from event key to provider response in one afternoon.
Top comments (0)