Short answer: ingest bounce and complaint webhooks first, then run a polling job as a reconciliation backstop; put both paths through one idempotent event handler and consult the resulting suppression state before sending another transactional email.
For a fintech order receipt, the expensive part of compliance evidence is usually retention, not the polling loop. Model it before choosing an intake mechanism: stored bytes = events per day x retained bytes per event x retention days x copy count. Copies include the primary database, replicas, backups, analytics exports, and logs. A useful design change is to retain a compact normalized decision record for the policy-defined period while expiring bulky raw payloads sooner. I would keep message content out of that record, because a receipt body adds exposure without proving that a bounce or complaint was honored.
The choice is deliberately uneven. Webhooks give prompt feedback; polling repairs gaps caused by deployments, expired subscriptions, or short network interruptions. The catch is operational complexity: two intake paths can duplicate events and reorder them. A single durable polling job is reasonable at low volume when delayed suppression is acceptable, while webhook-only handling is unsuitable when the provider cannot replay every event you need to audit.
Duplicates are normal.
What should the evidence record contain?
A provider payload is transport evidence, not yet a compliance decision. Normalize it into a small append-only event row, then derive the current recipient state from that history. The minimum useful row has an immutable provider event identifier, internal message identifier, recipient key, event type, provider occurrence time, ingestion time, source path, payload digest, and schema version. Store the recipient as a keyed hash when operators don't need the clear address; keep the key outside the event store and document its rotation policy.
The derived state answers a narrower question: may this stream send to this recipient now? Use separate scopes for promotional mail, security messages, and transactional receipts. An unsubscribe from a mailing list shouldn't silently erase a legally or operationally required receipt, while a hard delivery failure may make every email stream futile. Complaint handling deserves an explicit policy decision rather than a convenient boolean copied between systems.
For each attempted receipt, link the business order ID to an internal message ID before dispatch. That ordering matters. If the process stops after the email provider accepts a request but before the application writes the link, the team can have delivery telemetry it cannot join back to the settled payment. A transactional outbox closes that gap: payment settlement and the intent to send are committed together, then a worker delivers the outbox item. It isn't glamorous.
It is auditable.
The evidence table should explain a decision without relying on mutable application logs:
| Evidence | Why retain it | What to avoid |
|---|---|---|
| Message-to-order link | Connects the receipt to a settled payment | Full receipt body |
| Immutable event ID and digest | Detects duplicate or altered input | Trusting arrival order |
| Provider and ingestion timestamps | Shows occurrence and processing delay | Treating local time as provider time |
| Policy result and version | Explains why suppression changed | A bare suppressed=true flag |
| Source and schema version | Supports webhook/poll reconciliation | Provider-specific fields in business logic |
Retention is a policy input, not a universal constant. I'm not sure what period applies to a particular business without its jurisdiction, product, dispute window, and counsel-approved schedule. That uncertainty should be resolved in configuration and policy documentation, not hidden in a worker default.
How should a Node.js polling job handle email bounce and complaint events?
Treat the polling job as a cursor-driven reconciliation process. Fetch one bounded page, validate and normalize every event, commit the events and the next cursor in the same database transaction, then request another page. If the transaction fails, the cursor doesn't advance. The next run safely sees the page again, and a unique constraint on (provider, provider_event_id) turns re-delivery into a no-op.
Although the state machine is runtime-independent, the following Python reference makes the transaction boundary explicit. A Node.js worker should preserve the same boundaries in its database client and scheduler rather than translating the control flow into concurrent, unbounded promises.
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Iterable, Optional, Protocol
@dataclass(frozen=True)
class DeliveryEvent:
provider: str
event_id: str
message_id: str
recipient_key: str
kind: str
occurred_at: datetime
payload_digest: str
class EventSource(Protocol):
def list_events(
self, cursor: Optional[str], limit: int
) -> tuple[Iterable[DeliveryEvent], Optional[str]]: ...
def reconcile(source: EventSource, store, batch_size: int = 100) -> int:
cursor = store.lock_cursor("delivery-events")
events, next_cursor = source.list_events(cursor=cursor, limit=batch_size)
accepted = 0
with store.transaction() as tx:
for event in events:
if event.kind not in {"bounce", "complaint", "delivered"}:
continue
inserted = tx.insert_event_if_absent(
event=event,
ingested_at=datetime.now(timezone.utc),
)
if inserted:
tx.recompute_suppression(
recipient_key=event.recipient_key,
policy_version="receipt-delivery-v1",
)
accepted += 1
tx.save_cursor("delivery-events", next_cursor)
return accepted
Do not use the provider timestamp as the cursor unless its contract guarantees a total, stable order. Equal timestamps, clock precision, and late events make time-only checkpoints fragile. Prefer an opaque provider cursor. If only a time window exists, query with deliberate overlap and depend on the event ID constraint for deduplication. Your mileage may vary because replay and pagination contracts differ; verify them with recorded fixtures from the provider's documented event schema.
Retries belong around page acquisition and whole database transactions, not around individual state mutations. Use exponential backoff with jitter, a finite attempt budget per run, and a scheduler lock so two replicas don't race the same cursor. A request timeout or rate-limit response should leave the checkpoint unchanged and end cleanly for the next scheduled run.
Don't hammer it.
Webhooks call the same normalization and insertion function, but they must acknowledge only after durable acceptance. Authenticate requests using the provider's documented signing procedure, preserve the raw bytes needed for signature verification, reject stale or invalid signatures, and cap request size before parsing. Queueing after verification keeps the public endpoint short; the worker can then apply the event under the same unique constraint used by polling.
Suppression is a state machine, not a list
An address can receive delivered, bounce, and complaint events out of order. Last-write-wins logic therefore gives the wrong event authority. Define precedence by policy: a complaint produces an immediate durable suppression for the relevant stream; a permanent bounce suppresses according to the address policy; and a transient bounce updates retry eligibility without erasing a stronger prior state. A later delivery can provide evidence about one attempt, but it should not automatically clear a complaint.
The send path needs an atomic check close to dispatch. Read the suppression decision, create the message attempt with an idempotency key based on the order and receipt type, and enqueue it under one consistent boundary. Repeated settlement notifications then refer to the existing attempt instead of producing duplicate receipts. Keep retry counts on the attempt, not the recipient, because two orders for one address are separate business actions even though they share deliverability state.
One-click unsubscribe is frequently misapplied here. RFC 8058 defines a one-click mechanism for list email through List-Unsubscribe and List-Unsubscribe-Post headers, including an HTTPS POST action. It is relevant to promotional or subscription streams, but it should not be bolted onto a mandatory payment receipt without a product and legal decision about what the recipient is actually opting out of. Mixing those streams also makes suppression evidence harder to explain.
OTP and password-recovery mail needs another policy boundary. The OWASP Forgot Password Cheat Sheet calls for consistent responses, uniform timing, side-channel delivery, random single-use expiring tokens, rate limiting, and secure token storage. Those controls are useful for security messages, but an order receipt isn't an authentication token. Sharing a queue is possible; sharing retry rules, template data, and suppression semantics by accident is not.
Can the retry-safe pattern prove compliance during an incident?
Test the evidence path by changing order, duplication, and timing. Feed a complaint twice through webhook intake, replay it through polling, then deliver an older bounce after it. The event table should contain each immutable event once, the complaint decision should remain in force, and the cursor should advance only with the committed page. Also test an event with an unknown message ID: quarantine it with a reason and alert on the count rather than discarding evidence that may become joinable after delayed replication.
Deploy schema readers before writers when adding an event version. During rollout, keep one scheduler lease active, expose the age of the last successfully committed provider event, and alert on cursor stagnation, webhook signature failures, quarantine growth, and send attempts blocked by suppression. Counts should be partitioned by stream and event type. Addresses and receipt contents don't belong in metric labels or exception text.
There is a real cost to the compact-retention design. Expiring raw payloads removes the easiest way to reinterpret old events after a parser mistake or provider schema change. Offset that risk with versioned normalization, payload digests, contract fixtures, and a short raw-event review window chosen by policy. If the organization must reproduce the original signed payload years later, compact evidence is not suitable; retain encrypted raw events under stricter access control and accept the storage, key-management, and discovery burden.
Run one recovery exercise before calling the design complete. Restore the event store and cursor into an isolated environment, replay normalized records, and compare the resulting suppression snapshot by hash. A backup that cannot recreate the current decision is archival theater.
For a receipt sent after payment settlement, the practical choice remains webhook-first intake with polling reconciliation, one idempotent event table, and policy-scoped suppression. Stick with polling alone only when its documented delay is acceptable and the source guarantees sufficient replay; choose stronger raw retention when original payload reconstruction is a compliance requirement.
References
- RFC 8058, "Signaling One-Click Functionality for List Email Headers": https://datatracker.ietf.org/doc/html/rfc8058
- OWASP Forgot Password Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html
Top comments (0)