TL;DR: In a Node.js transactional app, treat email list hygiene as authorization checked at send time, not as a nightly cleanup job. For every generated marketplace report, resolve the recipient to a stable account identifier, consult a local suppression table immediately before handing off the attachment, record the attempt under an idempotency key, and sync bounce, complaint, and unsubscribe events into that same table. Poll for missed events and reconcile them by provider event ID. If suppression data is stale or the recipient state is ambiguous, hold the report rather than guessing.
That rule optimizes for the outcome that matters: a requested seller report reaches an address that can receive it without repeatedly sending to one that cannot or should not. A green “accepted” counter is insufficient evidence. The operational question is sharper: what page fires when event ingestion stops but report generation keeps succeeding?
How should a transactional app sync email list hygiene for bounced users?
An SMTP server can accept a message and later produce a delivery status notification. RFC 5321 explicitly separates a successful transfer from final delivery, while RFC 3463 defines enhanced status codes such as the 5.X.X permanent-failure class. The application therefore cannot equate a successful API or SMTP handoff with inbox delivery.
For a marketplace, that distinction becomes concrete at report time. Suppose seller account seller_48291 requests a weekly settlement report, the worker generates a 1.8 MB CSV attachment, and the current destination is finance@example.test. The send request may be accepted while a later event says the mailbox does not exist. During the gap, the report worker sees a completed handoff and the event consumer sees nothing; the next scheduled run can then create another accepted handoff to the same dead mailbox. If an address change also arrives during that interval, joining suppression state only on the raw address can either suppress the replacement incorrectly or lose the history attached to the seller. The useful model keeps account identity, destination identity, message class, source event identity, and effective time separate, so each transition can be explained after the fact. Retrying the same permanent failure on every scheduled run is not resilience. It is an uncontrolled loop that damages sender reputation and hides the actionable state change behind a reassuring submission metric. Unsubscribed users need a similarly explicit scope: an opt-out from subscribed marketplace updates is not automatically a ban on every requested account document, but the application must enforce the preference that was actually recorded.
Stop the loop.
There are at least four independently failing stages: report generation, suppression lookup, message handoff, and outcome ingestion. The first can be healthy while the last is dead. This is why I distrust a single delivery dashboard; it compresses a distributed workflow into one number and rarely answers which seller report is now unsafe to retry.
Open tracking does not repair this blind spot. Apple Mail Privacy Protection downloads remote content in the background, so an apparent open is not dependable proof that a person saw a report. For this workflow, stronger deliverability signals are request creation, preflight eligibility, handoff acceptance, definitive bounce classification, complaint receipt, and an explicit in-application report download.
Measure those.
Put one gate directly in the send path
The safe implementation has a local, transactional suppression decision close to the outbox. Do not fetch a provider-wide suppression list on every send; that adds a remote dependency to the critical path and still leaves a race between the lookup and the handoff. Instead, continuously project outcome events into a table owned by the application, then read that table in the same database transaction that claims an outbox item.
A minimal record needs more than an email string. Store the normalized recipient hash or protected address reference, the marketplace account ID, the scope, the reason, the effective time, the source event ID, and the latest observed source position. Scope matters because a marketing opt-out and an operational report are different consent questions, while a permanent invalid-mailbox result usually applies across message categories. Legal and policy review must define those scopes; code should not infer them from a convenient boolean.
The ordering rule is simple: a suppression update with a newer effective time wins, duplicate source event IDs do nothing, and an operator cannot silently clear a permanent failure without an auditable reason. Keep raw events long enough to replay the projection under the organization's retention policy. The derived table is disposable. The audit trail is not.
This Go example shows the boundary, even if the surrounding application uses another runtime. It deliberately omits provider-specific payloads and attachment construction; the important part is that eligibility and outbox creation share one transaction.
package mailflow
import (
"context"
"database/sql"
"errors"
"fmt"
)
var ErrSuppressed = errors.New("recipient is suppressed")
type ReportRequest struct {
SellerID string
RecipientKey string // Stable lookup key; do not log the raw address.
ReportID string
AttachmentRef string
}
func QueueReport(ctx context.Context, db *sql.DB, r ReportRequest) error {
tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
if err != nil {
return err
}
defer tx.Rollback()
var reason string
err = tx.QueryRowContext(ctx, `
SELECT reason
FROM email_suppressions
WHERE recipient_key = ? AND scope IN ('all', 'transactional')
LIMIT 1`, r.RecipientKey).Scan(&reason)
if err == nil {
return fmt.Errorf("%w: %s", ErrSuppressed, reason)
}
if !errors.Is(err, sql.ErrNoRows) {
return err // Unknown eligibility fails closed.
}
_, err = tx.ExecContext(ctx, `
INSERT INTO email_outbox
(idempotency_key, seller_id, recipient_key, report_id, attachment_ref, state)
VALUES (?, ?, ?, ?, ?, 'ready')
ON CONFLICT (idempotency_key) DO NOTHING`,
"seller-report:"+r.ReportID, r.SellerID, r.RecipientKey,
r.ReportID, r.AttachmentRef)
if err != nil {
return err
}
return tx.Commit()
}
The query syntax for placeholders and conflict handling varies by database driver, but the invariant does not: no worker should be able to create a second logical send for the same report, and no eligible send should bypass the most recent committed suppression state. The attachment belongs in durable object storage behind a short-lived fetch mechanism or in the queued MIME payload, according to the sender boundary; an ephemeral local filesystem path is not a durable reference.
Reconcile push events with a boring poller
Event delivery needs two lanes. The low-latency lane validates an authenticated webhook, stores the unmodified event, acknowledges only after durable persistence, and updates the suppression projection asynchronously. The recovery lane polls an event feed or export from the last durable cursor and inserts through the exact same deduplication path. Polling is not the primary transport. It is the audit mechanism for gaps.
Keep both lanes.
No source event ID should cause two state transitions. Use a uniqueness constraint, not an in-memory cache, because process restarts and concurrent consumers are normal. If the source offers no stable ID, derive a deterministic key from immutable fields and document the collision assumptions. Do not use arrival time as identity.
Classification deserves restraint. A permanent invalid-recipient result can create an all suppression. A transient 4.X.X SMTP condition belongs in a bounded retry policy with jitter and an expiry, not in a permanent suppression row. Complaints should stop applicable mail immediately. Unsubscribe events should update the scope expressed by that request, and senders covered by Google's bulk-sender requirements should implement the prescribed one-click unsubscribe mechanism for marketing or subscribed messages; a transactional report must not be relabeled as promotional merely to reuse a list pipeline.
The reconciliation alert should compare watermarks: newest source event time, newest locally persisted event time, and newest successfully projected event time. Alert on sustained lag and on a cursor that does not advance while sends continue. Page on risk, not traffic. A quiet webhook endpoint at night may be normal; thousands of new outbox rows paired with a frozen outcome cursor are not.
Verify the failure path before trusting the happy path
Deploy the gate in observe-only mode first. Record what it would block, without sending extra messages, then compare those decisions with the existing suppression source. Differences need an owner and a reason. A raw count is useless if it cannot distinguish expected propagation delay from a recipient-key normalization bug.
Before enforcement, run a small test matrix:
| Condition | Expected send decision | Expected operational evidence |
|---|---|---|
| No suppression, fresh cursor | Queue once | Outbox row and idempotency key |
| Permanent recipient failure | Block future reports | Suppression reason and source event ID |
| Transient SMTP failure | Retry within policy | Attempt history, next attempt, expiry |
| Duplicate outcome event | No state change | Deduplication hit |
| Poll cursor is stale | Hold or degrade by policy | Lag alert tied to affected sends |
| Attachment generation fails | Do not hand off email | Failed report job, no ready outbox row |
Exercise out-of-order events as well. Deliver an older transient result after a newer permanent failure and prove that the recipient remains suppressed. Stop the webhook consumer, create test outcomes, restart it, and confirm that polling closes the exact sequence gap without duplicating transitions. Then restore from a snapshot into an isolated environment and rebuild the projection from raw events. A backup that has never reconstructed the table is only a belief.
The service-level indicators should expose the pipeline, not decorate it: eligible reports queued, suppressed reports blocked by reason, accepted handoffs awaiting outcomes, permanent failures after handoff, event-ingestion lag, projection lag, and report availability through the marketplace UI. Break them down by message class and sending stream, while keeping recipient addresses out of labels and logs.
Roll back sends without rolling back evidence
There are two useful rollback switches. One pauses new report handoffs while allowing event ingestion and projection to continue. The other disables enforcement and returns to observe-only mode while preserving every would-block decision. Neither switch should delete suppressions, reset cursors, or discard raw events.
If reconciliation falls behind, pause only the affected stream when the boundary is known; otherwise hold all generated report emails and keep reports downloadable in the authenticated marketplace interface. This favors delayed notification over repeated delivery to a known bad destination. The trade-off is visible and reversible.
After recovery, advance from the last verified cursor, drain outcomes before the send backlog, rerun eligibility for every held outbox item, and retain the original idempotency key. Do not release the backlog merely because the webhook health check turned green. The incident ends when the suppression projection is current and each queued report has been re-evaluated against it.
That is the decision rule worth putting in a runbook: fail closed on unknown eligibility, preserve evidence, and make recovery replayable. A generated report is useful only if its notification path respects the recipient's latest state; acceptance graphs and open pixels cannot establish that on their own.
Top comments (0)