Short answer: run a scheduled, cursor-based delivery reconciler that records raw provider event identifiers, normalizes terminal outcomes, and suppresses an address only after a durable bounce decision; page on reconciliation lag or evidence gaps, not on every individual bounce. For a fintech welcome-email flow, that is the least complex design that still produces an audit trail when webhook delivery is unavailable.
The page fires at 02:17: email_reconciliation_lag_seconds > 900. On-call sees 1,842 accepted welcome messages, 1,799 terminal outcomes, 31 still inside the delivery window, and 12 records with no fresh observation. The immediate action is not to resend all 43 unresolved messages. It is to stop the next batch from outrunning the evidence pipeline, inspect the poller's last successful cursor, and determine whether the gap is provider latency, a scheduler miss, or a local persistence failure.
That distinction matters. A bounce is a recipient outcome. A missing observation is an observability outcome. Treating them as the same state can create duplicate mail, suppress a valid customer, or leave an invalid address eligible for another regulated communication.
What should a Node.js cron poll from an email events API without webhooks?
Poll an append-only or time-ordered event feed if the provider exposes one, using a stored cursor or a half-open time window with overlap. A Node.js cron process should ask for events after the last committed checkpoint, retain the provider's stable event and message identifiers, and advance the checkpoint only in the same durable transaction that stores the observations. If the API offers only per-message status reads, keep a local work queue keyed by provider message ID and poll only records that are still non-terminal.
The state model needs more care than the scheduler. accepted, delivered, bounced, and unknown are useful internal categories, but the raw provider value must remain beside the normalized value because compliance evidence should be reproducible after mapping rules change. An accepted handoff is not proof of inbox delivery. A hard bounce can justify suppression; a temporary outcome usually needs a retry policy and an expiry horizon. The exact classification vocabulary varies by provider, so confirm it against the provider's current documentation rather than guessing from a string.
Do not use page number as a checkpoint. Inserts during pagination can shift page boundaries, while a crash between reading page 7 and writing its results can lose or duplicate evidence. Duplicate reads are fine if writes are idempotent. Lost reads aren't.
Implementation: commit evidence before advancing the cursor
The job can be modeled as one small interface. The production Node.js implementation can use its normal cron library and HTTP client, but it should preserve these semantics: bounded batches, explicit deadlines, durable cursor commits, and idempotent event inserts. The Go example makes those invariants visible without tying the design to an SDK.
package reconciler
import (
"context"
"errors"
"time"
)
type Event struct {
ID string
MessageID string
Kind string
Observed time.Time
}
type Page struct {
Events []Event
NextCursor string
}
type EventSource interface {
List(ctx context.Context, after string, limit int) (Page, error)
}
type EvidenceStore interface {
Cursor(ctx context.Context) (string, error)
CommitPage(ctx context.Context, events []Event, nextCursor string) error
}
func Reconcile(ctx context.Context, source EventSource, store EvidenceStore) error {
cursor, err := store.Cursor(ctx)
if err != nil {
return err
}
for pages := 0; pages < 20; pages++ {
page, err := source.List(ctx, cursor, 100)
if err != nil {
return err
}
if len(page.Events) == 0 {
return nil
}
if page.NextCursor == cursor {
return errors.New("event cursor did not advance")
}
if err := store.CommitPage(ctx, page.Events, page.NextCursor); err != nil {
return err
}
cursor = page.NextCursor
}
return nil
}
Twenty pages of 100 events is a capacity guard, not a throughput claim. Size both values from measured arrival rate, API quotas, database write latency, and the maximum tolerable reconciliation lag. If a five-minute schedule can ingest less than five minutes of peak traffic, the queue has negative recovery capacity and will never catch up after a missed run. That's arithmetic, not an alerting problem.
A useful page names an action. bounce_count > 0 does not; bounces are expected in any real recipient population. reconciliation lag exceeds the evidence SLO does, because on-call can inspect a cursor and either restore processing capacity or pause downstream sends. The primary service-level indicator is the age of the oldest accepted message that has neither a terminal outcome nor an explicit unexpired pending state. A second indicator measures the fraction of polled events that cannot be normalized, and a third measures suppression propagation delay from terminal bounce observation to the send-time eligibility check.
The earlier signal should have been backlog growth. Instrument accepted messages, terminally reconciled messages, poll duration, pages consumed, events inserted, duplicate events ignored, cursor age, normalization misses, and suppression updates. Keep provider response bodies out of high-cardinality metric labels; identifiers and recipient data belong in access-controlled evidence records and trace links, not a metrics index.
One long-lived dashboard should answer a capacity question: at the current peak arrival rate, how many minutes does the worker need to clear one minute of new events? If the ratio reaches 1.0, alert before the lag SLO is consumed. This catches a slow database, a reduced API quota, and a schedule that silently overlaps itself.
Use a lease or scheduler-level concurrency policy so only one job owns a cursor partition. A second worker may improve capacity, but only after the partition key and commit order are explicit. Blind concurrency creates a deceptively healthy request rate while checkpoints race.
The suppression table is a control plane, not a cache. Store a pseudonymous recipient key, normalized reason, raw event reference, observed time, policy version, and decision time. The send path should check it before submitting another welcome or transactional message. Keep the raw address only where delivery actually requires it, under the retention and access controls appropriate to the system. A minimal decision rule is deliberately conservative: a recognized permanent bounce creates a suppression record; a transient status stays pending until its retry or observation horizon closes; an unrecognized event is quarantined for mapping review and does not silently become a permanent suppression. I'm not sure a universal retry horizon exists, because provider semantics, message urgency, and the organization's compliance interpretation differ. Resolve that locally with documented policy ownership and a tested mapping table. The FTC's CAN-SPAM guide says commercial email must provide a clear opt-out mechanism and honor opt-out requests within 10 business days. A bounce suppression list and a marketing opt-out list therefore solve different problems and should retain different evidence. Do not infer consent from successful delivery, and do not erase an opt-out merely because an address later appears valid. This separation also gives an auditor a clean answer when one address appears in both datasets: the delivery control records why further attempts are operationally invalid, while the opt-out control records the recipient's instruction, and neither record is allowed to overwrite the other's provenance.
Short version: suppression needs provenance.
Managed API vs self-hosted delivery for evidence custody
Managed event APIs reduce transport and provider-integration work, while a local reconciler keeps policy, evidence, and send-time enforcement inside the platform boundary. Self-hosting more of the mail path can increase control, but it also transfers queue durability, reputation operations, security patching, and on-call ownership to the team. The decision belongs in a capacity and ownership review, not a feature-count contest.
| Approach | Compliance evidence | On-call load | Lock-in boundary | Not suitable when |
|---|---|---|---|---|
| Managed delivery plus local reconciliation | Local immutable decisions linked to external event IDs | Moderate | Provider event vocabulary and identifiers | The API cannot supply stable IDs or sufficient history |
| Managed delivery plus provider-hosted history | Evidence remains mainly in the provider console or export | Lower initially | Retention, export format, and console access | Auditors require independently retained decision records |
| Self-hosted delivery and event pipeline | Full local custody if engineered correctly | High | Mail-transfer and reputation expertise | The team cannot staff continuous delivery operations |
| Dual-provider abstraction | Local canonical model across sources | High during integration | Canonical model and lowest-common-denominator features | Volume or risk does not justify duplicated integration work |
The catch is that abstraction does not remove provider semantics. It moves them into adapters and tests. Stick with one managed integration when the team can export adequate evidence, its availability objectives fit the product SLO, and migration risk is lower than the permanent cost of maintaining a common model. Choose a local control plane when suppression decisions must be enforced consistently across multiple sending paths or retained independently of provider history.
Before signing a contract, test cursor stability under concurrent inserts, duplicate event behavior, history retention, rate-limit signaling, export completeness, authentication rotation, and deletion workflows. Resend, Amazon SES, SendGrid, and Postmark are examples of products a team might evaluate, but their current event models and retention terms must be verified in their official documentation at decision time. Product selection is downstream of the evidence requirements.
Failure modes: tune the threshold before the page becomes noise
Deploy the poller in shadow mode first: write evidence and compute suppression decisions without enforcing them, then compare its terminal-state coverage and decision timing with the existing operational record. Exercise duplicate pages, a crash before commit, a crash after commit, an empty page, an unknown event kind, a scheduler overlap, and a backlog larger than one run's cap. The invariant is simple: replay can duplicate work but cannot change the final decision or skip an event.
Set the lag objective from the business consequence. A welcome message may tolerate a different reconciliation delay from a password-reset message, while suppression propagation should complete before the next eligible send. Alerting at 60 seconds because the cron expression runs every minute is usually noise; the threshold must include ordinary provider observation delay, job jitter, and processing time, with remaining budget for recovery.
False positives have a real cost. A threshold set too close to normal latency pages on-call during harmless variation, encourages broad mutes, and hides the later incident that actually consumes the evidence SLO. A threshold set too wide permits the invalid-recipient queue to grow and delays suppression. Track the distribution, review the budget after traffic or quota changes, and page only when a human action can still protect the objective.
No magic number.
Top comments (0)