The page says receipt_delivery_stale: 1,842 settled orders are waiting, the oldest for 11 minutes, while the email and SMS provider dashboards still look mostly green. The on-call can see the backlog but can't yet answer the useful questions: Were recipients filtered by preferences? Did a suppression list remove them? Did a batch leave the system? Is status polling merely behind?
Short answer: treat a settled payment as an immutable event, expand it through a Postgres worker with keyset pagination, apply recipient preferences and suppression before creating channel-specific attempts, then poll only unresolved attempts; page on the age of eligible receipts, not on raw send failures.
That design works when the event producer is Node.js even if the worker example below is Go. The language boundary should be a durable row or message, not an in-process SDK object. Integration effort stays bounded because the payment path records one event and leaves delivery policy, batching, retries, and provider adapters outside the checkout request.
Don't page on a provider error counter alone.
How can a Postgres worker implement batch email and SMS pagination, recipient preferences, and suppression lists?
Use a stable, monotonic cursor such as (settled_at, order_id), and persist the last committed cursor only after every eligible recipient in that page has either produced an attempt row or received a durable skip reason. Offset pagination looks convenient, but a live table changes under the worker; a cursor makes the boundary explicit and gives the on-call something concrete to inspect.
The worker should read a bounded page, join or fetch the recipient's current channel preferences, check the relevant suppression set, and create one logical attempt per channel. Keep skipped_preference, skipped_suppression, and queued distinct. A single not_sent state destroys the evidence needed to distinguish an intended policy decision from work that never entered a batch.
Here is the core shape. LoadPage is backed by a keyset query ordered by settlement time and order ID; RecordDecision and cursor advancement belong in one database transaction. The provider calls happen after commit, from attempt rows, so a slow network can't hold payment or preference locks.
package receipts
import (
"context"
"time"
)
type Cursor struct {
SettledAt time.Time
OrderID string
}
type Receipt struct {
OrderID string
RecipientID string
SettledAt time.Time
}
type Preference struct {
Email bool
SMS bool
}
type Store interface {
LoadPage(context.Context, Cursor, int) ([]Receipt, error)
Preferences(context.Context, string) (Preference, error)
Suppressed(context.Context, string, string) (bool, error)
RecordDecision(context.Context, Receipt, string, string) error
CommitCursor(context.Context, Cursor) error
}
func ExpandPage(ctx context.Context, store Store, after Cursor, limit int) (Cursor, error) {
rows, err := store.LoadPage(ctx, after, limit)
if err != nil {
return after, err
}
next := after
for _, receipt := range rows {
pref, err := store.Preferences(ctx, receipt.RecipientID)
if err != nil {
return after, err
}
for channel, enabled := range map[string]bool{"email": pref.Email, "sms": pref.SMS} {
decision := "queued"
if !enabled {
decision = "skipped_preference"
} else {
blocked, err := store.Suppressed(ctx, receipt.RecipientID, channel)
if err != nil {
return after, err
}
if blocked {
decision = "skipped_suppression"
}
}
if err := store.RecordDecision(ctx, receipt, channel, decision); err != nil {
return after, err
}
}
next = Cursor{SettledAt: receipt.SettledAt, OrderID: receipt.OrderID}
}
if err := store.CommitCursor(ctx, next); err != nil {
return after, err
}
return next, nil
}
In production, make decision insertion idempotent with a uniqueness rule over the event, recipient, and channel. Then a worker that loses its lease after writing 99 of 100 receipts can replay the page without creating a second customer message. The cursor is a throughput tool, not the source of correctness; the uniqueness rule carries that burden.
Batch size needs capacity planning rather than folklore. Start with the database transaction budget and the channel adapter's accepted request size, then load-test page sizes against the SLO. A page of 500 might be fine for a narrow indexed read and poor for a preference join with cold data. I'm not sure any fixed number survives different schemas, which is why page duration, rows expanded, and lock wait should be recorded together.
Trace backward through the notification pipeline
The final customer-facing objective is straightforward: an eligible order receipt should reach a terminal channel outcome within the delivery window after payment settles. “Eligible” matters. A customer who disabled SMS, or an address present on the email suppression list, should not count as a delivery failure. The service-level indicator therefore needs a denominator built from policy decisions, not every settled order.
The page at 11 minutes is late evidence. Work backward. Before the oldest eligible receipt breaches the window, the system can observe rising expansion lag: current time minus the oldest unexpanded settled_at. Before expansion lag rises, it can observe worker saturation through page duration and the ratio of claimed work to completed decisions. Before a provider status backlog becomes customer-visible, it can observe the age and count of unresolved attempts by channel.
This distinction changes the action attached to each alert. High expansion lag sends the on-call to Postgres worker concurrency, query plans, and lease ownership. Old queued attempts point to the dispatcher or its adapter. Old accepted attempts point to status polling. A spike in skipped_suppression is a policy signal to investigate, but it isn't proof that receipts failed. One aggregate “notification errors” chart sends everyone to the loudest dashboard and burns the first ten minutes.
Consider an illustrative window with 1,200 settled payments and a 200-row page limit. If 80 recipients have disabled both channels and 20 more are suppressed for email, the expansion stage should expose those decisions rather than report 2,400 expected sends. Suppose the worker has committed decisions through order 800, the dispatcher has submitted attempts through order 600, and polling has terminal results through order 400. Those cursor positions describe three different queues. Adding workers to the poller won't reduce the 400-order expansion gap; increasing expansion concurrency won't resolve accepted attempts whose next-check time is in the future. The useful dashboard puts each clock beside the receipt objective, and the useful runbook names which concurrency or deployment controls move that clock. These numbers are arithmetic for the trace, not a proposed threshold or benchmark.
Page sooner.
The early warning should be a burn-rate or age signal tied to the receipt SLO, with enough labels to separate email from SMS but not enough to create a series for every order. Order IDs belong in logs and traces. Metrics need low-cardinality states: unexpanded, queued, accepted, delivered, failed_terminal, skipped_preference, and skipped_suppression. The exact terminal vocabulary can follow each adapter, but the internal meanings must stay stable when a provider changes.
Instrumentation also needs a traceable event key. Carry the order event ID into the decision row, attempt row, structured log, and polling task. Avoid putting email addresses or phone numbers into metric labels. The on-call should be able to start with a backlog sample, find its attempt, see the last provider status timestamp, and determine the next retry time without querying three unrelated schemas.
Keep those clocks separate.
Give status polling its own reliability budget
Batch submission answers “did the provider accept this work?” Status polling answers “what terminal result has the provider reported?” Combining those questions in one worker encourages long transactions and ambiguous retries. Create attempts first, submit bounded batches second, and schedule polling only for attempts whose internal state is nonterminal.
Polling needs its own budget. Use a next-check timestamp and increasing intervals, cap the number claimed per pass, and stop after the attempt becomes terminal or crosses the product's explicit delivery horizon. If a provider supports callbacks, treat them as another status input rather than a reason to delete the poller immediately; the important invariant is that state transitions are idempotent and cannot move backward from terminal to pending.
The order receipt itself should remain channel-neutral. Rendering belongs near each adapter because email and SMS have different size, formatting, and abuse constraints. Postmark's transactional email guide is useful background for the email side, while Twilio's SMS pumping guidance is a reminder that bulk SMS entry points need abuse controls. Neither source resolves the internal state model, pagination boundary, or recipient policy order; those remain platform decisions.
There is a catch: polling every accepted attempt can become most of the system's request volume during a large campaign or provider delay. Capacity-plan poll demand as unresolved attempts / average poll interval, reserve that capacity separately from submissions, and expose poll lag as its own signal. If callbacks are dependable for the chosen integration, use polling as reconciliation at a slower cadence. If callbacks aren't part of the contract, keep polling first-class and set the receipt SLO with that delay included.
Measure integration effort before assigning ownership
The buy-versus-build decision is less about the first API call than the permanent operational boundary. A small team can integrate a managed email and SMS API quickly, but it still owns preference semantics, suppression synchronization, idempotency, audit history, and the alert that connects a settled payment to a missing receipt. Self-hosting can move adapter behavior under the team's control while adding deliverability work and on-call surface. A broker plus channel adapters sits between those poles.
| Ownership model | Initial integration | Platform team owns | Poor fit when |
|---|---|---|---|
| Managed channel APIs | Two adapters plus status mapping | Event ledger, policy, suppression, retries, polling, SLOs | The team can't absorb separate channel contracts and status vocabularies |
| Broker with replaceable adapters | One internal contract plus adapter deployment | Broker, mappings, data retention, every operational handoff | There is no platform capacity to operate shared messaging infrastructure |
| Fully self-hosted delivery | Largest build and deployment scope | Transport, reputation controls, abuse controls, storage, paging, upgrades | Integration effort and on-call load dominate lock-in concerns |
For the order-receipt job, the conservative default is a durable internal event and thin managed adapters, because it keeps checkout isolated while avoiding ownership of the delivery transport. Stick with direct synchronous sends only when losing or delaying a receipt is explicitly acceptable and the payment endpoint's latency budget includes both channels. Choose a broker when multiple teams need the same policy and observability contract. Choose self-hosting only when control requirements justify the extra on-call load; it isn't a neutral escape from vendor lock-in because it replaces contract risk with operating risk.
The false-positive cost closes the loop. An alert threshold set below normal batch completion time pages during healthy bursts; responders learn to wait, and the alert stops producing action. A threshold set above the receipt objective reports the breach instead of preventing it. Measure the healthy distribution of expansion lag and polling lag, place a warning where added capacity can still recover inside the SLO, and page only when the runbook has a concrete lever such as worker concurrency, a paused deployment, or a constrained batch queue. Your mileage may vary — the threshold depends on arrival shape, database headroom, and the delivery window — but an alert without recovery time in its math is just a delayed status report.
Top comments (0)