A page saying "receipt backlog breached" is already late. TL;DR: persist one notification intent when payment settles, resolve each recipient's current preferences and suppression state, claim bounded pages with PostgreSQL workers, make provider submission idempotent, and poll only deliveries that remain unresolved. Alert first on the leading signals—oldest eligible intent age, claim rate, and provider acceptance latency—because a raw failure count can stay quiet while customers wait.
The target is not "send every message immediately." It is an explicit delivery SLO with controlled load: choose a receipt-age objective from business requirements, then size workers and downstream quotas to sustain the peak settlement rate plus recovery headroom. That distinction matters at 02:00, when an operator needs to know whether to add capacity, isolate one channel, or stop retries that are amplifying an external failure.
What should wake the on-call engineer?
The page should identify the affected channel and workload boundary, the age of the oldest eligible receipt, the number ready to claim, and whether acceptance latency or worker throughput moved first. A total queue-depth alarm is ambiguous: a large fresh batch may be healthy, while a small queue of old receipts may violate the customer promise. Age is the symptom closest to user impact.
Work backward one interval. The earlier warning should have been a burn-rate alert on the receipt-age SLO, supported by a saturation forecast such as eligible_intents / observed_claims_per_second. Set its window long enough to ignore one normal batch and short enough to leave operator response time before the SLO is consumed.
This is where false positives become an operational cost, not a cosmetic nuisance. A threshold that pages on every settlement burst trains responders to distrust the signal. A threshold based only on a long average hides a sharp stall. Replay known peak shapes in staging, require sustained evidence across several claim cycles before paging, and route a lower-severity warning earlier.
Noise wins otherwise.
How should a bulk event notification system batch email and SMS?
The payment transaction should create an immutable notification intent through an outbox record, or publish to a queue only after a durable database commit. The record needs a stable event ID, order ID, recipient reference, template version, channel candidates, creation time, and status. Keep sensitive address data behind the recipient reference where the architecture permits it. A worker can then crash after an external submission without losing the business event.
Exactly-once delivery is the wrong promise across a database and an external messaging service. Aim for at-least-once processing plus idempotent effects. Derive an idempotency key from the settled-payment event and channel; store each attempt before submission; and reconcile uncertain outcomes instead of immediately sending again. If the transport accepted the request but the worker lost the response, a blind retry can produce two receipts.
That gap is real.
Preferences are evaluated when an intent becomes eligible, not copied indefinitely into every queued row. An order receipt may be governed differently from marketing traffic, but that policy belongs in a versioned decision function reviewed by the business. The worker should receive only the result: allowed email, allowed SMS, locale, quiet-hours decision, and the suppression reason if blocked. Do not infer consent from the presence of a phone number.
Suppression is a state transition with provenance. Record whether an address was blocked by a hard bounce, complaint, invalid destination, recipient choice, or operator action, along with the source timestamp. Transactional email guidance emphasizes separating transactional traffic from bulk marketing concerns and protecting sending reputation. The practical consequence is that a newly suppressed destination must be checked immediately before submission, not merely when the batch was assembled.
Claim bounded pages without turning PostgreSQL into a queue fire
Use keyset pagination over a stable tuple such as (available_at, id). Offset pagination makes the database repeatedly walk earlier rows and can skip or duplicate work as statuses change. Multiple workers can claim a small page using row locks with SKIP LOCKED, write a lease in the same transaction, commit quickly, and perform network I/O afterward. Never hold a database transaction open across transport calls.
The Go sketch shows the boundary. Production code also needs transaction rollback, lease expiry, and per-channel concurrency limits.
type Intent struct {
ID int64
EventID string
RecipientID int64
Channel string
}
func claimPage(ctx context.Context, db *sql.DB, worker string, limit int) ([]Intent, error) {
tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted})
if err != nil {
return nil, err
}
defer tx.Rollback()
rows, err := tx.QueryContext(ctx, `
WITH picked AS (
SELECT id
FROM notification_intents
WHERE status = 'ready' AND available_at <= now()
ORDER BY available_at, id
FOR UPDATE SKIP LOCKED
LIMIT $1
)
UPDATE notification_intents AS n
SET status = 'leased', leased_by = $2,
lease_until = now() + interval '60 seconds'
FROM picked
WHERE n.id = picked.id
RETURNING n.id, n.event_id, n.recipient_id, n.channel`, limit, worker)
if err != nil {
return nil, err
}
defer rows.Close()
var intents []Intent
for rows.Next() {
var in Intent
if err := rows.Scan(&in.ID, &in.EventID, &in.RecipientID, &in.Channel); err != nil {
return nil, err
}
intents = append(intents, in)
}
if err := rows.Err(); err != nil {
return nil, err
}
return intents, tx.Commit()
}
Small pages reduce lease contention and bound the duplicate window after a crash; larger pages amortize database work. There is no honest magic number. Measure transaction duration, rows claimed per second, database CPU, downstream acceptance latency, and expired leases, then increase page size until one curve bends. Leave recovery capacity rather than sizing to the median.
This PostgreSQL worker pattern also has a clear limitation: it is not suitable when notification volume would make claim traffic contend with the application's primary transactional workload, or when routing must span regions without a single database authority. In that case, choose a dedicated durable log or queue for dispatch while keeping recipient preferences and suppression decisions behind a stable policy interface. The trade-off is more infrastructure and another consistency boundary; the benefit is independent scaling and failure isolation. At modest volume, the database pattern can be easier to operate because the outbox and claim state share one recovery model. Capacity evidence, rather than architectural fashion, should decide.
Poll status only when it can change a decision
Transport acceptance is not delivery. Normalize channel-specific results into a compact internal state machine: ready, leased, submitted, then a terminal delivered, failed, or suppressed. Preserve the raw external event separately for audit and later reprocessing. Authenticated callbacks should be the primary completion path where supported; polling is a repair loop for missing callbacks and uncertain submissions.
Poll by due time with keyset pagination, exponential backoff, jitter, and a terminal cutoff. A fresh submission might be checked sooner, while an old unresolved SMS should be checked less often. Stop when a terminal state arrives or when the internal retention policy expires; otherwise status polling quietly becomes the dominant outbound workload. Correlate every transition with the stable event ID, attempt ID, channel, and external message reference.
Retries require classification. Timeouts, connection resets, and explicit rate limits may be transient, but malformed destinations and permanent rejections are not. Retry only the former, honor a server-provided retry delay where the protocol exposes one, cap attempts, and move exhausted work to a reviewable dead-letter state. For SMS, destination-level rate limits and anomaly detection also constrain toll-fraud exposure; sudden geographic or prefix shifts should stop or challenge traffic before they consume the retry budget.
Capacity and integration choices belong on the same page
A receipt path has two rates: steady-state settlements per second and recovery throughput after an outage. If peak input is 300 intents per second and the system can safely process only 300, backlog never drains. Those numbers are illustrative, not a benchmark. Capacity planning should use measured channel mix, average attempts per intent, preference rejection rate, transport quotas, database claim rate, and the maximum tolerable recovery period.
| Decision | Managed transport | Direct channel integration | Self-hosted transport |
|---|---|---|---|
| Initial integration | A normalized submission and status contract can reduce application work | Separate email and SMS contracts expose channel detail | Team builds submission, status, and policy surfaces |
| On-call ownership | Boundary and escalation depend on the contract | Team owns adapters; transport owns the delivery edge | Team owns the data plane and abuse response |
| Lock-in pressure | Highest around normalized events and templates | Concentrated in each adapter and status model | Lower vendor dependency, higher operational coupling |
| Capacity work | Validate quotas, backpressure, and callback behavior | Validate each channel independently | Provision headroom, reputation controls, routing, and upgrades |
| Exit cost | Preserve a narrow internal interface and raw events | Replace one adapter at a time | Migrate operational state and delivery infrastructure |
Integration effort is lifecycle work. Count preference semantics, suppression ingestion, callback verification, status reconciliation, template deployment, observability, quota management, and incident escalation—not just the first successful request. A managed layer can shrink adapter code but widen the contract later replaced. Direct integrations expose more channel-specific behavior. Self-hosting buys control by putting reputation, abuse handling, upgrades, and 24-hour operations on the team's roadmap.
Define the internal contract first, then test each option against it with failure injection. Drop callbacks. Delay acceptance responses. Expire a lease after external acceptance. Insert a suppression between claim and send. A credible design can explain every resulting state without an operator editing rows by hand.
No manual repair.
The final dashboard should let the responder move from page to action in one screen: receipt-age SLO burn, eligible and leased counts, claim throughput, lease expiry, submissions and terminal outcomes by channel, callback lag, polling volume, retry reasons, and suppression counts. Keep cardinality bounded; event IDs belong in traces or logs, not metric labels. The best alert points to a constrained resource or failing boundary while there is still time to act.
Top comments (0)