Short answer: put each signup verification request in a transactional outbox, let a queue worker claim small batches, and page on the age of unprocessed intent rather than waiting for email or SMS delivery reports. Keep an append-only attempt record beside that flow. This gives an operator evidence of what the system intended, attempted, and observed without pretending that a provider acceptance means a person received or opened the link.
The page I care about reads something like oldest_ready_verification_seconds > 120, split by channel and deployment. It does not say "email is down." The on-call sees the oldest eligible row, the last successful claim time, recent attempt outcomes, and whether the cron reconciler is advancing. That is enough to decide whether work is accumulating before shoppers start retrying signup.
I've been paged by both missed jobs and duplicate deliveries. They look like opposite failures, but they usually expose the same design mistake: treating "run this sender" as the durable fact instead of recording a notification state transition. For a verification link, the audit question arrives later and is blunt: what did we decide to send, which worker claimed it, what changed, and can we show that without retaining the secret itself?
Incident timeline: read the verification alert backward
Start at the alert and work backward. A useful page proves that eligible work is getting older while the system is expected to process it. A weak page counts failed calls. Failure counts can stay at zero when no worker is polling, when a scheduler stopped enqueueing, or when every process is healthy but looking at the wrong queue.
For an e-commerce signup, I would expose four signals from system-owned records:
| Signal | What it answers | Operator action |
|---|---|---|
| Oldest eligible intent age | Is any verification request waiting too long? | Inspect claiming and queue progress |
| Ready intent count | Is delay isolated or growing? | Compare arrival and completion rates |
| Attempts by outcome and channel | Where did processing stop? | Inspect a bounded set of attempt records |
| Reconciler watermark age | Is the safety sweep still moving? | Check cron ownership and its last completed range |
The word "eligible" matters.
A row scheduled for ten minutes from now should not age the alert. Neither should an intent already terminally suppressed by policy. The query behind the page needs to use the same eligibility predicate as the worker; if dashboards invent a second definition, the chart and the queue will disagree exactly when the on-call needs them to agree.
I don't know your required evidence-retention window. Legal and security owners have to set it. The operational schema can still make that decision explicit: retain identifiers and timestamps according to policy, store a digest or reference for the verification token rather than the token, and restrict access to destination data. An event log is evidence only if its fields have defined meaning and its access is controlled; collecting more personal data does not automatically make the record stronger.
How should a Node.js queue worker batch email and SMS notifications?
Let the Node.js signup handler write the account change and a notification intent in one database transaction. A worker may be implemented in another runtime without changing that contract. The Go sketch below shows the important boundary: claim a bounded batch, create a stable attempt ID, call a channel adapter, and commit the observed outcome. ClaimReady must use a database-supported concurrency strategy so two workers cannot own the same ready row at once.
package notifications
import (
"context"
"time"
)
type Intent struct {
ID string
Channel string
Destination string
Template string
TokenRef string
}
type Receipt struct {
ProviderRef string
AcceptedAt time.Time
}
type Store interface {
ClaimReady(ctx context.Context, limit int, lease time.Duration) ([]Intent, error)
BeginAttempt(ctx context.Context, intentID, attemptID string, at time.Time) error
MarkAccepted(ctx context.Context, intentID, attemptID string, receipt Receipt) error
ReleaseForRetry(ctx context.Context, intentID, attemptID string, next time.Time, reason string) error
}
type Sender interface {
Send(ctx context.Context, intent Intent, idempotencyKey string) (Receipt, error)
}
func ProcessBatch(ctx context.Context, store Store, senders map[string]Sender, now time.Time) error {
intents, err := store.ClaimReady(ctx, 50, 30*time.Second)
if err != nil {
return err
}
for _, intent := range intents {
attemptID := intent.ID + ":initial"
if err := store.BeginAttempt(ctx, intent.ID, attemptID, now); err != nil {
return err
}
receipt, err := senders[intent.Channel].Send(ctx, intent, attemptID)
if err != nil {
if releaseErr := store.ReleaseForRetry(ctx, intent.ID, attemptID, now.Add(time.Minute), "send_rejected"); releaseErr != nil {
return releaseErr
}
continue
}
if err := store.MarkAccepted(ctx, intent.ID, attemptID, receipt); err != nil {
return err
}
}
return nil
}
This is deliberately not a promise of exactly-once delivery. A process can lose contact after an external service accepts a request but before MarkAccepted commits. The stable attempt ID gives an adapter an idempotency key where that capability exists, while the local attempt history tells the reconciler what remains ambiguous. Where a channel cannot deduplicate, the product decision may be to delay retrying an ambiguous attempt rather than risk sending two live verification messages. That choice belongs in policy, not in a generic retry library.
Don't batch unrelated state changes into one all-or-nothing send call merely because the transport permits bulk requests. Database claiming can be batched for efficiency while every intent retains its own attempt and outcome. One malformed destination should not erase evidence for the other 49 records, and a batch-level response must be expanded into per-intent observations before the lease is released.
Email and SMS also need separate content controls. DKIM defines a domain-level signing mechanism for email and describes verification of that signature; it does not prove that a recipient read a message. SMS length depends on encoding and segmentation: Twilio's reference documents 160 GSM-7 characters or 70 UCS-2 characters for a single segment, with lower per-segment limits for concatenated messages. A signup template test should therefore check the rendered text and encoding, especially after localization. A single curly quote can change the encoding assumption.
Small details bite.
Compliance model: acceptance is not delivery evidence
By the time a delivery-receipt alarm fires, the system may already have held signup requests for several minutes. Instrument the records you own first. On each polling cycle, record the number claimed, claim latency, completion count, retry count, and the age of the oldest eligible intent. Emit the deployment identity and channel as low-cardinality dimensions; keep intent IDs in logs or traces where operators can query them, not as metric labels.
The reconciler is a cron-triggered safety path, not a second sender. It searches for expired leases, ambiguous attempts, and ready records older than the normal polling horizon, then moves each record according to the same state machine used by the worker. If both paths can send independently, the recovery mechanism becomes a duplicate generator. I use a state vocabulary that makes uncertainty visible: ready, claimed, attempting, accepted, retryable, and suppressed. accepted means the channel adapter returned a positive submission result and nothing more. Avoid a bare sent state unless the team can say precisely which observation it represents. For compliance review, write the definition next to the schema and version the notification template; otherwise an old row cannot explain what content rules applied at the time. A compact attempt event might contain intent_id, attempt_id, channel, template_version, claimed_at, attempted_at, observed_at, outcome, and a provider reference when one exists. It should not contain the verification token. The account or destination can be referenced through an access-controlled identifier, allowing an investigator to correlate records without copying sensitive values into every log sink. Test the trace as a system: pause worker polling in a staging environment, insert eligible intents, and verify that the age signal crosses the proposed threshold; resume workers and confirm it clears; then force a worker to stop after claiming but before completing, wait for lease expiry, and verify that reconciliation produces one new state transition under the chosen ambiguity policy. These are controlled tests, not production anecdotes, and their expected event sequence should be stored with the runbook.
Failure policy: test ambiguous retries before release
Classify outcomes before choosing delay. Invalid destinations and policy suppression are terminal. Explicit transient rejections can be scheduled with bounded backoff. Ambiguous outcomes need their own branch because immediate retry trades lower latency for a higher chance of duplicate messages. A verification flow should also make old links invalid according to the account policy, so delivery of an older attempt cannot silently restore an obsolete credential path.
The catch is that an outbox plus polling adds database load, lease logic, and a reconciler that somebody must operate. It is not suitable when a team cannot own those state transitions or test recovery. In that case, stick with a managed queue whose delivery and retention semantics meet the evidence requirement, but keep the application-level intent and attempt IDs; transport history alone cannot describe why the application chose a channel or template.
There is another trade-off. Email-first with SMS fallback can reduce unnecessary SMS traffic, but it increases verification latency and turns fallback timing into user-visible policy. Sending both channels immediately lowers dependence on either path while creating duplicate prompts and more destination data to govern. I would make that choice from the signup risk model and consent rules, then encode it in the intent record. I wouldn't bury it in a worker timeout.
Runbook review: account for an early page's false-positive cost
Work backward from the maximum acceptable time for a fresh verification link to become usable. Reserve time for claiming, one normal attempt, any permitted retry, and on-call reaction. The page threshold belongs before that budget is exhausted. Use separate warning and paging behavior only when each leads to a different action; two colors on the same unactionable chart buy nothing.
Then watch the false positives.
A threshold below normal batch jitter teaches responders to ignore the page, while a threshold above the verification deadline reports failure after the shopper has already retried. Your mileage may vary across regions and signup peaks, so review the observed age distribution after deployment and record why the threshold changed. Do not auto-tune away a sustained shift without checking whether arrival rate, worker capacity, or the eligibility query changed.
The final runbook should begin with the same questions the evidence model can answer: Is ready work aging? Is one channel isolated? Are claims progressing? Is reconciliation advancing? Which attempt states are ambiguous? If the dashboard cannot answer those questions, add the instrumentation before adding another alert.
Top comments (0)