The page says reset_delivery_deadline_at_risk. A media subscriber asked for a password reset, the link has a short expiry, and the SMS still has no terminal delivery result. The answer is to page on the remaining delivery budget, poll receipts through a durable state machine, and permit one idempotent email fallback before the reset expires. Retrying a send call until it returns success is not delivery reliability.
The on-call view needs the event ID, destination region, reset expiry, current channel state, last observation time, and next action. It should not expose the phone number, email address, or reset link. With those fields, the responder can distinguish a slow receipt from a stuck worker and can see whether fallback is still possible.
This is a deadline problem.
Alert on exhausted action budget, not transport failure
The bad page is “SMS failed.” It hides both urgency and action. A permanent failure with enough time for email is routine control flow; a pending receipt with almost no time left is the incident. The alert should therefore be derived from the reset deadline and the state machine's next eligible action, not directly from a provider error string.
I've been paged for missed jobs and duplicate deliveries. Those incidents leave the same lesson: an on-call engineer cannot repair an event whose ownership and transition history are ambiguous. Before the first outbound request, write an event record with an opaque event ID, expiry, policy region, and channel intents. Give one worker a time-bounded lease. Each observation then appends evidence and attempts a conditional state transition. Logs help explain the result, but the stored transition decides what may happen next.
Work backward one more step. The signal that should fire earlier is not a delivery failure; it is an exhausted action budget. Compute whether the system still has enough configured time to make the next poll, select fallback, submit email, and record the outcome before expiry. That calculation is local policy, so there is no universal number to copy. A newsroom with tightly controlled account recovery may choose differently from a streaming service. I'm not sure which threshold is right without the actual receipt-latency distribution and token-expiry policy; a replay of production-shaped, de-identified timing data would resolve it.
The instrumentation change is small but consequential: record state age and remaining deadline budget as separate values. State age answers “how long has SMS been pending?” Remaining budget answers “how long can we still act?” A single latency timer blends those questions and often pages too late.
Receipt age isn't enough.
Keep US and EU policy out of the worker
US and EU destinations can use the same state machine while selecting different policy records. Destination validation, sender configuration, consent, retention, and message content are policy inputs; they should not become if region == ... branches scattered through delivery code. Resolve a versioned policy when the reset is created and store that version with the event, so a mid-flight configuration change does not make the audit trail incoherent.
Transactional password-reset messages also need a firm boundary from marketing mail. RFC 8058 specifies one-click unsubscribe behavior for relevant email list traffic, but a password-reset workflow should not casually inherit bulk-campaign headers or suppression semantics. Classify the message purpose explicitly, have privacy and legal owners approve regional rules, and test the resolved policy as data. The RFC is a primary source for the unsubscribe mechanism; it is not a complete US-or-EU compliance playbook.
Avoid embedding the reset credential in telemetry. Use a salted recipient reference for correlation, restrict raw transport payload access, and set retention according to the applicable policy. The operational dashboard needs timing and state, not the secret that grants account access.
How should urgent event notifications poll SMS delivery before email fallback?
Model the workflow as observations and guarded transitions. sms_requested may become sms_delivered, sms_terminal_failure, or remain pending. Only a terminal failure or a policy deadline allows the email intent to be claimed. A late SMS receipt can still be stored after escalation, but it must not reopen the event or trigger another message.
Polling is evidence collection, not proof that a person read the text. Twilio's SMS documentation, for example, describes message status and status callbacks; those mechanisms report progress through a delivery system, not human attention. Keep the provider's raw value for investigation and map it to a deliberately small internal vocabulary. The mapping belongs at the adapter boundary. This prevents a vendor-specific status from leaking into retry policy.
The order of operations matters more than the loop syntax:
- Persist an SMS intent and stable idempotency key.
- Let a transport adapter submit or recover that intent.
- Poll or consume a callback, then store the raw and normalized observation.
- Claim the next transition with a compare-and-set operation.
- Persist an email intent before submitting the fallback.
Don't hold a database transaction open while calling a remote transport. Persist intent, commit, perform the call, and record the observation under the same key. If a worker stops between the remote acceptance and the local write, the next worker must recover the existing intent rather than invent a new one. The transport contract needs to define how that recovery works. Without it, “retry” is just another word for “possibly duplicate.”
Callbacks and polling can coexist. A callback provides a prompt observation when it arrives; bounded polling covers a missing or delayed callback. Both feed the same transition function, and both may race. The database guard absorbs that race. Whichever observer commits first advances the state; the other records evidence without repeating the side effect.
Make the durable claim the unit of retry
The following Go example concentrates on concurrency and deadlines. The durations are injected policy values, not universal recommendations. Storage and transport are interfaces so tests can stop a worker at every boundary without contacting a commercial service.
package resetnotify
import (
"context"
"errors"
"time"
)
type SMSState string
const (
SMSPending SMSState = "pending"
SMSDelivered SMSState = "delivered"
SMSTerminal SMSState = "terminal_failure"
)
var ErrClaimLost = errors.New("notification transition already claimed")
type Event struct {
ID string
PhoneToken string
EmailToken string
ExpiresAt time.Time
SMSState SMSState
EmailRequested bool
}
type Store interface {
Load(context.Context, string) (Event, error)
RecordSMSObservation(context.Context, string, SMSState, string, time.Time) error
ClaimEmailIntent(context.Context, string, string) error
}
type Transport interface {
PollSMS(context.Context, string) (SMSState, string, error)
SubmitEmail(context.Context, string, string, string) error
}
type Policy struct {
PollEvery time.Duration
SMSBudget time.Duration
FallbackReserve time.Duration
}
func Advance(ctx context.Context, store Store, tx Transport, eventID string, p Policy) error {
event, err := store.Load(ctx, eventID)
if err != nil {
return err
}
smsDeadline := time.Now().Add(p.SMSBudget)
latestFallback := event.ExpiresAt.Add(-p.FallbackReserve)
for time.Now().Before(smsDeadline) && time.Now().Before(latestFallback) {
state, raw, pollErr := tx.PollSMS(ctx, event.ID)
if pollErr == nil {
if err := store.RecordSMSObservation(ctx, event.ID, state, raw, time.Now()); err != nil {
return err
}
if state == SMSDelivered {
return nil
}
if state == SMSTerminal {
break
}
}
timer := time.NewTimer(p.PollEvery)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
key := "reset-email:" + event.ID
if err := store.ClaimEmailIntent(ctx, event.ID, key); err != nil {
if errors.Is(err, ErrClaimLost) {
return nil
}
return err
}
return tx.SubmitEmail(ctx, key, event.EmailToken, event.ID)
}
There are two intentional sharp edges in this compact example. First, the caller must validate that every duration is positive and that the fallback reserve leaves usable time before expiry. Second, the email claim and its eventual outcome need separate stored states in a full worker; a claimed intent is not the same as an accepted message. The code does not retry email inline because doing so would hide ownership from the scheduler. A queue should schedule the next attempt, retain the same key, and stop when the event deadline or configured attempt policy is reached.
Test transitions, not happy-path function calls. Use a fake clock and pause execution immediately before and after ClaimEmailIntent. Deliver the same SMS observation twice. Deliver a terminal result after email has been claimed. Run two workers against one event. The invariant is crisp: one reset event can create at most one intent per channel, and no observation can move a terminal workflow backward. Property tests are useful here because the damaging cases are orderings, not exotic payloads.
One claim. One owner.
The last tuning step is to keep the earlier signal from training people to ignore it.
After deployment, compare two distributions by policy region: time from SMS intent to terminal observation, and remaining reset lifetime when email is claimed. Also count suppressed duplicate claims, events that expire without a terminal channel outcome, and fallbacks followed by a late SMS delivery. Those measures tell you whether the state machine is working and whether the threshold is useful. They don't tell you that the subscriber read either message.
The catch is false positives. Set the threshold too early and ordinary receipt delay produces two valid messages, extra privacy exposure, and avoidable on-call noise. Set it too late and email is technically selected but has no practical delivery budget before the link expires. Start alerting in observation-only mode, review the would-have-paged events by region, then enable paging only for states with an actionable next step. Revisit the threshold when expiry policy, routing, or receipt behavior changes.
This pattern is not suitable when policy forbids SMS for account recovery, when there is no verified email fallback, or when the product cannot maintain a durable event record. In those cases, stick with a single approved channel and an in-product recovery path rather than pretending two best-effort calls form a reliable workflow. A managed scheduler can reduce operational ownership; a self-hosted queue can provide tighter control over storage and routing. Choose based on receipt semantics, idempotency support, regional controls, and incident evidence. No dashboard compensates for an undefined transition contract.
The closing runbook check is simple: can the responder identify the current owner, the last durable observation, the remaining action budget, and the one transition that is still allowed? If yes, the page can lead to action. If not, adjust the instrumentation before tightening the alert.
Top comments (0)