sawyerflynn1578

Posted on Oct 10

Go Event Notifications: Delayed Email-to-SMS Fallback Without Webhooks

#go #notifications #email

For B2B SaaS password-reset event notifications with a short expiry, implement delayed email-to-SMS fallback without webhooks by keeping the reset intent, rendered-message version, and delivery status in the application database, then polling after a timeout. The least complex workable multi-channel retry logic is one durable state machine, one application-owned template contract, and channel-specific renderers behind it.

TL;DR: Do not treat a delayed email receipt as proof of failure, and do not let a polling timeout create a second logical notification. Persist one intent with an idempotency key, record every attempt as an append-only event, and make the fallback decision from a deadline plus the latest observed provider state. The template owner must also own the cross-channel semantic invariant: both messages identify the same action, expiry, tenant, and security context, while exposing only what each channel can safely carry.

The bill is made of more than sends. If E is email submissions, S is SMS submissions, P is status polls, W is retained payload bytes, and D is retention duration, the variable operating term is approximately E*Ce + S*Cs + P*Cp + W*D*Cstorage, before engineering and support labor. In an email-first design, unnecessary SMS escalation is usually the term the state machine can directly reduce: replace “no success yet means send SMS” with “the email attempt crossed its decision deadline in a state that permits escalation.” This article therefore optimizes the fallback decision, not a quoted unit price.

How should delayed event notifications trigger email-to-SMS fallback?

A password reset is one security intent, but email and SMS are not interchangeable envelopes. Email can carry a tenant-branded explanation and a full link; SMS has a tighter presentation budget and appears on lock screens. If channel templates belong independently to separate delivery integrations, a retry can silently become a semantic fork: different expiry language, a stale host name, or inconsistent instructions about ignoring an unsolicited reset.

Application ownership makes the invariant reviewable. A versioned template specification should name the intent ID, tenant ID, locale, expiry instant, and reset-link origin, then produce channel-specific content from that common input. It should never place the reset token, rendered body, or destination address in routine logs. The delivery adapter owns transport encoding and provider correlation identifiers, but it does not reinterpret the security event.

This boundary also decides what can be retained. Keep the immutable intent metadata, a template version, a destination fingerprint, attempt timestamps, normalized status transitions, and external correlation IDs for the period required by the organization's audit and deletion policies. Stop keeping rendered bodies and raw reset links once submission no longer needs them. The trade-off is explicit: an investigation can prove which template version and state transition occurred, but it cannot reproduce the exact secret-bearing message after that material has been discarded.

Model one intent, many attempts

Exactly-once delivery is not available across an application database, an email system, a mobile carrier, and a recipient device. An exactly-once mindset is still useful, provided it is applied to the decision record: create one reset intent once, make every side effect idempotent where the interface permits it, and reconcile uncertain outcomes rather than guessing.

The useful distinction is between an intent and an attempt. The intent has one stable idempotency key and one expiry. Each channel attempt has its own number, submission token, polling schedule, and normalized state. A timeout while submitting may mean “not accepted” or “accepted but the response was lost”; the next action is therefore a lookup or an idempotent resubmission with the same attempt key, not an unconditional new attempt.

package notification

import "time"

type AttemptState string

const (
    Pending   AttemptState = "pending"
    Accepted  AttemptState = "accepted"
    Delivered AttemptState = "delivered"
    Failed    AttemptState = "failed"
    Unknown   AttemptState = "unknown"
)

type Intent struct {
    ID              string
    TemplateVersion string
    ExpiresAt       time.Time
    EmailDeadline   time.Time
    SMSStarted      bool
}

type Attempt struct {
    IntentID      string
    Channel       string
    AttemptNumber int
    State         AttemptState
    NextPollAt    time.Time
    CorrelationID string
}

func ShouldStartSMS(now time.Time, intent Intent, email Attempt) bool {
    if intent.SMSStarted || !now.Before(intent.ExpiresAt) {
        return false
    }
    if now.Before(intent.EmailDeadline) {
        return false
    }
    return email.State == Failed || email.State == Unknown || email.State == Pending
}

ShouldStartSMS is only a policy predicate. Correctness depends on evaluating it inside a database transaction that atomically changes SMSStarted from false to true and inserts the SMS outbox record, guarded by a unique constraint on (intent_id, channel). Two workers may race; one durable row wins. This is stronger than trusting an in-memory lock and easier to audit than inferring intent from scattered request logs.

The states should be monotonic where the transport allows it, but observations can arrive out of order. Store the raw observation as an audit event, then apply a transition rule to the materialized attempt state. A late “accepted” observation must not downgrade “delivered,” while a terminal failure may still require qualification if the channel later supplies a definitive delivery record.

Poll delayed status without extending the security window

Polling is a recovery mechanism, not a clock that renews the reset. The reset expiry is set when the intent is created. Every worker checks it before submission, resubmission, rendering, and escalation; no delivery retry extends it. Once the intent expires, polling may continue briefly for reconciliation if policy requires, but no new user-facing attempt should begin.

Use bounded exponential backoff with jitter and a persisted NextPollAt. Persisting the schedule prevents a process restart from resetting every attempt to its fastest interval, and jitter prevents a recovered fleet from polling in lockstep. The poll response is an observation; the database transaction that records it should append an event and update the materialized state together.

Short timeouts deserve special treatment. A client timeout describes the caller's wait, not the remote system's disposition. MDN's Fetch API documentation makes the same general boundary visible in browser code: fetching is asynchronous, and HTTP error statuses are not automatically equivalent to promise rejection. Backend adapters need an equally deliberate taxonomy for transport errors, HTTP responses, and provider states, even though the implementation below is Go.

Keep uncertainty visible.

For an ambiguous submission, write Unknown, retain the stable attempt key, and query status before escalating. If no lookup is possible, retry the same submission idempotently when that contract exists. If neither lookup nor idempotent submission is available, the architecture cannot prove that a duplicate will not occur; the policy must choose between delayed recovery and possible duplicate delivery, document that choice, and expose it to operations.

Put fallback behind an auditable deadline

The email deadline is not the reset expiry. It is an earlier policy instant after which SMS may be worthwhile, leaving enough time for the recipient to act before the reset expires. Store both absolute timestamps. Recomputing them from queue age invites clock and deployment differences, and using poll count as elapsed time fails under worker outages.

Consider an illustrative intent created at t0, with its email decision deadline at t0 + D and its credential-reset expiry at t0 + X, where policy requires D < X. The outbox dispatcher submits email attempt 1 with key intent-7f-email-1, but its request times out before a response arrives. The transaction records Unknown and schedules a status poll; it does not create email attempt 2. A worker restart then delays that poll until after t0 + D. When the worker returns, it first checks whether the reset was consumed and whether t0 + X has passed, loads the latest attempt observation, and tries a provider lookup using the stored correlation material. If lookup still cannot establish a terminal result, the fallback transaction compares and sets the intent's SMS flag, writes one SMS outbox row tied to the same template specification, and appends the policy decision. Suppose a delayed poll later says the email was delivered: that observation is appended, the email materialized state advances, and nothing attempts to retract the SMS. The audit trail now describes an honest duplicate-channel outcome caused by bounded uncertainty, while the unique constraints prove that neither a queue redelivery nor the worker restart created a second logical SMS. The exact values of D and X remain product and compliance choices; correctness comes from their ordering and durable evaluation, not from any universal number.

An auditable transition might read: “At decision_time, email attempt 1 was unknown; reset intent 7f... was unexpired; no SMS attempt existed; policy version 4 authorized SMS attempt 1.” That record supports reconciliation without claiming the email was never delivered. If the email later resolves to delivered, retain the observation; do not erase the fact that the system made its decision under uncertainty.

The state machine should enforce these rules:

A unique intent key prevents repeated password-reset events from creating parallel workflows when the originating request is retried.
A unique (intent, channel) constraint permits at most one logical channel attempt, while transport retries reuse its attempt key.
A compare-and-set transition starts SMS only once and only before intent expiry.
Terminal user action, such as successful token consumption, closes the intent and suppresses later sends.
Every decision event records policy version, prior state, new state, observed time, and source, without recording the secret.

This is where template ownership returns to the center. The same policy version that authorizes escalation should point to reviewed email and SMS template versions. Otherwise an operator can reconstruct why SMS started but cannot establish what semantic contract was active.

Test recovery, reconciliation, and privacy together

Happy-path unit tests are insufficient because the difficult states sit between systems. Use a fake clock and a transport adapter that can accept a submission while dropping the response. Then crash the worker after each durable boundary: before the outbox commit, after the commit, after remote acceptance, and after recording an observation. On restart, assert that there is still one intent, no more than one logical attempt per channel, and a complete event trail.

Property tests are particularly appropriate for the transition function. Generate reordered observations, repeated queue deliveries, clock advances, and duplicated submissions; assert that expiry never moves forward, consumed intents never reopen, and a terminal state is not downgraded. Integration tests should exercise rate limiting, malformed status responses, slow reads, and unavailable status endpoints without depending on a real recipient.

Privacy tests matter too. Apple documents that Mail Privacy Protection can prevent senders from learning whether a recipient opened an email and can mask IP-related information. An open event is therefore a poor control signal for password-reset escalation. Use transport status as limited evidence about transport, and use token consumption as evidence that the security action completed; do not infer human attention from tracking pixels.

Operationally, alert on age and inconsistency rather than raw failure count alone: intents nearing expiry with no usable attempt, Unknown submissions older than the reconciliation objective, SMS rows lacking an outbox dispatch, or delivered observations attached to an already consumed intent. Dashboards should separate “provider accepted,” “transport delivered,” and “reset completed.” Those are different claims.

What should the team retain?

Retain enough metadata to answer who decided what, under which policy and template version, at what time, based on which observation. Set the exact retention period from applicable legal, contractual, and organizational requirements; there is no universal duration that can be asserted for every B2B SaaS tenant. Access to the audit trail should itself be logged and restricted.

Deliberately discard secrets and rendered content as soon as their operational purpose ends, and avoid collecting open-tracking data as a fallback trigger. This reduces sensitive storage and keeps the cost term W*D bounded. The price is narrower forensic detail: after deletion, investigators can establish the template version and transition history, but cannot inspect the exact message. For a short-lived password reset, that is usually the defensible side of the trade-off, provided template artifacts are immutable and independently retained as code or release assets.

The final design rule is compact: own the cross-channel meaning in the application, own each fallback decision in a transactional log, and treat every external status as evidence with a timestamp rather than truth that arrived in order. That structure survives delayed polling, lost responses, worker restarts, and duplicate queue delivery without extending the credential-reset window.

DEV Community