Short answer: keep OTP issuance separate from delivery, treat SMS and email as two attempts under one challenge, and put an auditable recipient-suppression decision in front of each attempt. Poll provider events into an append-only evidence log with a durable cursor; do not infer delivery from an accepted API call. The least complex design that works has one challenge table, one attempt table, one normalized event table, and one suppression table.
The page fires at 03:07: fallback_attempt_rate has crossed its threshold for the US login flow. The on-call sees 84 email attempts following 84 SMS attempts, several repeated recipient hashes, and no trustworthy answer to the first question: what page should have fired earlier? In a fintech login path, the useful earlier signal is not a generic delivery dashboard turning red. It is a rising count of attempts admitted for recipients already known to be invalid, grouped by channel, region, and suppression reason.
This framing changes the beginner architecture. SMS-primary and email-fallback sounds like a routing rule; operationally, it is a small state machine whose decisions may later need to explain who was contacted, why a second channel was used, which evidence authorized it, and why a known-bad destination was or was not retried. Cheap infrastructure cannot compensate for missing evidence.
How should a beginner OTP login architecture handle fallback?
Work backward from the alert. The fallback ratio rose because primary attempts reached a terminal failure or exceeded the delivery deadline. Some fallback email attempts then bounced. If the application records only sms failed, email sent, it has thrown away the facts needed to distinguish a temporary carrier delay from a permanently invalid phone number, or an accepted email request from a mailbox rejection.
The first instrumentation mistake is counting provider API acceptance as delivery. Acceptance means an attempt entered another system; the later event determines what happened next. The second is overwriting one status column as callbacks or polled records arrive. That destroys ordering, duplicate evidence, and the raw identifiers needed to investigate a disagreement.
The correction is deliberately plain: append every normalized event, preserve its provider event ID and observed time, and derive current attempt state from that history. Polling should advance only after the fetched page is committed. If a process dies after the commit but before acknowledging its own progress, a uniqueness constraint on (source, provider_event_id) makes replay harmless.
No dashboard fixes this.
Put suppression before routing
A recipient can be eligible for login while one destination is ineligible for delivery. Keep those concepts apart. A disabled phone destination must not disable an account automatically, and it must not cause an automatic email attempt unless policy permits that fallback for the challenge.
Use a narrow decision order:
- Load the challenge and verify that it is unexpired, unconsumed, and still within its attempt policy.
- Resolve the primary destination to a stable internal recipient hash; do not use plaintext contact data as a metric label.
- Check channel-specific suppression before creating a delivery attempt.
- If primary delivery reaches a policy-defined terminal state or deadline, evaluate fallback separately, including its own suppression record.
- Append the decision and its reason, even when no message is sent.
A permanent invalid-recipient event should create or extend suppression through an idempotent transition. A timeout should not. That distinction is the center of the design: overly broad suppression locks legitimate users out, while overly narrow suppression sends repeated traffic to destinations already shown to be invalid.
The evidence model can remain small:
| Record | Minimum purpose | Never overwrite |
|---|---|---|
| Challenge | Bind one OTP lifecycle to policy | Creation time and policy version |
| Attempt | Represent one channel send decision | Recipient hash, region, channel, reason |
| Event | Preserve externally observed evidence | Source ID, raw status, observed time |
| Suppression | Block a destination for a documented cause | Cause event ID and decision time |
| Poll cursor | Resume ingestion without gaps | Last committed position |
Do not place the OTP itself in the event log. Store only what the verifier needs, restrict its lifetime, and keep delivery evidence free of message content. The log should answer why an attempt happened without becoming a second store of authentication secrets.
Make the poller boring and replayable
A polling worker needs fewer clever abstractions than most teams give it. Fetch after the committed cursor, normalize known event types, persist the events and the next cursor in one transaction, then repeat. Unknown statuses belong in evidence storage and an alert stream; silently mapping them to failed can trigger an unjustified fallback or suppression.
Here is the boundary I want the application to own. It is Go because the contract matters more than a client-library tutorial:
type DeliveryEvent struct {
Source string
ProviderEventID string
AttemptID string
RawStatus string
ObservedAt time.Time
}
type EventPage struct {
Events []DeliveryEvent
NextCursor string
}
type EventSource interface {
Fetch(ctx context.Context, cursor string, limit int) (EventPage, error)
}
type EvidenceStore interface {
CommitPage(ctx context.Context, events []DeliveryEvent, nextCursor string) error
}
func PollOnce(ctx context.Context, src EventSource, store EvidenceStore, cursor string) error {
page, err := src.Fetch(ctx, cursor, 200)
if err != nil {
return fmt.Errorf("fetch delivery events: %w", err)
}
if err := store.CommitPage(ctx, page.Events, page.NextCursor); err != nil {
return fmt.Errorf("commit delivery evidence: %w", err)
}
return nil
}
CommitPage must enforce uniqueness for source event IDs and update the cursor atomically with inserts. The worker may then retry a page without inventing exactly-once transport semantics. Set the page size and polling interval from backlog age and provider limits, not from a hope that every event arrives instantly.
The same normalization layer should retain raw status alongside the internal class. Internal classes such as delivered, temporary_failure, permanent_invalid, and unknown are useful for policy, but erasing the source value makes later audits weaker and turns new upstream statuses into invisible behavior changes.
The main limitation is latency. The polling trade-off is delayed event arrival in exchange for a smaller inbound security surface and straightforward replay, so this design is not suitable when the fallback decision has a hard latency target shorter than the polling interval. In that case, ingest authenticated push events for the fast path and retain cursor-based reconciliation as the slower completeness check. The extra path raises deployment and testing cost, so a beginner system should add it only after a measured latency requirement demands it.
Instrument the decision, not the vendor
The earlier page should fire on application decisions: known-suppressed attempts admitted, event-ingestion lag, unknown event classes, cursor stagnation, fallback attempts per challenge, and challenges with conflicting terminal evidence. Provider dashboards may help during diagnosis, but they cannot show whether the application checked suppression before routing or which policy version allowed fallback.
For US and EU traffic, attach region as a low-cardinality field to an attempt and retain the policy version used at decision time. Avoid treating geography as a proxy for consent or legal basis; those are explicit policy inputs, not properties inferred from a phone prefix. The architecture supplies evidence. Legal and compliance owners still define the policy it records.
Message construction also affects operations. SMS segmentation changes with encoding: the referenced character-limit guide documents 160 characters for a single GSM-7 message and 70 for UCS-2, with lower per-segment limits for concatenated messages. An unexpected character can therefore alter segmentation. Keep an OTP SMS short, test its actual encoding, and record the template version rather than the secret-bearing rendered body.
For fallback email, authentication and sender hygiene are part of deliverability. The Google sender guidelines describe SPF or DKIM requirements for senders and stronger requirements for bulk senders, including SPF, DKIM, and DMARC. An OTP system should treat those controls as deployment prerequisites, then consume later bounce evidence to suppress invalid destinations; successful submission alone is not proof of inbox delivery.
The threshold is part of the incident
A page on one failed OTP would be noise. A page only after fallback volume doubles may arrive after the system has repeatedly contacted bad destinations. Start with two layers: a fast alert on suppression bypass, because that is an invariant violation, and a slower rate alert on fallback or permanent-invalid outcomes, segmented by channel and region.
There is a real cost on both sides. Tight rate thresholds page on ordinary carrier or mailbox variation, training the on-call to distrust the signal. Loose thresholds delay detection and expand the evidence review. Record every threshold change with its reason, then test alerts using synthetic events that exercise duplicate delivery, out-of-order arrival, cursor replay, unknown status, primary timeout, suppressed fallback, and simultaneous workers.
The postmortem question is concrete: could the first invalid-recipient event have prevented the next attempt, and would the evidence prove that decision? If the answer depends on reconstructing a mutable status cell or opening a provider dashboard, the architecture is incomplete. A small append-only trail, explicit suppression transitions, and a replayable poll cursor give the on-call an answer before a fallback spike becomes the only visible symptom.
Top comments (0)