Why Not Use an SMTP Relay for OTP Email in a Mixed-Provider Setup?

#security #email #architecture

Short answer: use an SMTP relay when it is already a governed part of your mail estate, but keep OTP generation, expiry, attempt limits, and verification in the authentication service. A mixed-provider setup should change delivery routing, not who decides whether a login challenge is valid.

For an e-commerce password reset, the useful artifact is not an email accepted by a relay. It is an auditable authentication decision: which challenge was issued, when it expires, how many attempts remain, and whether it was consumed once. That distinction is where many designs go wrong.

Why should an SMTP relay not own OTP email in a mixed-provider auth setup?

SMTP is a transport boundary. OTP is an authentication boundary. Combining them makes delivery behavior part of the security model, even though neither an SMTP acceptance response nor an inbox event proves that the person entered the right code.

The application should create the challenge, bind it to the account and purpose, enforce a short expiry, limit guesses, and invalidate the previous challenge when a new one is issued. The sender should receive only the message it needs to deliver. A provider identifier can be retained for support and audit, but it should not be the value used to authorize a reset.

This separation also makes mixed-provider routing less dangerous. Provider A may accept a message before Provider B, and their status vocabularies may not match. Normalize delivery into a small operational result such as accepted, rejected, or observed later. Do not let a delivery callback extend a challenge's life. The clocks are different: challenge expiry, message delivery, and user verification.

Keep that rule visible in the runbook. It prevents a late message from becoming a valid message.

No callback can authorize a user.

What does a safe transactional email design record?

Start with invariants rather than a vendor matrix. For one account and one purpose, define how many active challenges may exist, how replacement works, what happens after expiry, and what consumes a successful code. Then record the evidence needed during a compliance review: issuance time, expiration time, delivery request identifier, result, and the reason a verification attempt was accepted or rejected.

The datastore must handle concurrent requests. A double-click on “send code” can issue two messages whose delivery order is unknowable, and a retry after a client timeout can produce the same ambiguity even when the sender accepted the first request. Only the newest challenge should remain valid, even if the older email arrives later. Store a keyed digest instead of the plaintext code, decrement attempts atomically, and mark successful use atomically with the authorization step. Keep the challenge's purpose in the key or record as well: a password-reset code should not accidentally become a general login factor, and a code issued for one account should not be comparable under another account's context. Those are ordinary state transitions, but they are the ones a postmortem will inspect when a customer reports two messages, a late message, or a reset that cannot be reconstructed from the audit trail.

Here is the critical path in Go. The delivery function is deliberately an interface: it can call SMTP, an email API, or a routing service without moving OTP authority out of the application.

package otp

import (
    "crypto/hmac"
    "crypto/sha256"
    "crypto/subtle"
    "encoding/hex"
    "errors"
    "fmt"
    "time"
)

var ErrInvalidChallenge = errors.New("invalid challenge")

type Challenge struct {
    Digest       string
    ExpiresAt    time.Time
    AttemptsLeft int
    Consumed     bool
}

type Store interface {
    Replace(userID string, challenge Challenge) error
    Get(userID string) (Challenge, bool)
    Consume(userID string) error
    UseAttempt(userID string) error
}

type Deliver func(address, body string) error

func digest(secret []byte, userID, code string) string {
    mac := hmac.New(sha256.New, secret)
    fmt.Fprintf(mac, "password-reset:%s:%s", userID, code)
    return hex.EncodeToString(mac.Sum(nil))
}

func Verify(store Store, secret []byte, userID, code string, now time.Time) error {
    challenge, ok := store.Get(userID)
    if !ok || challenge.Consumed || !now.Before(challenge.ExpiresAt) || challenge.AttemptsLeft <= 0 {
        return ErrInvalidChallenge
    }

    provided := digest(secret, userID, code)
    if subtle.ConstantTimeCompare([]byte(provided), []byte(challenge.Digest)) != 1 {
        if err := store.UseAttempt(userID); err != nil {
            return err
        }
        return ErrInvalidChallenge
    }
    if err := store.Consume(userID); err != nil {
        return err
    }
    return nil
}

The persistence implementation still matters. Get followed by Consume must be protected by the store's transaction or compare-and-set rule; otherwise two simultaneous requests can both pass verification. The sample shows the boundary, not a claim that an in-memory map is sufficient for production.

How do SMTP relay and mixed-provider choices affect operations?

An existing relay can be the right choice. It keeps a mature authentication, monitoring, and compliance path in place, and it avoids adding another credential or status model. The catch is that it is only a good fit if the relay is approved for the recipients and message class, and if its operational evidence is strong enough for password-reset investigations.

A dedicated API can be a better boundary for a team that wants direct HTTPS delivery and a focused email workflow. A mixed-provider layer can be justified by regional routing, independent failover, or a requirement to separate delivery domains. It also creates work: routing policy, suppression behavior, status normalization, credential rotation, abuse controls, and reconciliation all become application concerns.

Choice	What the application must still own	Main trade-off
Existing SMTP relay	Challenge state, verification, expiry, and audit evidence	Low migration cost; relay policy and evidence may constrain recipients
Direct email API	The same auth state plus an email adapter	Clear transport boundary; adds a separate integration to operate
Mixed-provider router	Auth state plus routing, failover, and normalized status	More control; more states and more compliance surface
SMTP plus API fallback	Auth state plus deterministic retry and duplicate control	Can improve reachability; retry mistakes can create duplicate messages

Do not use delivery success as proof of delivery, and do not use delivery failure as proof that a code was never seen. The authentication service needs its own decision log. Yahoo's sender guidance also makes clear that sender requirements and sending behavior belong to the mail layer, not to the OTP verifier.

What should verification, testing, and rollback prove?

Test the state machine with the failures that page people: a message accepted twice, a provider response arriving late, a replacement code delivered before the original, a retry after a timeout, and two verification requests arriving together. Assert that only one code succeeds, that expiry is final, and that a delivery event cannot reopen a consumed challenge.

Use provider-neutral contract tests for the adapter. Verify authentication, recipient handling, response classification, retry policy, and correlation identifiers without putting real login challenges in a test mailbox. A 429 response should trigger bounded backoff according to the provider contract, not an unbounded loop. I treat that status as flow control, not as permission to spin. Your mileage may vary on exact retry windows; the runbook should state the chosen limit and the evidence behind it.

For rollback, keep the old transport adapter deployable while the new route is evaluated. Roll routing back at the adapter boundary, not by changing verification rules or accepting old challenge records indefinitely. Monitor issuance-to-verification conversion, expiry rates, duplicate sends, rejection categories, and support lookups by dispatch identifier. Those signals tell you whether the change affected delivery or authentication.

I've been paged for missed jobs and duplicate deliveries, and the lesson is boring but durable: a transport retry must never become an authorization retry. Short expiry helps. State ownership helps more.

When is SMTP still the better answer?

Choose the relay when it already has the required sender controls, recipient coverage, audit trail, and on-call familiarity. Choose a mixed-provider design only when its routing requirement is real enough to pay for the extra failure modes. A direct API is not automatically safer, and a relay is not automatically obsolete.

The decision rule is simple: keep identity proof in the authentication service, keep sender policy in the delivery layer, and preserve enough raw evidence to reconstruct what happened. I'm not sure any universal provider status model can retain every useful detail, so store the original event alongside the narrow status used by the auth path.

For password-reset email with a short expiry, that boundary is more important than the transport brand. It gives compliance reviewers a clear chain of evidence and gives operators a clean rollback point when delivery changes.

DEV Community