A retry policy cannot tell you whether a signup verification message was sent. The hard case is an API timeout after submission: the service may have accepted the email or SMS although the worker never received its response. Treat that result as unknown, keep one logical verification challenge, and reconcile before sending again. Backoff protects an API from request pressure; idempotency protects a user from repeated messages.
No guessing.
TL;DR: model each channel attempt with six states: ready, submitting, accepted, unknown, retryable, and terminal. Persist the attempt before the network call, honor an explicit retry delay when returned, and never turn an ambiguous timeout directly into a fresh send. Email and SMS can be separate paths while pointing to the same short-lived signup challenge.
This framing comes from operating cron and queue infrastructure where missed jobs and duplicate deliveries both become pages. A queue acknowledgment is evidence about the queue, not evidence that a person received a message.
How should an email or SMS API retry event notifications after a rate limit?
A signup event creates one email job. The worker records no attempt, calls the delivery API, and loses the connection before reading the response. Its queue lease expires. A second worker receives the job and sends the template again. Queue redelivery behaved correctly; the application discarded the fact that the first outcome was ambiguous. A definite rate-limit response belongs on a delayed schedule, using an explicit server delay when available and otherwise a bounded backoff policy. A timeout after submission belongs in unknown. Mixing those cases is the small coding shortcut that creates the large operational problem: the retry loop has pressure control but no evidence model.
I initially treated a transport error as a failed send because that is the convenient branch in worker code. It is the wrong claim. After request bytes may have crossed the boundary, a timeout means unknown, not failed.
The application needs a stable logical identity derived from account, challenge generation, channel, and purpose. A retry reuses it; a newly issued challenge gets a new generation. SMS fallback also needs an explicit deadline. If email is merely slow, firing SMS in an exception handler creates two valid-looking prompts.
Six states make uncertainty visible
| State | Known fact | Next action |
|---|---|---|
ready |
No active submission | Claim and submit |
submitting |
A worker owns a lease | Wait or recover an expired lease |
accepted |
The service acknowledged submission | Stop API retries |
unknown |
Submission may have crossed the boundary | Reconcile |
retryable |
A definite response permits retry | Schedule bounded backoff |
terminal |
Policy stopped attempts or challenge expired | Do not send |
Do not call accepted delivered. Preserve that distinction. There are also two clocks: the attempt clock says when another call is allowed, while the challenge clock says whether the link remains useful. If the next attempt falls after expiry, terminate it. No retry is better than a dead link.
Keep the preventative path boring
package dispatch
import (
"context"
"errors"
"time"
)
type DefiniteError interface {
error
RetryAllowed() bool
RetryAfter() (time.Duration, bool)
}
type Attempt struct {
ID, IdempotencyKey string
ChallengeUntil time.Time
Number int
}
type Store interface {
Begin(context.Context, string, time.Time) (Attempt, error)
Accepted(context.Context, string, string) error
Unknown(context.Context, string) error
Retry(context.Context, string, time.Time) error
Terminal(context.Context, string) error
}
type Sender interface { Submit(context.Context, string) (string, error) }
func Deliver(ctx context.Context, db Store, out Sender, key string, now time.Time) error {
a, err := db.Begin(ctx, key, now) // Atomically claims eligible work.
if err != nil { return err }
if !now.Before(a.ChallengeUntil) { return db.Terminal(ctx, a.ID) }
providerID, err := out.Submit(ctx, a.IdempotencyKey)
if err == nil { return db.Accepted(ctx, a.ID, providerID) }
var definite DefiniteError
if !errors.As(err, &definite) { return db.Unknown(ctx, a.ID) }
if !definite.RetryAllowed() { return db.Terminal(ctx, a.ID) }
delay := []time.Duration{5*time.Second, 20*time.Second, time.Minute, 5*time.Minute}
n := a.Number
if n < 0 { n = 0 }
if n >= len(delay) { n = len(delay)-1 }
wait := delay[n]
if d, ok := definite.RetryAfter(); ok && d > wait { wait = d }
next := now.Add(wait)
if !next.Before(a.ChallengeUntil) { return db.Terminal(ctx, a.ID) }
return db.Retry(ctx, a.ID, next)
}
Those four delays are example policy, not universal constants. Add jitter in the scheduler and cap attempts. The schedule belongs in the job record; sleeping inside a worker holds capacity and loses intent when the process exits.
The unknown branch deliberately schedules nothing. Reconcile when suitable evidence exists. Otherwise document a policy: wait, retry with the same key only when the receiving contract defines idempotency, or offer a user-triggered resend. A header-shaped value does not prove that an arbitrary API deduplicates it.
Test the gaps
Inject failure after a fake sender records acceptance but before it returns. Restart the worker, redeliver the queue item, and assert that the existing logical attempt is recovered. Exercise lease expiry with two workers too; exactly one owns submitting.
Track API attempts, acknowledged submissions, ambiguous submissions, downstream signals, expired challenges, and user-requested resends separately. Keep tokens and destinations out of logs. Alert on growing unknown counts, oldest eligible job age, and accepted attempts without downstream progress. A single “sent” counter cannot locate the break.
DMARC defines policy and reporting around domain alignment; it does not make a retry safe. The WebOTP documentation describes domain-bound SMS formatting and browser permission behavior. That can improve OTP consumption, but it does not replace expiry, attempt state, or replay protection.
Deploy state transitions before retries. Shadow-classify errors first, then enable definite rate-limit retries, and finally add reconciliation for ambiguity. This keeps possible duplicates behind evidence.
This approach has a real trade-off: it adds durable state, a reconciler, and another runbook. It is not suitable for disposable notifications where duplicates carry no meaningful cost and immediate latency matters more. It is also insufficient for regulated records requiring stronger retention or approval controls. Teams without provider-side reconciliation evidence must accept a longer ambiguous interval or require an explicit user resend; the state machine cannot manufacture certainty.
For signup verification, preserve one challenge, distinguish accepted from delivered, and refuse to guess after a timeout. Reliability starts with recording uncertainty accurately. Backoff is scheduling, not delivery state.
Top comments (0)