Short answer: put imported users into a durable work queue, send in bounded batches, retry only transient failures with backoff, and make every delivery idempotent. A loop that calls an email API for every imported row is easy to write and hard to operate.
I care about this because the failure mode is rarely “the email call was difficult.” It is the queue that gets drained twice, the process that dies after the provider accepts a message, or the import job that quietly turns a welcome campaign into a rate-limit storm. In a logistics system, the same pattern appears after payment settles: an order receipt has to be attempted once as a business action, even though the transport may be attempted more than once.
How should a user import trigger bulk welcome email retries?
Treat the import and the notification as two separate workflows. The import transaction records the user and an outbox event. A worker later claims that event and creates a delivery record with a stable key such as welcome:<user-id>:<import-id>. The worker can be restarted without creating a second business event.
That boundary matters. If the import handler sends mail before committing the user row, a timeout leaves the system unsure whether to send again. If it sends after the commit but before recording a job, a process crash can lose the welcome entirely. An outbox row closes that gap: the same database transaction commits the user and the request to send, then a separate publisher moves pending rows to the queue.
The delivery record should carry the recipient, template version, event key, attempt count, next-attempt time, and a state such as pending, sent, or dead. Store the provider message identifier when one is returned. Do not use a recipient address as the idempotency key; an address can belong to more than one imported account or be corrected later.
One distinction keeps the runbook honest: idempotency protects the business operation, while a retry protects an individual transport attempt. Neither one can prove that a message was read. For an order receipt, “sent” should mean that the chosen transport accepted the request, not that the customer opened it.
The incident pattern: accepted, timed out, and sent again
I have been paged for missed jobs and duplicate deliveries. The bounded version of the incident is familiar: an import starts, the worker submits a receipt, the request times out while the remote service is processing it, and the worker retries. Without a stable delivery key and a durable state transition, the customer receives two messages.
The dangerous part is the gap between those two observations. A timeout says that the caller did not receive a result; it does not say that the remote system did nothing. Suppose the worker claims 500 rows and the process disappears after transport acceptance for row 317. A restart must be able to distinguish “never attempted” from “attempted, outcome unknown” and “accepted, state update lost.” The first case is safe to send, the second needs reconciliation or a carefully controlled retry, and the third must be deduplicated by the delivery key. If all three cases are represented as pending, the recovery code has no information left to use. That is how an import that looked successful at 10:02 turns into a second receipt wave at 10:07, while the dashboard still reports only a generic worker restart. The runbook should name this ambiguity explicitly and give the operator a way to pause claims while the accepted-message records are reconciled.
Pause the import.
The first repair is not a larger timeout. It is a state machine that makes the ambiguous result visible. Mark a delivery as sending with a lease, submit it with an idempotency key when the transport supports one, and reconcile an ambiguous response before releasing it for another attempt. A lease must expire so a dead worker does not hold work forever; its duration should be based on observed request latency, not a guess copied from another service.
Here is the essential worker shape. The repository and sender interfaces are deliberately generic so the policy remains testable and the provider-specific adapter stays at the edge.
package welcome
import (
"context"
"errors"
"fmt"
"math/rand"
"time"
)
var ErrRateLimited = errors.New("rate limited")
type Delivery struct {
ID string
Recipient string
IdempotencyKey string
Attempts int
}
type Repository interface {
Claim(ctx context.Context, limit int, now time.Time) ([]Delivery, error)
MarkSent(ctx context.Context, id string, providerID string) error
Reschedule(ctx context.Context, id string, next time.Time, reason string) error
MarkDead(ctx context.Context, id string, reason string) error
}
type Sender interface {
Send(ctx context.Context, d Delivery) (providerID string, err error)
}
func RunBatch(ctx context.Context, repo Repository, sender Sender, batchSize int, now time.Time) error {
deliveries, err := repo.Claim(ctx, batchSize, now)
if err != nil {
return err
}
for _, d := range deliveries {
providerID, sendErr := sender.Send(ctx, d)
if sendErr == nil {
if err := repo.MarkSent(ctx, d.ID, providerID); err != nil {
return err
}
continue
}
if d.Attempts >= 8 {
if err := repo.MarkDead(ctx, d.ID, sendErr.Error()); err != nil {
return err
}
continue
}
if !errors.Is(sendErr, ErrRateLimited) {
// The adapter should classify only retryable transport failures here.
if err := repo.MarkDead(ctx, d.ID, sendErr.Error()); err != nil {
return err
}
continue
}
delay := backoff(d.Attempts)
if err := repo.Reschedule(ctx, d.ID, now.Add(delay), sendErr.Error()); err != nil {
return err
}
}
return nil
}
func backoff(attempt int) time.Duration {
base := time.Second * time.Duration(1<<min(attempt, 6))
return base + time.Duration(rand.Int63n(int64(base/2)))
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
var _ = fmt.Sprintf // Template adapters may use fmt for provider payloads.
The example marks unknown errors dead, which is intentionally conservative. In a production adapter, classify responses before they reach this loop: a temporary capacity response can be delayed, while a malformed address or rejected policy should go to a reviewable dead-letter state. Never retry every error because the request was convenient to repeat.
The sample uses a random delay component to avoid workers waking together. Seed and inject randomness in tests. More importantly, make Claim atomic: two workers must not receive the same unleased rows. The code cannot provide that guarantee by itself; it belongs in the repository transaction and its database constraints.
Rate limits are a scheduling problem, not a loop setting
A provider limit is only one part of the schedule. Your own SMTP or HTTP connection pool, recipient-domain behavior, campaign size, and the import's arrival rate also shape the safe send rate. Start with a deliberately small concurrency limit, measure accepted and rejected requests, then increase it during a controlled run. A queue gives you somewhere to put the remaining work.
Use a token bucket or another explicit limiter around the sender, and keep the batch size independent from concurrency. Batch size controls how much work a worker claims; concurrency controls how many calls can be active. Combining both into an unbounded goroutine fan-out makes a restart particularly expensive.
Honor a server-provided retry delay when the adapter exposes one. Otherwise use exponential backoff with jitter and a maximum attempt count. A retry should preserve the original delivery key and template version. It should not re-render from mutable user data in a way that changes a receipt or welcome message halfway through the run.
Email deliverability is also a policy concern. Google’s sender guidance calls out authentication, spam rates, and sender requirements; a batch that is technically within a request limit can still create a deliverability problem if the sending identity and consent model are poor. Keep transactional messages separate from promotional mail, retain unsubscribe and suppression decisions where applicable, and monitor bounces rather than treating API acceptance as final success.
SMS makes the same architecture more visible because message length changes the number of segments. Twilio’s character-limit guidance explains the GSM-7 and UCS-2 distinction and how longer messages are segmented. If the workflow later adds an SMS fallback, calculate and record the segment count before sending. A fallback is a new delivery policy, not a free second attempt at email.
What should the runbook measure before and after an import?
The operator needs to answer four questions during a run: how much work was created, how much was claimed, how much was accepted, and how much is waiting for another attempt. Put those counts on one dashboard with the import identifier and template version.
Track queue age, claim age, send latency, retry count, rate-limit responses, permanent failures, ambiguous outcomes, and dead-letter volume. Alert on the age of the oldest pending delivery, not only on worker process health. A healthy process can be doing nothing while a transactionally important queue is stuck behind a bad schedule.
Log the delivery ID and provider message ID, but redact message bodies and protect recipient addresses. Correlate the import, outbox row, delivery, and transport request with separate identifiers. This makes a duplicate investigation possible without putting customer content into every log line.
Before enabling a large import, test these cases:
- the worker exits after the transport accepts a message but before
MarkSent; - two workers claim the same row at the same time;
- the provider returns a rate-limit response with a delay;
- an address is permanently rejected;
- the same import event is delivered twice;
- a lease expires while a slow request is still in flight.
Use a fake sender that records keys and a repository that can replay each transition. A staging run should use sink addresses or a provider test mode. Do not validate the design by emailing real users first.
The trade-offs that decide the design
There is no universal retry count or batch size. Those values depend on the provider contract, the importance of the message, and how much delay the business accepts.
| Choice | Useful when | Cost or limitation |
|---|---|---|
| Synchronous send in the import request | Tiny, low-risk imports where immediate feedback matters | A timeout can lose or duplicate work; the request holds open while the transport runs |
| Durable outbox plus worker | Receipts and welcome messages must survive process restarts | Requires storage, leasing, reconciliation, and queue monitoring |
| Aggressive retries | A short-lived dependency hiccup is common and duplicates are impossible | Can amplify an outage and crowd out newer work |
| Small batches with low concurrency | Deliverability and provider capacity are uncertain | The import drains more slowly and needs visible queue-age tracking |
| Email-only delivery | Email is the declared channel and the address is usable | It does not cover users without usable email or an explicitly supported fallback |
The catch is that this approach is not suitable when the business needs a hard, immediate response inside the import request, or when no durable store is available. Keep the synchronous path for a small administrative tool with a clear failure response. For a large import, stick with an outbox and a worker; accepting slower completion is usually easier to reason about than guessing after a timeout.
I’m not sure any fixed backoff value will suit every sending identity. Your mileage may vary. Resolve that uncertainty with a small canary, provider documentation, queue-age measurements, and a rollback rule that pauses new claims without deleting pending work.
The decision rule is simple: model the welcome email as durable business work, keep transport retries bounded and idempotent, and make rate limiting observable. The same rule protects a logistics receipt after payment settles. It also gives an operator a safe pause button when the import is larger than expected.
Top comments (0)