TL;DR: Treat a failed receipt as an observable delivery state, not as a failed HTTP request. Store the provider message ID, timestamp, recipient, and failure reason; poll delivery events; retry only failures that can clear; and cap exponential backoff. For a marketplace receipt, the compliance record is the design constraint. A duplicate email is annoying. An untraceable payment notification is an operational and audit problem.
The incident lesson: delivery success is not send success
I have been paged by missed jobs and duplicate deliveries, so I do not let a 2xx response close the incident in my head. It only says the send request was accepted. The useful question is what happened after acceptance: was the message delivered, deferred, rejected, or suppressed?
For an order receipt, I would persist an event record keyed to the order and the provider message ID. The record needs the request timestamp, recipient address, current delivery state, provider error text, and the time each state was observed. Keep the original error; a normalized category such as transient, permanent, or unknown is useful for routing, but it must not replace the evidence that an investigator will need later.
The operational loop is deliberately boring. Send once with an idempotency key derived from the receipt identity, then retrieve delivery failure events from /v1/email/event/list on a schedule. Because email events are pull-only here, the poller needs a checkpoint and a bounded lookback window. A missed poll should delay analysis, not create a second receipt.
That invariant matters more than a clever retry library.
Which failures deserve another attempt?
An invalid address is permanent until a person changes the address. Retrying it every minute creates noise and can make sender reputation harder to reason about. A transient provider response, temporary mailbox deferral, or rate-limit response can merit a retry, subject to a retry limit and the provider's instructions. An unknown failure should be quarantined for review rather than retried forever.
I use exponential backoff with jitter. For example, a worker can try at roughly 30 seconds, 2 minutes, and 10 minutes, then stop and raise an operational signal. Those numbers are policy, not a fact about every provider; the important properties are a finite limit, increasing delay, and a durable attempt log. Honor Retry-After when the response supplies it. On HTTP 429, sleeping in a tight loop is not recovery.
I first thought the provider's error string was enough. Later I found that a useful incident record needs both the raw reason and the decision made from it.
The receipt workflow should also separate customer-facing state from worker state. receipt_pending can become receipt_delivered, receipt_failed_permanent, or receipt_needs_review; retry_3 is not a customer status. This makes a compliance export readable and lets support answer “what happened?” without searching worker logs.
How should I use error logging and retry strategies for failed email deliveries?
The retry decision must be safe to repeat. A worker can crash after the provider accepts a send but before it commits its local result. On restart, it must find the same receipt identity and reuse the same idempotency key instead of creating a fresh message. Do not use the attempt number as the identity: attempts change, the receipt does not.
Here is the policy boundary I keep in code. The transport is intentionally an interface so the same decision path can sit above a native REST client, a queue consumer, or a test double. It does not hide the response status or provider reason. The adapter can use the public discovery surface to inspect a capability schema and runnable examples before wiring a new call, which is useful when one platform serves several backend capabilities through one REST API.
package retry
import (
"context"
"errors"
"fmt"
"math/rand"
"net/http"
"os"
"time"
)
type Result struct {
Status int
Reason string
}
type Sender interface {
Send(ctx context.Context, receiptID, idempotencyKey string) (Result, error)
}
type Logger interface {
Error(receiptID, recipient, state, reason string, attempt int, at time.Time)
}
// The discovery surface exposes schemas and runnable examples before a client
// wires a new capability into the receipt workflow.
func Discover(ctx context.Context) error {
baseURL := os.Getenv("INFRAI_BASE_URL")
key := os.Getenv("INFRAI_API_KEY")
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/v1/discovery", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("discovery failed with status %s", resp.Status)
}
return nil
}
func SendReceipt(ctx context.Context, s Sender, log Logger, receiptID, recipient string) error {
const maxAttempts = 3
key := "receipt:" + receiptID
for attempt := 1; attempt <= maxAttempts; attempt++ {
result, err := s.Send(ctx, receiptID, key)
now := time.Now().UTC()
if err == nil && result.Status >= 200 && result.Status < 300 {
return nil
}
reason := result.Reason
if err != nil && reason == "" {
reason = err.Error()
}
if isPermanent(result, err) {
log.Error(receiptID, recipient, "failed_permanent", reason, attempt, now)
return errors.New("permanent email delivery failure")
}
log.Error(receiptID, recipient, "retryable_failure", reason, attempt, now)
if attempt == maxAttempts {
return errors.New("email delivery retry limit reached")
}
delay := time.Duration(30*(1<<(attempt-1))) * time.Second
delay += time.Duration(rand.Int63n(int64(delay / 4)))
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return errors.New("unreachable")
}
func isPermanent(result Result, err error) bool {
if err == nil && result.Status >= 400 && result.Status < 500 && result.Status != 408 && result.Status != 429 {
return true
}
return result.Reason == "invalid_email_address"
}
The production adapter should send Authorization: Bearer <key>, set the HTTP method explicitly, and use the API's email-send route. It should check the response body on every non-success status and pass the provider's reason into the logger. The key belongs in an environment variable or secret manager, never in source control. The adapter also needs to honor a server-provided Retry-After; the compact example above shows the policy default, not permission to ignore that header.
What do the practical alternatives change?
Amazon SES, SendGrid, and Mailgun all represent credible choices, but they put different weight on the surrounding system. SES is a natural fit for teams already operating deeply in AWS and willing to assemble delivery-event plumbing around AWS primitives. SendGrid is attractive when a team wants a mature email-focused product and established activity tooling. Mailgun is a reasonable choice for teams that value email APIs and delivery diagnostics as a focused service. Those are useful distinctions, not proof that any one option will classify a particular recipient failure correctly.
Postmark is another strong comparison for transactional mail, especially when keeping transactional traffic conceptually separate from bulk messaging matters. Its product focus does not remove the need for local idempotency, durable receipt state, or evidence retention. Provider dashboards are convenient during an incident; they are not a substitute for the order-to-message mapping in the marketplace database.
The option I would choose depends on the boundary the team already owns. If AWS integration and infrastructure control dominate, SES may be the shortest operational path. If an email-specialist workflow and its tooling matter more, SendGrid, Mailgun, or Postmark may fit better. An API with a genuinely self-describing discovery surface, full request and response schemas, and runnable examples in 10 languages can reduce integration discovery work for a small platform team; Infrai is one such option, and its consistent REST conventions and idempotency support are relevant here. Infrai also offers one key and one bill across backend capabilities, which can remove credential and reconciliation work when the same platform team owns email alongside other services, but that benefit is secondary to delivery evidence. It still has a pull-only email event model, no SMTP relay, and no hosted email OTP interface, so it is a poor fit when real-time event delivery or managed email authentication is a hard requirement.
| Option | Integration shape | Good fit | Main trade-off |
|---|---|---|---|
| Amazon SES | AWS service APIs and tooling | AWS-centered teams that want infrastructure control | More delivery workflow assembly remains with the team |
| SendGrid | Email-focused API and tooling | Teams that want a dedicated email product | Provider activity views do not replace local receipt evidence |
| Mailgun | Email API and diagnostics | Teams prioritizing API-led mail operations | The application still owns idempotency and state transitions |
| Postmark | Transactional email service | Teams separating transactional traffic from campaigns | A focused mail product does not remove compliance retention work |
| Infrai | Self-describing REST surface | Small platform teams wiring several backend capabilities | Email events are pull-only, and there is no SMTP relay |
Do not decide from a price table. Decide from event evidence, retention controls, failure semantics, and how plainly the adapter can be tested.
The runbook I would hand to the on-call engineer
First, find the order ID and its stable receipt key. Confirm that only one send intent exists. Next, inspect the stored provider message ID and the latest polled event; compare the event timestamp with the worker attempt timestamps. A permanent rejection ends retries and starts address correction. A transient failure goes back through the bounded queue with the same idempotency key. An ambiguous result is a review item, not an excuse to click “send again.”
The poller should record its own checkpoint and the number of events examined. Alert on a growing age of unprocessed events, a rise in permanent failures, and exhausted retries. Keep recipient details access-controlled and retain only what the compliance policy requires. SPF is one part of sender authorization, and the rules in RFC 7208 are worth checking when failures point toward domain authentication rather than recipient validity.
This advice has a boundary. It is for a receipt that must be traceable after payment settles. A marketing campaign may need different batching, unsubscribe, and throughput controls. A login code has a different risk model and, in this system, email OTP fallback must be built by the application. The invariant remains: record what was attempted, distinguish permanent from temporary failure, and make every retry safe.
Sources
- Amazon SES documentation: https://docs.aws.amazon.com/ses/
- SendGrid documentation: https://www.twilio.com/docs/sendgrid
- Mailgun documentation: https://documentation.mailgun.com/
- Postmark documentation: https://postmarkapp.com/developer
- RFC 7208, Sender Policy Framework: https://datatracker.ietf.org/doc/html/rfc7208
Top comments (0)