DEV Community

IgnatiusCole6932
IgnatiusCole6932

Posted on

Implementing 2026 Seller Compliance Practices for Retrying Failed Email API Calls

#email #reliability #observability #go

TL;DR: Retry a failed email API call only when the response is plausibly transient, use capped exponential backoff with jitter, and write an immutable attempt record before each sleep. For a B2B marketplace order, the useful artifact is not merely sent=true; it is a chain showing which order was notified, which policy decision was made, what the transport returned, and when the next attempt became eligible.

The decision rule is deliberately boring: classify first, schedule second, send third, and record every transition. A 429 or a temporary 5xx can justify a later attempt. A malformed request, an authentication failure, or a permanent recipient rejection should go to review or a dead-letter path instead. Replaying every error at increasing intervals creates duplicate messages, obscures the real failure, and makes compliance evidence harder to defend.

That is the failure boundary.

What evidence does a seller notification actually need?

The scenario is a marketplace seller receiving notice of a new order. The notification service owns delivery attempts; the order service owns the order itself. Keep those responsibilities separate. The attempt record should contain an order identifier, notification identifier, recipient address or a privacy-preserving reference to it, policy version, attempt number, decision, response class, and UTC timestamps. Store the provider response body only under a retention rule reviewed for personal data.

A useful record distinguishes accepted, retry_scheduled, permanent_failure, and cancelled. It should preserve the idempotency key used for the request. That key does not make an arbitrary remote API idempotent by magic, but it gives a compatible transport a way to suppress an accidental replay and gives an investigator something concrete to correlate.

Record it.

Email has several layers. An HTTP API may accept a submission while later mail processing produces a bounce or delivery event. SPF is an authorization mechanism for hosts sending mail for a domain; it is not proof that a mailbox accepted a message. Record API acceptance as one fact, and downstream delivery or bounce evidence as another.

What are sound practices for retrying failed email API calls?

Start with the response class, not the number of attempts. A 429 Too Many Requests response is a rate signal; honor Retry-After when present and parseable, while still applying an operational maximum. A 5xx response generally indicates a server-side problem, but the remote contract is the authority. Transport timeouts and connection resets are ambiguous: the request may have reached the remote system before the client lost the response, so retry them only with an idempotency strategy and a bounded budget.

Do not retry a 400 caused by invalid payload data, a 401 or 403 caused by credentials or policy, or a confirmed permanent recipient rejection. A retry loop cannot repair bad data. It can multiply traffic and delay the queue behind a message that should have been quarantined.

Bound it.

For attempt n, a common schedule is base * 2^n, capped at a maximum, then randomized with jitter so a fleet does not wake every worker on the same boundary. The base and cap belong in capacity planning: model worst-case retry volume against API quota, queue depth, worker concurrency, and the notification SLO. As concrete policy inputs, a 30-second request deadline and a 15-minute retry horizon can be useful starting values for review, but they are not universal constants; measure queue age and dependency behavior before adopting them.

The queue math is easy to underestimate. Suppose 500 workers each hold one retryable job and the dependency returns 429 for two minutes. If every worker wakes at the same second, the second wave recreates the rate-limit event, and a queue that looked healthy before the incident can become a second source of load. Full jitter turns that synchronized wave into a distribution, while a cap prevents a single seller notification from consuming capacity indefinitely. Capacity planning should include the initial send plus every eligible retry, the number of workers that can become runnable during the same interval, the API quota's refill behavior, and the delay introduced by a database lease. The operational record should show the chosen delay, the cap, and the policy version so an auditor can reproduce the decision without guessing what the worker knew at the time.

Here is the policy boundary in Go. It has no vendor-specific endpoint, so the decision can be tested without network I/O.

package retry

import (
    "math/rand"
    "strconv"
    "strings"
    "time"
)

type Decision struct {
    Retry  bool
    Delay  time.Duration
    Reason string
}

func Next(status int, retryAfter string, attempt int, base, cap time.Duration, r *rand.Rand) Decision {
    if attempt < 0 || base <= 0 || cap < base {
        return Decision{Reason: "invalid retry policy"}
    }
    if status != 429 && status < 500 {
        return Decision{Reason: "non-retryable response"}
    }
    delay := base
    for i := 0; i < attempt && delay < cap; i++ {
        if delay > cap/2 {
            delay = cap
            break
        }
        delay *= 2
    }
    if parsed, ok := parseRetryAfter(retryAfter); ok && parsed < cap {
        delay = parsed
    }
    if r != nil && delay > 0 {
        delay = time.Duration(r.Int63n(int64(delay) + 1))
    }
    return Decision{Retry: true, Delay: delay, Reason: "transient response"}
}

func parseRetryAfter(value string) (time.Duration, bool) {
    seconds, err := strconv.Atoi(strings.TrimSpace(value))
    if err != nil || seconds < 0 {
        return 0, false
    }
    return time.Duration(seconds) * time.Second, true
}
Enter fullscreen mode Exit fullscreen mode

This treats Retry-After as a scheduling input, not permission to wait forever. A production implementation should accept an HTTP-date if its contract requires it, reject values above the policy cap, and record that a malformed header was ignored. The policy is only one part of the design; the evidence model and queue transaction matter just as much.

Where should the attempt record sit in the send path?

Persist the notification job before a worker can claim it. A worker claims one job with a lease, creates an attempt record with a monotonically increasing attempt number, sends with the stable idempotency key, then records the result and next state. The lease timeout must exceed the expected request deadline but remain short enough to recover a crashed worker without creating overlapping sends.

The awkward case is the important one: the remote service accepts the request, then the worker dies before recording the response. On lease expiry, the next worker cannot know whether the message was accepted. That is why the idempotency key, the remote event history when available, and a bounded reconciliation state are more valuable than an aggressive retry count. Marking the job unknown_delivery is honest evidence; pretending it failed is not.

type Attempt struct {
    NotificationID string
    OrderID        string
    Attempt        int
    IdempotencyKey string
    StartedAt      time.Time
    FinishedAt     time.Time
    HTTPStatus     int
    Decision       string
    NextAttemptAt  *time.Time
    PolicyVersion  string
}
Enter fullscreen mode Exit fullscreen mode

Write the attempt and job transition in one database transaction when both live in the same store. Publish a wake-up message after commit, or use an outbox table so a crash cannot leave the database saying retry_scheduled while the queue knows nothing about it. The outbox is easy to inspect, replay, and explain during an audit.

Buy or build the delivery boundary?

The decision is about control and operational load, not a race to the smallest invoice. A managed transport can remove SMTP operations and expose delivery events, while a self-hosted boundary brings deeper routing control at the cost of reputation management, patching, feedback loops, and on-call work.

Boundary Evidence you can usually own Capacity and SLO concern Main trade-off
Managed email API Request IDs, submission status, event callbacks, policy records Quota behavior and callback lag must be measured Less transport toil, more dependency on contract and retention limits
Self-hosted SMTP relay Local queue, logs, configuration, delivery attempts IP reputation, queue growth, and receiving-domain behavior become SLO inputs More control, much larger operations surface
Hybrid boundary Application audit trail plus selected transport events Two failure domains require reconciliation and alerting Stable application evidence with limited transport ownership

Define minimum evidence before selecting an implementation. Ask how long request and event records remain available, whether callbacks are signed, how rate limits are communicated, and how deletion requests affect immutable audit records. A buy decision that cannot answer those questions is an unpriced compliance dependency.

How do you verify and roll back safely?

Unit-test every response class, malformed Retry-After, cap boundary, attempt limit, and clock conversion. Use a fake transport that can return a timeout after accepting the request; the expected result is an explicit unknown-delivery path, not an unbounded replay. Then run a small canary with synthetic seller addresses and a separate mailbox, checking that attempt record, queue state, and downstream event agree.

Watch retry rate by status class, age of the oldest queued notification, queue depth, lease expiry count, duplicate-suppression count, callback lag, and the percentage of jobs ending in unknown delivery. Set an SLO for notification freshness and another for evidence completeness. A system can meet a send-latency target while quietly losing the records needed to prove what happened.

Rollback should disable new retries through a configuration flag while leaving in-flight records readable. Drain or quarantine scheduled jobs according to their policy version; do not rewrite history to make a dashboard look healthy. Keep the previous classifier available until no remaining jobs use its rules.

The practical endpoint is modest: a retry policy that is bounded, observable, and explainable. Exponential backoff protects a struggling dependency, but the audit trail protects the business decision around a marketplace order. Treat both as part of delivery design.

References

Top comments (0)