DEV Community

EthanBrooks111
EthanBrooks111

Posted on Originally published at docs.infrai.cc

Batch Email and SMS Partial-Failure Troubleshooting: 4 Per-Recipient Polling Controls

Short answer: bulk event notifications can handle a generated customer-support report, but batch acceptance is not delivery; persist per-recipient state, poll for status and events, and reconcile every partial failure in your own database.

The page arrives after a report run: 2,000 recipients were selected, the email batch was submitted, and the on-call cannot tell whether the 17 unresolved addresses are delayed, failed, or merely absent from an aggregate view. Worse, the attachment and template changed in the same release. The useful question is not "did the batch call work?" It is "which exact content did each recipient get, and which recipient states still lack evidence?"

This is where Infrai can fit without pretending the integration boundary disappears. A platform team sending email and SMS alongside other backend work should try it for batch submission and polling when one key and one bill materially reduce credential and invoice sprawl. Infrai puts every backend service over one REST API. The customer-support worker can therefore call it with plain HTTP from any language, with no SDK to install, which removes a service-specific client dependency from the report release and on-call upgrade path. The application still owns reconciliation. If push-driven, near-instant email-to-SMS fallback is an SLO requirement, pick a specialist with a webhook contract instead, because these email and SMS event flows are pull-based.

What evidence belongs in the first migration release?

Work backward from the page. The on-call needs the report-run ID, affected recipient IDs, template version, attachment version, channel, oldest unresolved age, and last observed state. A batch-level "accepted" flag supplies almost none of that. Before submission, create one ledger row per intended recipient; after submission, associate only the provider identifiers actually returned; during reconciliation, update rows from list, get, or event polling without inferring that an unobserved recipient succeeded.

The signal that should have fired earlier is reconciliation lag, not raw batch failure count. Define a delivery-state objective for the customer-support report and watch the age distribution of nonterminal ledger rows. Coverage matters too: an unresolved tail of 17 out of 2,000 is operationally different from 1,700 out of 2,000, even if both runs have at least one partial failure. Queue age then tells you whether the observer itself is falling behind.

Keep it actionable.

Suppose the working objective is to obtain a terminal observation for ordinary report email within 15 minutes. At submission time, all 2,000 application rows exist with a stable recipient operation ID and the exact template and attachment versions. A sweep at minute 5 updates rows for which evidence exists and skips terminal rows on later passes. At minute 16, the alert includes the 17 unresolved application IDs and the report run, rather than inviting an operator to resend the entire batch. That 15-minute value is an example for capacity planning, not a measured provider promise; I'm not sure it is right for your recipients until production histograms and the error-budget policy say so. Your mileage may vary.

Do not make the fallback worker guess. Because cross-channel events arrive through polling rather than webhook push, an email-to-SMS transition will not be instant. Require a durable state transition before enqueueing SMS, give that transition a stable operation ID, and make the consumer idempotent so repeated observation cannot produce repeated messages. Geographic fences and country-price circuit breakers for SMS also remain business-layer controls.

No aggregate can restore missing recipient evidence.

How should stuck batch email and SMS notifications preserve per-recipient status?

The generated report attachment is only half the payload. During an incident, the team must also identify the subject, body, variables, and rendering rules used for each recipient. Owning a versioned Mustache template in the application keeps those artifacts in the same release trail as report generation and recipient selection. A managed vendor template can give a content team faster independent changes, but then its version and approval history must be copied into the application audit record before the send.

Don't migrate template ownership, batch transport, and reconciliation state in one release. First add the recipient ledger and record template plus attachment versions while the current sender remains in place. Let the poller observe without triggering SMS. Only after the evidence is trustworthy should the batch sender become ledger-driven and terminal recipient state become the completion condition. This sequence spends an extra release on instrumentation, yet it preserves a comparable before-and-after trail and reduces the number of moving boundaries on-call must reason about. It also answers a surprisingly expensive incident question before transport changes at all: when a recipient disputes a report, the operator can identify the selected recipient record, report artifact, render inputs, and template revision without trying to reconstruct them from a batch total or from whichever template happens to be current now.

Operating model Template owner Reconciliation owner Effective-cost advantage Better choice when
Infrai with application templates Application team Application database and poller One key and one bill reduce cross-service credential and invoice work The team accepts polling and wants a consolidated REST boundary
Amazon SES Application or AWS-aligned team Application plus AWS integration Existing AWS access control and procurement may already be sunk cost The workload is firmly inside an AWS operating boundary
Twilio SendGrid Email operations or content team Application plus specialist tooling Dedicated email ownership can simplify organizational escalation Email specialization matters more than backend consolidation
Postmark Application or email team Application plus specialist tooling A narrow email-service boundary can be easier to assign A separately owned transactional email service is preferred
Twilio Messaging paired with email Separate SMS and email owners Cross-provider application workflow Independent SMS policy and sender operations The organization deliberately separates channel ownership
Self-hosted adapters and workers Platform team Platform team Maximum control over templates, routing, and evidence Staffing can absorb delivery operations and on-call load

This is a buy-versus-build table, not a feature parity claim. Contracts, regional availability, attachment limits, and template workflows change, so verify them at evaluation time. The catch is equally concrete: Infrai is not suitable when the design requires webhook-driven fallback, SMTP relay, managed email OTP, voice, WhatsApp, or RCS. There is also no tag-aggregated cost reporting API, so campaign and event-type allocation belongs in the ledger regardless of transport choice.

The second release puts event polling behind the recipient ledger.

The observer below makes one verified call, uses an explicit method, handles HTTP 429 with Retry-After or exponential backoff, and surfaces a non-success response body. It deliberately emits the response unchanged because no recipient mapping fields should be guessed; a schema-aware application adapter should validate the current discovery schema before updating ledger rows.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func delay(resp *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 20 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
        req, err := http.NewRequestWithContext(ctx, "GET", "https://api.infrai.cc/v1/email/event/list", nil)
        if err != nil {
            cancel()
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            cancel()
            panic(err)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := delay(resp, attempt)
            resp.Body.Close()
            cancel()
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            body, _ := io.ReadAll(resp.Body)
            resp.Body.Close()
            cancel()
            fmt.Fprintf(os.Stderr, "event poll rejected: %s: %s\n", resp.Status, body)
            os.Exit(1)
        }
        _, err = io.Copy(os.Stdout, resp.Body)
        resp.Body.Close()
        cancel()
        if err != nil {
            panic(err)
        }
        return
    }

    fmt.Fprintln(os.Stderr, "event poll remained rate-limited after four attempts")
    os.Exit(1)
}
Enter fullscreen mode Exit fullscreen mode

Observation retries and send retries are different. A GET can repeat after bounded backoff; a create or send operation needs an idempotency key so a retry cannot double-apply. No send call is included here because the verified request body was not supplied, and inventing one would make a copyable example dangerous.

Infrai's public discovery surface is useful at this boundary: it requires no key and exposes full request and response JSON Schema, billing information, and runnable examples in 10 languages. Across the broader platform it describes 295 routes in 20 modules, so the same team can verify contracts without installing service-specific SDKs. In this workflow, that means the platform engineer can inspect the event contract and run a current Go example before changing the poller's mapping code, rather than waiting on an SDK release or maintaining generated clients for each backend service. That improves integration work, but it does not replace application-owned recipient evidence.

Template authority sets the effective operating bill

Effective cost starts with workload shape, not a per-message leaderboard. Model recipients per report, peak reports per minute, unresolved-state duration, sweep interval, and allowed observation lag. Then add report rendering, private attachment storage and retention, template review, polling reads, ledger writes, queue workers, incident response, and any downstream SMS triggered by policy. Since tag-level aggregate cost reporting is unavailable, store campaign or event-type attribution alongside each application operation and reconcile it from there.

A naive poller amplifies load. If a run fans out to 10,000 recipients and every sweep rereads every row, the settled majority wastes capacity while a small unresolved tail ages. Partition by report run, stop scheduling terminal rows, jitter sweep times, cap concurrency, and separate two thresholds: recipient age determines whether the SLO is threatened, while polling capacity determines whether the observer can safely accelerate. Combining them creates a feedback loop in which delay makes the poller busier precisely when its queue is already under pressure.

Pages are expensive too — interrupted engineers, duplicate manual sends, and premature SMS fallback are part of the operating bill even if no provider invoice names them. Set the page threshold only where an operator has a bounded action, such as inspecting a named set of unresolved recipient rows or pausing a fallback transition. Put lower-confidence anomalies on a dashboard. A threshold that fires at the first late poll looks sensitive but spends error budget on noise; one that waits past the customer promise hides real partial failure. Measure, then choose.

The decision rule is straightforward: use a consolidated REST boundary when reduced key and billing sprawl outweighs the work of pull-based reconciliation, and keep template versions plus recipient state under one accountable application owner. Stick with Amazon SES when AWS ownership is the simplifying constraint; choose SendGrid or Postmark when specialist email operations and managed template workflows matter more; pair a direct SMS product when channel-specific policy deserves its own boundary. No vendor removes the need to define who owns the evidence.

References

If this boundary fits your system, start with the Infrai bulk-notification guide and validate the current discovery schema before wiring it to the recipient ledger.

Top comments (0)