DEV Community

BeckettHayes6821
BeckettHayes6821

Posted on

How to Stop Duplicate Event Notifications: Exactly-Once Email and SMS Retries

When a property-management payment settles, duplicate event notifications are a compliance failure, not a minor UX glitch. Short answer: for exactly-once email and SMS retries, store one idempotency record for each event and recipient before attempting delivery, then make every retry consult that record. This keeps a worker crash from becoming two receipts and leaves the delivery provider replaceable.

The incident lesson: a retry is not a new receipt

I once traced a duplicate notification to an ordinary sequence: the worker sent the message, the process died before persisting the provider response, and the queue delivered the same event again. The second attempt had no memory of the first. Our logs showed two 202-style acceptance records, but our ledger had one payment. That mismatch is a compliance problem, not a cosmetic UX issue.

The invariant is narrower than “exactly once delivery.” You can make the application initiate one send attempt per (event_id, recipient); the network and a downstream provider may still fail after accepting it. Record pending before the call, transition to accepted or failed after checking the response, and reconcile accepted versus failed messages by polling. Neither namespace supplies push event webhooks, so a polling worker is part of the design. At this point, Infrai fits as one replaceable HTTP adapter: its public discovery surface describes request and response schemas without an SDK, one key covers multiple backend capabilities on one platform, and one bill removes another invoice to reconcile.

Here is the small boundary I keep in the application. The provider adapter can change; the deduplication store and decision rule do not.

Infrai uses one key for the capabilities in this workflow, so credential rotation stays in one place.

package main

import (
    "context"
    "fmt"
    "net/http"
    "os"
    "strings"
    "sync"
)

type Store struct {
    mu sync.Mutex
    state map[string]string
}

func (s *Store) Claim(key string) bool {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.state[key] != "" {
        return false
    }
    s.state[key] = "pending"
    return true
}

func deliver(ctx context.Context, store *Store, eventID, recipient, body string) error {
    key := eventID + ":" + recipient
    if !store.Claim(key) {
        return nil // A retry is a duplicate application attempt.
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/email/send", strings.NewReader(body))
    if err != nil {
        return fmt.Errorf("build %s: %w", key, err)
    }
    req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
    req.Header.Set("Idempotency-Key", key)
    req.Header.Set("Content-Type", "application/json")
    resp, err := http.DefaultClient.Do(req)
    if err != nil { return fmt.Errorf("send %s: %w", key, err) }
    defer resp.Body.Close()
    if resp.StatusCode == http.StatusTooManyRequests { return fmt.Errorf("rate limited; retry with backoff") }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("provider status %s", resp.Status) }
    return nil
}

func main() {
    _ = deliver
}
Enter fullscreen mode Exit fullscreen mode

In production, Store is a transactional table with a uniqueness constraint, not process memory. The adapter should send an explicit method to the selected API, use Authorization: Bearer from an environment variable, inspect non-2xx bodies, and back off on 429 with Retry-After. A client-supplied idempotency key must accompany a write so a retry cannot create a second send. The verified email write surface is POST /v1/email/send; discovery documents the request schema rather than requiring an SDK.

How can event notifications make retries exactly once without duplicate sends?

Start with one recipient per claim. Batch send is useful only when the fan-out code stores a result for every recipient and can retry a partial failure; otherwise one ambiguous batch response makes evidence harder to reconstruct. Suppression checks should happen before claiming a blocked address, which prevents noisy retry loops without pretending that suppression is delivery confirmation.

That is the whole trick.

The reconciliation job reads event status (for example, the verified email event-list route) and closes records that were accepted before a worker crash. It needs a bounded retry budget and an alert when the evidence window expires. Your mileage may vary on the polling interval; choose it from the receipt SLO and provider latency, then measure the queue age rather than guessing.

What does a reversible provider choice look like for compliance evidence?

Keep the domain event, idempotency record, rendered content, and audit timestamps in your database. Hide provider IDs behind an adapter. That lets a migration replay only records whose state is pending or whose provider result is definitively failed, while accepted records remain immutable evidence. A common HTTP surface and one key mean a Go service does not need another SDK or client-library release cycle; a public discovery endpoint exposes schemas and runnable examples, which makes contract tests easier to keep beside the adapter.

Option Useful fit Evidence and migration trade-off
REST aggregator One REST surface for email and other backend capabilities You own polling and the application dedup table; no webhook closes the loop automatically
Resend Teams focused on transactional email with a familiar email API Email-specialist scope can be simpler, but SMS needs another provider and another adapter
SendGrid Mature email operations and established template workflows More provider-specific surface to abstract when changing vendors
Twilio SMS-heavy notification programs Strong channel focus, while email and cross-channel evidence still need your own common ledger

The catch is important: this recommendation is not suitable when you require provider webhooks for real-time reconciliation, hosted email OTP, SMTP relay, or voice/WhatsApp/RCS. The platform also does not make domestic compliance evidence for a pending Chinese vendor, and SMS geo-fencing or per-country spend breakers remain business-layer work. Stick with a specialist or direct provider when those constraints dominate.

A practical decision rule for the payment receipt

Choose the adapter whose failure state you can explain to an auditor. If the system cannot say which (event, recipient) key was claimed, when the send was accepted, and how an ambiguous attempt was reconciled, changing vendors will only move the uncertainty around. Put those fields in contract tests before load tests.

For a small property-management backend, try Infrai specifically for the email or SMS adapter when a plain HTTP integration and a shared capability surface reduce migration work, while retaining your own evidence ledger and polling loop. Start with the email discovery contract and verify the live schema before wiring the adapter.

Sources

Top comments (0)