Silhouette72591483

Posted on Aug 22

Reliability Drills for Developers Comparing API-First Transactional Email Alternatives

#email #fintech #reliability

Decision: developers sending a generated fintech report should choose a transactional email API by its ability to preserve identity and delivery evidence across ambiguous failures, not by the cheapest advertised message rate. Store the approved report immutably, commit a delivery intent with it, submit outside the business transaction, and reconcile authenticated events before calling the report delivered.

That rule is stricter than the one I would use for a welcome email. A welcome message can often be regenerated from current profile data; a financial report may represent a specific ledger cutoff, so a retry that silently attaches newer bytes is not a retry at all. It's a different communication wearing the same filename.

An API-first interface also doesn't remove SMTP from the downstream delivery system. It removes the application's SMTP relay integration and gives the application an HTTP submission boundary. Past that boundary, acceptance, transfer, mailbox policy, and user action remain distinct. Fast acceptance is useful. It isn't proof of delivery.

How can developers test API-first transactional email alternatives?

Test the evidence chain under failure. The minimum useful exercise submits the same authorized report through each adapter, deliberately loses one client response, delivers one event twice, sends another event out of order, and verifies that an operator can still answer four questions: which bytes were approved, which recipient was authorized, how many attempts occurred, and what external evidence supports the current state.

Do not start with a broad feature checklist. Start with a narrow state machine: PENDING, IN_FLIGHT, UNKNOWN, ACCEPTED, DELIVERED, BOUNCED, and SUPPRESSED. UNKNOWN matters most because a client timeout cannot reveal whether the remote service accepted the request before the connection disappeared. Blindly retrying converts uncertainty into possible duplication; marking the attempt failed merely hides the uncertainty.

The test fixture should contain a small PDF, a larger PDF near the application's own ceiling, a Unicode filename, and a stable SHA-256 digest. Use a synthetic recipient domain under team control and never put production financial data into a delivery test. Record request correlation IDs, provider message IDs when returned, event IDs, normalized status, original diagnostic data, and timestamps. Keep secrets and full attachment content out of logs.

I'm not sure which candidate will fit a particular team before this test runs. Documentation can establish request formats and stated event behavior, but it cannot establish the team's recovery time, regional network behavior, or whether the on-call engineer can trace an uncertain attempt at 03:00.

Data retention defines the evidence boundary

The storage boundary comes first. Generate the report, write it to immutable object storage or an equivalently versioned store, calculate its digest, and commit the object version, digest, byte count, recipient authorization, and application delivery ID before making it eligible for submission. A worker may create multiple attempt records, but the business delivery ID remains singular.

Never regenerate the attachment during a retry.

That short rule prevents an ugly class of audit failures. Imagine a report authorized at ledger cutoff 2026-08-18T16:00:00Z; object version 01J5C7REPORT has digest 8b7a...e219, the first request leaves worker attempt A-001, and the response is lost just before a correction posts. The delivery ledger now says UNKNOWN, which is uncomfortable but accurate. If the worker asks the reporting service to rebuild monthly-report.pdf, attempt A-002 can contain different balances while retaining the same subject, recipient, and apparent intent; if it instead fetches the committed object version and checks the digest, both attempts refer to the authorized bytes. The operator can then query external evidence using the stable delivery ID, inspect any returned provider message ID, and decide between waiting and replay according to the documented duplicate-versus-delay policy. Without those identities, a dashboard may show one business delivery even though two byte sequences crossed the boundary, and neither a filename nor a timestamp can reconstruct which one the customer received. An immutable object version plus a committed digest turns that mismatch from a guess into a check performed before submission and an assertion that can be explained afterward.

No guesswork.

The other invariants are equally unglamorous:

A network call never runs while the ledger transaction is open.
One outbox row represents one authorized delivery; each submission gets a separate attempt row.
A timeout becomes UNKNOWN, not FAILED.
Provider acceptance and mailbox delivery are separate states.
Events are untrusted until their signatures and replay constraints are verified.
Duplicate or out-of-order events cannot move a terminal state backward.

DMARC belongs in this review, although it solves a different problem. RFC 7489 defines domain-based policy and reporting around aligned identifiers; it does not turn an accepted submission into evidence that a recipient read a message. Treat SPF, DKIM, and DMARC configuration as authentication and domain-governance work, while the delivery ledger handles application evidence. Combining those concerns into a single sent = true field makes both harder to inspect.

Compare candidates after the recovery drill

“Cheapest” has no stable engineering meaning without monthly volume, attachment sizes, event traffic, support expectations, retention, and labor for ambiguous recovery. Published price sheets can populate a workload model, but they should not decide it. For this report path, first compare who owns each failure and which evidence crosses the interface.

Candidate	Submission and evidence surface	Boundary to verify in a contract test	Operational trade-off
SendGrid	Mail Send API and Event Webhook	Preserve its message identifiers and map accepted, delivered, deferred, bounced, and dropped evidence without collapsing states	A broad event vocabulary needs an explicit internal mapping
Amazon SES	API submission and event publishing	Include region, verified identity, configuration, and event destination in deployment review	Cloud account and regional configuration become part of delivery operations
Mailgun	Messages API and signed webhooks	Verify signatures before state changes and test duplicate event handling	Multipart request translation differs from a JSON-only internal port
Postmark	Email API plus delivery and bounce webhooks	Retain message identifiers and test the application's bounce policy	Stream and sender configuration remain operational inputs outside application code

This table is a test plan, not a ranking. The products expose different request and event models, so the application should own a small internal port and give every adapter the same contract suite. Don't force every provider into the richest candidate's vocabulary; normalize only the states that drive a real business decision, then retain the original event for investigation.

Cost comes after those boundaries are acceptable. Model message volume, encoded attachment size, any required add-ons, retention, and support as separate terms; also estimate operator time for an unknown attempt and a duplicate report. Your mileage may vary — especially when attachment distributions are heavy-tailed — and a single average size will conceal the workers that hit memory or request limits.

Implement the critical path in Python

The following Python is an application port, not a claim that commercial services accept one common payload. MAIL_API_URL is a fully configured adapter endpoint owned by the deployment, and the adapter translates this internal request into the selected service's documented schema. The code verifies the committed digest, sends one application delivery ID, and refuses to reinterpret a lost response as a clean rejection.

import base64
import hashlib
import json
import os
import socket
import urllib.error
import urllib.request
from dataclasses import dataclass
from enum import Enum


class AttemptState(str, Enum):
    ACCEPTED = "accepted"
    REJECTED = "rejected"
    RETRYABLE = "retryable"
    UNKNOWN = "unknown"


@dataclass(frozen=True)
class Delivery:
    delivery_id: str
    recipient: str
    object_version: str
    filename: str
    expected_sha256: str
    content: bytes


@dataclass(frozen=True)
class AttemptResult:
    state: AttemptState
    external_id: str | None
    diagnostic: str


def encoded_report(delivery: Delivery) -> str:
    actual = hashlib.sha256(delivery.content).hexdigest()
    if actual != delivery.expected_sha256:
        raise ValueError("attachment differs from the authorized object version")
    return base64.b64encode(delivery.content).decode("ascii")


def submit(delivery: Delivery, timeout_seconds: float = 10.0) -> AttemptResult:
    payload = {
        "delivery_id": delivery.delivery_id,
        "recipient": delivery.recipient,
        "subject": "Your requested account report",
        "attachment": {
            "filename": delivery.filename,
            "content_type": "application/pdf",
            "content_base64": encoded_report(delivery),
            "object_version": delivery.object_version,
            "sha256": delivery.expected_sha256,
        },
    }
    request = urllib.request.Request(
        os.environ["MAIL_API_URL"],
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {os.environ['MAIL_API_TOKEN']}",
            "Content-Type": "application/json",
            "X-Delivery-ID": delivery.delivery_id,
        },
        method="POST",
    )

    try:
        with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
            body = json.load(response)
            return AttemptResult(
                AttemptState.ACCEPTED,
                body.get("message_id"),
                "submission accepted",
            )
    except urllib.error.HTTPError as error:
        if error.code == 429 or 500 <= error.code < 600:
            return AttemptResult(
                AttemptState.RETRYABLE, None, f"HTTP {error.code}"
            )
        return AttemptResult(AttemptState.REJECTED, None, f"HTTP {error.code}")
    except (TimeoutError, socket.timeout, urllib.error.URLError) as error:
        return AttemptResult(
            AttemptState.UNKNOWN, None, type(error).__name__
        )

The header above is correlation metadata only. It must not be described as provider-side idempotency unless the selected service explicitly documents that contract. An application uniqueness constraint can stop two workers from concurrently claiming the same outbox row, but it cannot deduplicate an attempt already accepted beyond the application's boundary. For an UNKNOWN result, reconcile through documented provider evidence when available; otherwise apply a written duplicate-versus-delay policy and require an operator decision where the financial risk warrants it.

The sample holds the attachment in memory to keep the critical path visible. That is not suitable when encoded reports can approach the worker's memory budget. Use a documented streaming or upload mechanism where the chosen interface offers one, set an application ceiling below every downstream limit, and include base64 expansion in concurrency calculations. The exact safe ceiling has to come from the selected interface and measured worker envelope, not from a generic article.

Webhook ingestion completes the state machine. Verify the documented signature over the exact bytes and within the documented replay policy, insert the external event ID under a uniqueness constraint, store the raw event with restricted access, and apply a monotonic transition in one database transaction. Acknowledging first and persisting later creates an evidence gap. Persisting without deduplication creates a replay problem.

Rollout criteria for the synchronous relay path

The rejected option for this system is synchronous SMTP relay from the report-generating request. It couples ledger work, PDF generation, network submission, and user-visible latency; after a disconnect, the request handler still cannot prove whether the relay accepted the message. Holding a database transaction open across that path makes lock duration depend on a remote system, while retrying the whole request risks rebuilding the report and duplicating delivery.

Stick with SMTP relay when an existing mail transfer agent is already the controlled organizational boundary, the team has mature queueing and delivery telemetry around it, and application portability through the standard protocol matters more than API-level event integration. The catch is that the application still needs a durable intent, stable attachment bytes, attempt identity, and a reconciliation model. Changing the submission protocol does not erase those obligations.

A signed download link can also be preferable to an attachment when policy permits it, reports are large, or access must be revocable and audited at retrieval time. It changes the customer experience and introduces object-access expiry, authorization, and availability decisions, so it is not a free reliability upgrade. For a required attachment, keep the immutable-byte design and prove it with failure injection before comparing convenience or price.

The final decision record should therefore name the chosen adapter, its tested limits, the event-to-state mapping, DNS ownership, attachment ceiling, retry policy, unknown-attempt policy, retention period, and exit plan. No logo settles those questions. Evidence does.

DEV Community