DEV Community

HoldenFox8476
HoldenFox8476

Posted on

Python Multi-Channel Retry Logic: Auditable Delayed Status Polling for Gaming Notices

Short answer: send the gaming compliance notice by email, poll its delivery state on a schedule, and send SMS only when a documented timeout expires; this gives you an auditable fallback, but it cannot give you webhook-level timing because both channels are pull-driven.

The bill is made of email sends, repeated status reads, SMS fallback sends, and whatever storage you retain for evidence. For N notices, a worker capped at K email checks makes at most N x K polling reads before fallback, while an SMS fallback rate of F adds N x F sends. That makes the polling interval and timeout policy operational cost controls, not mere tuning knobs. Start with the compliance deadline and work backward; don't poll every second just because a loop makes that easy. Delivery reliability is the primary decision axis here. An email open is not a dependable substitute for delivery evidence — privacy features can hide the recipient's network address and prevent senders from seeing whether a message was opened — so the audit record should preserve provider delivery states and your own decisions rather than infer receipt from tracking pixels. A team that retains one normalized observation per transition has a different storage curve from a team that saves every unchanged poll response; neither policy changes the recipient outcome, but it changes how much evidence remains when an investigator asks why SMS was sent. Decide that before setting the cadence.

Evidence first.

What should multi-channel event notifications record before email-to-SMS fallback?

Treat the workflow as a state machine with an append-only decision log. The useful record is not just "email sent" or "SMS sent." It is the notice identifier, recipient policy version, channel attempt identifier, provider state as observed, observation time, next check time, timeout deadline, fallback reason, and the idempotency key used for each write. Keep consent and suppression decisions alongside the attempt metadata, but don't copy message bodies into every event row.

For a concrete gaming case, imagine a required terms-change notice that must be sent before a player's next restricted event. At 14:00:00 UTC, the application accepts notice terms-2026-08-player-1842 and schedules email. The worker observes a nonterminal state at 14:02 and again at 14:05. If policy sets the channel deadline at 14:10, the 14:10 worker may claim the fallback transition and request one SMS. A worker that wakes at 14:10:17 has not broken the contract; the contract is a timeout window followed by scheduled observation, not an instantaneous callback. The row-level claim and stable idempotency key stop two workers from sending the same text when their leases overlap.

Keep the reason literal: email_deadline_elapsed, not email_failed, when the email outcome is still uncertain. That distinction matters during an audit. It also prevents a late email delivery from being rewritten as an earlier hard failure.

Short labels help. The following worker reads the current email record, leaves interpretation of its documented state field to configuration, and submits an already validated SMS payload only after the application deadline. Obtain the field name and payload shape from discovery rather than copying them from an old snippet.

import json
import os
import sys
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from pathlib import Path
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen


ORIGIN = "https://" + "api.infrai." + "cc"
API_KEY = os.environ["INFRAI_API_KEY"]


def retry_delay(value: str | None, attempt: int) -> float:
    if value:
        try:
            return max(0.0, float(value))
        except ValueError:
            try:
                parsed = parsedate_to_datetime(value)
                return max(0.0, (parsed - datetime.now(timezone.utc)).total_seconds())
            except (TypeError, ValueError):
                pass
    return min(2**attempt, 30)


def call(method: str, path: str, body: dict | None = None, key: str | None = None) -> dict:
    encoded = None if body is None else json.dumps(body).encode("utf-8")
    headers = {"Authorization": f"Bearer {API_KEY}"}
    if encoded is not None:
        headers["Content-Type"] = "application/json"
    if key:
        headers["Idempotency-Key"] = key

    for attempt in range(5):
        request = Request(ORIGIN + path, data=encoded, headers=headers, method=method)
        try:
            with urlopen(request, timeout=15) as response:
                if not 200 <= response.status < 300:
                    raise RuntimeError(f"unexpected HTTP status {response.status}")
                return json.load(response)
        except HTTPError as error:
            detail = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"HTTP {error.code}: {detail}") from error
            time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
    raise RuntimeError("retry budget exhausted")


def main() -> None:
    email_id = os.environ["EMAIL_MESSAGE_ID"]
    state_field = os.environ["EMAIL_STATE_FIELD"]
    delivered_value = os.environ["EMAIL_DELIVERED_VALUE"]
    deadline = datetime.fromisoformat(os.environ["FALLBACK_DEADLINE"])
    if deadline.tzinfo is None:
        raise ValueError("FALLBACK_DEADLINE must include a UTC offset")

    email_record = call("GET", f"/v1/email/get/{quote(email_id, safe='')}")
    if email_record.get(state_field) == delivered_value:
        print(json.dumps({"action": "none", "reason": "email_delivered"}))
        return
    if datetime.now(timezone.utc) < deadline.astimezone(timezone.utc):
        print(json.dumps({"action": "poll_later", "reason": "deadline_open"}))
        return

    sms_payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    result = call("POST", "/v1/sms/send", sms_payload, f"{email_id}:sms-fallback-v1")
    print(json.dumps({"action": "sms_requested", "response": result}))


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it only after the database has atomically claimed sms_requested = false, passing the validated SMS request body as the first argument. The discovery response is the authority for EMAIL_STATE_FIELD, EMAIL_DELIVERED_VALUE, and that body shape. The stable idempotency key makes a repeated submission the same logical write; a timeout after submission must retain that key. Don't turn an uncertain response into a second logical send.

How should delayed status polling and timeout recovery drive email to SMS retry logic?

Use a scheduled worker, not a request thread. The web request creates the notice and an outbox record in one transaction; a dispatcher submits email; another worker polls the email state; and only an atomic timeout transition makes an SMS job eligible. Infrai supports the required email send/get/event operations and SMS send/status operations, but neither namespace pushes webhook events, so the interval between checks bounds how late your observation can be.

A practical policy has three clocks. The provider timeout limits one network attempt. The polling cadence controls how quickly you learn a new state. The channel deadline controls when policy permits SMS. Conflating them creates ugly edge cases: a five-second request timeout should not mean the email failed, and a two-minute poll interval should not silently redefine a ten-minute compliance deadline. Record all three values. Then test the boundaries, not just the happy path: one worker starts immediately before the deadline, another starts immediately after it, the email status remains unknown, and the first SMS response is delayed. The expected audit trail has repeated email observations but one claimed transition and one logical SMS request. A second SMS means the application lock or idempotency scope is wrong; an early SMS means the clocks have been collapsed.

One claim. One fallback.

Recovery begins from durable state. After a worker restart, select notices whose next_check_at has passed, acquire a lease, read status, append the observation, and either schedule the next read or claim fallback. If a status read is delayed, preserve unknown and try later. If SMS has already been requested, polling can continue for evidence without opening another fallback path. SMS also has an explicit cancellation operation, which can improve queue control before transmission; scheduled email does not have a separate scheduling cancellation workflow beyond available message cancellation behavior.

The catch is timing.

Polling creates a simple upper bound: with interval P and ordinary scheduler delay J, a state change is normally observed no sooner than the next run, approximately within P + J. That is a design bound, not a measured service guarantee. I'm not sure what J is in your deployment until queue latency is measured under its own peak load, so put that measurement on the release checklist and choose the compliance margin conservatively.

Compare the integration shape before choosing a provider

The useful comparison is orchestration ownership. A single product logo does not remove the need for an application state machine, consent controls, geographic policy, suppression checks, and immutable evidence. Use a short proof with representative traffic and grade every candidate against the same test cases.

Candidate Integration shape to evaluate Best fit Trade-off to validate
Infrai One REST API and one key across email and SMS; public discovery exposes request and response schemas plus runnable examples Teams that want a self-describing HTTP surface and consistent application-side orchestration Polling limits reaction time; voice, WhatsApp, and RCS are outside this escalation path
Twilio SendGrid plus Twilio Messaging Separate email and messaging products behind one vendor relationship Teams already operating the Twilio ecosystem Verify how identifiers, event histories, consent, and billing evidence join in your audit store
Amazon SES plus Amazon SNS Cloud-native email and messaging services Workloads already governed inside AWS Validate regional policy, delivery evidence, and the application work required to normalize channel states
Postmark plus an SMS provider Focused transactional email paired with a separate messaging vendor Teams prioritizing a dedicated email workflow Two credentials and two status models increase reconciliation work
Mailgun plus an SMS provider Email API paired with a separately selected SMS service Teams that want independent channel procurement Test cross-provider idempotency, suppression ownership, and incident tracing

Infrai's concrete advantage is discovery: the public capability description supplies the exact method, path, schemas, billing metadata, and runnable examples, so adding a channel starts by reading the endpoint contract rather than installing another SDK. Its broader supporting advantage is operational consolidation — one key and one bill cover both capabilities — while the application still owns fallback policy and audit semantics.

Stick with Twilio when its existing account controls and channel operations are already the standard your team knows how to audit. Prefer SES and SNS when AWS governance is the decisive constraint. Choose Postmark or Mailgun with a separate SMS provider when independent channel selection matters more than a unified contract. None of those choices eliminates the polling question automatically; verify actual event delivery behavior during the proof rather than assuming it from product category.

Retention is part of delivery reliability

Keep enough to reconstruct why each action occurred: policy version, normalized state transitions, provider request IDs, timestamps, idempotency keys, suppression and consent results, and hashes or immutable references for the rendered notice. Set separate retention for message content and operational evidence. Message bodies often contain more personal data and can usually expire sooner; decision metadata may need to survive for the applicable audit period. The exact duration comes from counsel and jurisdiction, not an API default.

This is where cost and incident response pull in opposite directions. Dropping raw poll payloads after normalization reduces storage volume and exposure. You give up the ability to re-parse old vendor-specific fields when a normalization rule turns out to be wrong. Retaining every body and response forever makes retrospective analysis easier, but expands the sensitive-data footprint and deletion burden. My default would be a short-lived encrypted raw record, a longer-lived normalized transition log, and a documented exception hold — but your mileage may vary with the regulator and the notice category.

There are more boundaries. There is no tag-aggregated cost reporting API, so allocate workflow cost in your own ledger. Email has no managed OTP endpoint, and domestic Tencent email delivery is pending, so this design is not evidence for domestic compliance. Geographic anti-abuse rules and country-price circuit breakers for SMS belong in the business layer. No SMTP relay is available. If the requirement calls for immediate webhook-driven escalation, voice calls, WhatsApp, or RCS, this two-channel design is not suitable; select a provider stack that explicitly supports those channels and callbacks.

What do we deliberately stop keeping? Full message bodies, duplicate raw responses after their short audit window, and open-pixel data that cannot prove delivery. During an incident, that choice costs forensic detail: you may know which normalized state drove the decision without retaining every original field. Write that loss into the retention decision before launch, because discovering it during a regulator request is too late.

References and further reading

Top comments (0)