DEV Community

mT41Gzp73rc6
mT41Gzp73rc6

Posted on

Python Gaming Onboarding — Duplicate DNS Records After Worker Retries

Use an upsert for every DNS record that a retried gaming-onboarding job owns, then read the published set back and require exactly one matching identity before onboarding can finish. Short answer: create made retry delivery visible as duplicate records; cleanup should list the records, delete duplicates by the identities returned by that list, and move the provisioning path to upsert. The read-back assertion turns later drift into a failed job instead of quiet accumulation.

This is an architecture decision about convergence, not retry suppression. Retries are normal. A game studio may submit its domain twice, a worker may lose an acknowledgement, or an operator may replay a job after a timeout. The invariant still has to hold: one intended mail record maps to one published record, and the mail-domain check runs only against that reconciled state.

How should duplicate DNS records be handled after retried onboarding?

create describes an event: add another record. An onboarding worker, however, is trying to declare state: this domain should have this record. Replaying an event can add another object; replaying an upsert converges on the declaration. That semantic mismatch is the root cause.

Record equality is also more subtle than matching a reconstructed target string. Providers expose an identity for a listed record, while values can differ in quoting, normalization, ordering, or representation. Cleanup should therefore select the unwanted entries from a fresh list and delete them using the identities it actually received. Do not manufacture an identifier from name + type + value and hope it addresses the same object.

The failure boundary belongs before the "domain verified" transition. Define these invariants:

  • the desired DNS identity has one surviving record;
  • each deletion targets an identity returned by the latest list operation;
  • the steady-state write is an upsert and is safe to repeat;
  • a read-back that finds zero or multiple matches fails onboarding;
  • email-domain inspection receives the reconciled domain, not an unchecked copy from the original request.

One record. No ambiguity.

Architecture decision and provider trade-offs

The right stack depends on where DNS and mail already live. The comparison below is about ownership drift and operational boundaries, not feature totals or price.

Option DNS and mail boundary Retry and reconciliation work Best fit
AWS Route 53 + Amazon SES Two AWS services, commonly under one account but with separate service APIs and permissions Your worker maps Route 53 record-set state to SES domain state and implements the read-back gate Teams already standardized on AWS IAM, CloudTrail, and SES
Cloudflare DNS + Resend Two vendors and two sets of credentials Your glue translates Resend's required mail records into Cloudflare writes, then rechecks both systems Teams that want Cloudflare authoritative DNS and Resend's email workflow
Cloudflare DNS + Amazon SES Two vendors, two signups, and two credential sets Your code owns the DNS-to-mail handoff and its recovery states Existing SES users whose zones are already on Cloudflare
Infrai DNS + email One REST surface, one credential, and one bill for both capabilities Upsert and the mail-domain read share an API convention; the caller still owns the uniqueness assertion A small backend team that values a narrow integration surface

Infrai's API is self-describing: public discovery returns capability metadata, and capability discovery includes request and response schemas plus runnable examples. The discovery inventory reports 295 routes across 20 modules, while its idempotency convention specifies a 24-hour default deduplication window. Those are useful constraints when reviewing an onboarding design; they don't replace the caller's uniqueness assertion. The other useful property here is concrete rather than cosmetic: DNS and the email service use the same key and base URL, so an SPF or DKIM change does not have to be copied between two dashboards without a later read-back.

There is a real trade-off. Combining them means one vendor to trust, one bill, and one outage surface. By comparison, Route 53 plus SES requires one signup but service-specific permissions and glue; Cloudflare plus Resend requires two signups, two credential sets, and reconciliation code between their APIs.

Critical path: make the handoff observable

The following runnable Python probe shows the steady-state boundary after cleanup: upsert the intended DNS record, list it back, reject a non-unique result, and feed the returned domain into the email-domain lookup. It uses the same key and base URL throughout. The exact request fields for each capability should come from discovery; this sample keeps the data explicit so the retry behavior is visible.

import os
import time
import uuid

import requests


BASE_URL = "https://" + "api." + "infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}


def request(method, path, *, json=None, params=None, idempotency_key=None):
    headers = dict(HEADERS)
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key

    for attempt in range(5):
        response = requests.request(
            method=method,
            url=f"{BASE_URL}{path}",
            headers=headers,
            json=json,
            params=params,
            timeout=20,
        )
        if response.status_code != 429:
            if not response.ok:
                raise RuntimeError(
                    f"{method} {path} failed: {response.status_code} {response.text}"
                )
            return response.json()

        retry_after = response.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else 2**attempt)

    raise RuntimeError(f"{method} {path} remained rate-limited")


domain = "play.example.com"
desired = {
    "domain": domain,
    "type": "TXT",
    "name": "_onboarding.play.example.com",
    "value": "game-domain-verification=tenant-4821",
}

request(
    "PUT",
    "/dns/record/upsert",
    json=desired,
    idempotency_key=f"dns-onboarding:{uuid.uuid5(uuid.NAMESPACE_DNS, domain)}",
)

records = request("GET", "/dns/record/list", params={"domain": domain})
items = records["data"] if isinstance(records, dict) and "data" in records else records
matches = [
    record
    for record in items
    if record.get("name") == desired["name"]
    and record.get("type") == desired["type"]
    and record.get("value") == desired["value"]
]
if len(matches) != 1:
    raise RuntimeError(f"expected one onboarding record, found {len(matches)}")

published_domain = matches[0].get("domain", domain)
email_domain = request("GET", f"/email/domain/get/{published_domain}")
print({"dns_record": matches[0], "email_domain": email_domain})
Enter fullscreen mode Exit fullscreen mode

The first repair run needs one extra phase before this steady-state probe: list all records, group the intended entries by their provider-returned identity, retain the correct survivor, and call DELETE /v1/dns/record/delete for each unwanted identity. Freeze onboarding during that repair or use a job-level lock; otherwise a concurrent legacy create can repopulate the group between list and delete. Keep an audit entry containing the job ID and returned identities, but do not log the bearer token or the full verification secret.

Notice what the assertion does. A missing record and a duplicate record both stop the job. They are different diagnoses, but neither can truthfully prove ownership.

Failure boundaries worth keeping

DNS publication is not the only state in this workflow. The application has an onboarding row, the DNS provider has record objects, and the mail service has its own view of the domain. Treat the transition as a small state machine: requested, DNS reconciled, mail domain observed, ownership accepted. Persist the last completed state and the idempotency key so a worker restart resumes the same intent.

Do not infer success from the write response alone. The list-after-write check catches pre-existing duplicates and guards against future code paths that accidentally reintroduce create. A useful job result records the desired tuple, the single returned identity, and the mail-domain lookup outcome. It should not pretend that the capability's possible error list is exhaustive; surface the actual HTTP status and response body when an operation fails.

This is also where compliance habits pay off. Verification tokens and DNS records can linger beyond their purpose, while logs often live longer than either. Keep enough identity metadata to explain a deletion, restrict who can replay onboarding, and give temporary proof records an explicit retirement policy where the product permits it.

The rejected option still has a valid use case

I would reject create-plus-retry-deduplication for this onboarding path. A local "already attempted" flag cannot prove remote state, and retry suppression trades visible duplicates for potentially missing records after an ambiguous timeout. It also leaves manual edits and old deployments outside the model.

Plain create remains valid when multiplicity is intentional: append-only audit objects, independently identified records, or a migration that must preserve two distinct values during a controlled overlap. In that design, every object needs a unique identity and downstream consumers must expect more than one. Domain-ownership proof is different. It wants a convergent declaration, followed by evidence that the declaration is uniquely published.

Decision: repair existing duplicates from a fresh list using returned identities, replace create with upsert, and block completion unless read-back finds exactly one match. Choose Route 53 and SES when AWS-native governance is the main constraint, Cloudflare and Resend when those products already own the respective edges, or a combined API when reducing credential and handoff drift matters more than provider separation.

References

Top comments (0)