TL;DR: A retried onboarding job produced duplicate mail DNS records because it treated provisioning as creation. Read the current records, delete duplicate objects by the identities returned by that read, replace future creates with upserts, and fail the job unless a final read proves that each intended record has exactly one surviving identity. For company mail, that last assertion matters: a successful write response is not deliverability evidence; the converged DNS state is.
The bill for this mistake is mostly not the price of a DNS request. It is retained ambiguity. If one onboarding attempt intends three records and runs four times, a create-only path can leave as many as twelve record objects for operators and later automation to interpret. That is a simple upper-bound model, not a claim about every provider: retained_objects = intended_records * successful_create_attempts. The change that moves the dominant term is convergence, because an upsert holds the desired cardinality at three rather than letting it grow with attempts.
It compounds.
Short answer: make retries converge on one record identity per intended mail record, then verify cardinality from a fresh read. Cleanup is necessary once. Upsert plus read-back is the durable fix.
How should duplicate DNS records be repaired after retried onboarding?
Retries are normal. Networks time out after a server has committed a write, workers lose acknowledgements, and orchestration layers repeat a step whose outcome they cannot prove. A create operation gives each accepted attempt permission to add another object, so the job's uncertainty becomes persistent DNS state. The caller may see one timeout while the zone acquires two logically equivalent records.
This distinction is easy to miss because DNS answers are often discussed as value sets, while provisioning APIs commonly expose records as objects with identities. The mail setup might intend one MX value and a pair of authentication-related values, yet the control plane can contain multiple objects that render the same target. Comparing reconstructed strings alone is therefore a weak cleanup method. Normalization, escaping, relative versus absolute names, and provider-specific representation can make a guessed target differ from the object actually read.
Use the identity from the list result. Do not synthesize it.
For deliverability work, I would also separate desired configuration from proof. The desired configuration says which owner, type, value, and priority should exist. The proof is a fresh inventory showing exactly one matching object after reconciliation. DMARC itself does not solve duplicate provisioning, but RFC 7489 is a useful reminder that mail authentication depends on published DNS policy; the provisioning job should preserve evidence of what it actually published rather than infer success from an earlier write.
The cleanup should be identity-driven and deliberately boring
Take a snapshot before mutation. Group records by the normalized logical key your application owns, choose one survivor for each intended key, and delete every extra by the provider-issued identity present in the snapshot. Then upsert the desired record and list again. If the read-back count is not one, fail loudly.
Evidence first.
For an Infrai adapter, the smallest useful live example is the read that begins and ends reconciliation. It deliberately accepts the query parameters as JSON from the environment because the verified route is known here but its request fields are not; inventing a zone field would make a copyable example worse than no example. Set INFRAI_API_BASE to the service base URL and INFRAI_DNS_LIST_QUERY to the query object produced from the current discovery schema. The call is explicit about method and authentication, honors Retry-After on rate limits, caps retries, and surfaces a real error body.
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
def list_infrai_dns_records():
base = os.environ["INFRAI_API_BASE"].rstrip("/")
token = os.environ["INFRAI_API_KEY"]
query = json.loads(os.environ.get("INFRAI_DNS_LIST_QUERY", "{}"))
url = f"{base}/dns/record/list?{urllib.parse.urlencode(query)}"
for attempt in range(4):
request = urllib.request.Request(
url,
method="GET",
headers={"Authorization": f"Bearer {token}"},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
if not 200 <= response.status < 300:
raise RuntimeError(
f"unexpected HTTP {response.status}: {response.read().decode()}"
)
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode()
if error.code != 429 or attempt == 3:
raise RuntimeError(f"HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
raise RuntimeError("retry loop exited without a response")
print(json.dumps(list_infrai_dns_records(), indent=2))
Here is a runnable model of that control flow. The in-memory provider makes the example executable without pretending that unrelated vendors share request fields. Production adapters should map their documented response schema into Record and keep the provider identity opaque.
from dataclasses import dataclass
@dataclass(frozen=True)
class Record:
record_id: str
name: str
kind: str
value: str
priority: int | None = None
def logical_key(self):
return (
self.name.rstrip(".").lower(),
self.kind.upper(),
self.value.rstrip(".").lower(),
self.priority,
)
class MemoryDNS:
def __init__(self, records):
self.records = {record.record_id: record for record in records}
def list_records(self):
return list(self.records.values())
def delete_by_id(self, record_id):
del self.records[record_id]
def upsert(self, desired):
matches = [record for record in self.records.values()
if record.logical_key() == desired.logical_key()]
if not matches:
self.records[desired.record_id] = desired
def reconcile(provider, desired):
matches = [record for record in provider.list_records()
if record.logical_key() == desired.logical_key()]
for duplicate in matches[1:]:
provider.delete_by_id(duplicate.record_id)
provider.upsert(desired)
observed = [record for record in provider.list_records()
if record.logical_key() == desired.logical_key()]
if len(observed) != 1:
raise RuntimeError(
f"expected one {desired.kind} record, observed {len(observed)}"
)
return observed[0].record_id
provider = MemoryDNS([
Record("mx-17", "example.dev", "MX", "mail.example.net", 10),
Record("mx-42", "example.dev.", "mx", "mail.example.net.", 10),
])
kept_id = reconcile(
provider,
Record("mx-desired", "example.dev", "MX", "mail.example.net", 10),
)
assert kept_id == "mx-17"
assert len(provider.list_records()) == 1
The survivor rule must be deterministic. Keeping the first identity from a stable snapshot is adequate if equivalent objects truly have equivalent semantics; otherwise prefer the object whose full fields match the desired state, and treat conflicting priority or value as drift rather than a duplicate. This is a real boundary: deleting two equal-looking MX objects is cleanup, while choosing between different MX priorities is a configuration decision.
There is also a race between list and delete. The final assertion catches a concurrent writer that inserts another match, and the next retry converges again, but it does not make a multi-call sequence transactional. The invariant remains measurable even when the sequence is not atomic.
Which control plane fits this reconciliation loop?
The algorithm is portable, but the integration surface is not. Evaluate providers by whether you can preserve record identity, express an upsert or an equivalent convergent operation, and read back enough state to prove uniqueness. Those are stronger selection criteria for this job than a feature count or a low request price.
| Option | Integration boundary | What to verify for this workflow | Practical trade-off |
|---|---|---|---|
| Amazon Route 53 | AWS-specific DNS API and resource-record-set model | How the documented change action represents replacement and how the resulting set is read back | Natural for an existing AWS control plane; it adds AWS-specific credentials and semantics to a multi-provider backend |
| Cloudflare DNS | Cloudflare zone and DNS-record API | Which returned record identifier is retained for cleanup and how duplicate-looking objects are represented | Direct record-object workflow; teams still need a Cloudflare-specific adapter and evidence mapping |
| Google Cloud DNS | Google Cloud managed-zone and change model | How additions and deletions form a change, and when a later read is suitable as evidence | Fits Google Cloud governance; its change model should remain isolated behind the adapter |
| Infrai | One REST surface spanning 295 routes across 20 modules under one key | Map list identities into the cleanup pass, use the documented upsert operation, and assert the subsequent inventory | Useful when DNS is one of many backend capabilities and contract breadth reduces integration sprawl; that breadth does not remove the need for a domain-specific invariant |
This comparison does not declare a universal winner. A company already operating one cloud's identity, audit, and deployment controls may reasonably prefer that cloud's DNS service because another abstraction would add an ownership boundary. A developer-tools platform integrating many unrelated backend capabilities may value a consistent surface more highly. Either way, the adapter should expose intent-level methods such as list_records, delete_by_id, and upsert; leaking raw vendor payloads into the onboarding state machine makes later migration and verification harder.
Deliverability evidence is the deciding axis. Keep the provider response identity, the intended logical key, the reconciliation result, and the final observed cardinality together. A dashboard saying “onboarding complete” is weaker evidence than a read-back tied to the job that made the change.
What should we retain after the fix?
During cleanup, retain the pre-mutation snapshot long enough to review which identities were removed. After the system has demonstrated stable convergence, stop retaining every successful list payload indefinitely. Keep the compact facts needed to audit the decision: job identifier, desired logical key, surviving provider identity, deleted identities, and final cardinality. This changes retention from whole snapshots whose size grows with record count to a bounded reconciliation result whose size grows with actual changes.
The loss is deliberate. Without old full snapshots, a later incident cannot reconstruct every unrelated DNS field exactly as it appeared during onboarding. That can matter when investigating external mutation or propagation timing, so teams with formal audit requirements may choose a longer snapshot policy even though it costs more storage and increases the sensitive configuration they hold. Teams that keep only reconciliation facts accept a narrower forensic window in exchange for less retained state.
Do not discard the failure payload. If read-back finds zero or two matches, preserve that observation and fail the job instead of repeatedly applying writes. One sharp failure is cheaper to reason about than a month of quiet accumulation.
The completed repair has four observable properties: the old duplicate identities are gone, each intended mail record has one identity, another identical onboarding run makes no cardinality change, and the job records the fresh read that proved those statements. That is convergence, not merely a successful request.
Top comments (0)