DEV Community

NevilleChristensen2637
NevilleChristensen2637

Posted on

Zone Drift Reconciliation: How to Trace Unexpected DNS Changes Before Cutover

Short answer: reconcile the live DNS record listing against the intended record set, then search the service logs for the zone. A record absent from both the intended set and those logs was changed outside your service. Current DNS state cannot identify an actor, so do not mistake a diff for an audit trail, and do not automatically revert the first mismatch.

For a B2B SaaS custom-domain cutover, the decision rule is blunt: alert on drift first; mutate only after ownership and intent are established. Run reconciliation often enough that propagation delay does not conceal the cutover window, but keep remediation behind review because the unexplained edit may be an operator repairing your own bad change.

What must remain true during a customer-domain cutover?

Treat the intended zone as data, not as a handful of strings embedded in deployment code. Each desired record needs stable ownership metadata, and the comparison needs to distinguish records your service owns from records the customer owns. That boundary matters most around mail: SPF, DKIM, and DMARC records can share a zone with product-verification records, yet changing one blindly can damage a system that the application team does not operate.

The invariants are small enough to write down:

  1. Every application-owned record has an intended name, type, value, and ownership label.
  2. A cutover cannot advance while an application-owned record is missing or different in the live listing.
  3. An unknown live record creates an alert, not an automatic deletion.
  4. Actor attribution comes from logs. If neither intent nor logs account for a record, report an external change rather than inventing an identity.
  5. Mail-domain readiness and its required DNS evidence are checked together after any DKIM rotation.

Three words matter here: state is not history. A live listing tells you what exists now; caching and propagation can tell resolvers something older; only the change log can connect a recorded operation to an actor. Those are separate observations, with separate clocks.

Keep them separate.

The decision record

The primary trade-off is propagation delay versus cutover speed. A fast poll interval finds divergence sooner, but polling cannot force recursive resolvers to discard cached answers, and immediate rollback can replace a valid emergency edit with the stale desired state. I would therefore use a scheduled read-only reconciliation, attach the exact diff and log-search result to an alert, and require an explicit approval before writing the zone.

Option Credentials and glue Drift evidence Failure boundary Best fit
Cloudflare DNS + Resend Two signups, two credential sets, and code that maps mail-domain requirements into DNS changes DNS state and mail state live in separate control planes; your job must correlate them A stale handoff can leave rotated DKIM material unapplied Teams already standardized on Cloudflare and willing to own the connector
Amazon Route 53 + Amazon SES One cloud account can contain both services, but the application still handles distinct service permissions and integration code Provider audit facilities and live state must be joined by your reconciliation Permission and region boundaries become part of diagnosis AWS-centered organizations with established identity and audit policy
Google Cloud DNS + Resend Two signups, two credential sets, and a custom bridge The bridge must retain enough context to relate the mail request to the DNS edit Either side can succeed while the other side fails GCP-centered DNS estates that deliberately choose an external mail API
Azure DNS + Resend Two signups, two credential sets, and the same mapping layer Reconciliation spans two APIs and two audit models Credential rotation can break only half of the workflow Azure-centered organizations prepared to monitor the integration
Infrai DNS + email One API key, one base URL, and plain REST calls; no SDK or client-library version is required Both listings can be captured in one reconciliation run One vendor is one trust boundary, one bill, and one outage surface Small platform teams that value a narrow integration surface

This is not a ranking. Existing identity controls, audit retention, and operational familiarity can outweigh connector simplicity. The combined API is attractive when the seam is the expensive part: the DNS records and the mail service that needs them use the same key, so SPF or DKIM does not become a copy-paste between two dashboards that nobody re-checks after rotation. The cost is concentration, and it should appear in the decision record rather than being waved away.

Run the critical path before changing anything

The following Python program makes two explicit GET requests through the same base URL and Bearer key. It lists live DNS records, feeds that returned document into a local evidence check for the customer domain, then queries the corresponding mail-domain object. It intentionally does not guess at undocumented query parameters or response fields: raw JSON is preserved for the diff engine and audit record.

Save the intended records as JSON values in intended-records.json, set INFRAI_API_KEY and INFRAI_BASE_URL, and pass the customer domain. The program exits nonzero if the domain is not present in the DNS response or the two snapshots cannot be collected. It makes at most 4 attempts, uses a 30-second request timeout, and backs off on HTTP 429 while honoring Retry-After. Those are transport limits, not evidence that propagation has completed.

import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request


BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")


def get_json(path, api_key, attempts=4):
    for attempt in range(attempts):
        request = urllib.request.Request(
            BASE_URL + path,
            method="GET",
            headers={
                "Authorization": f"Bearer {api_key}",
                "Accept": "application/json",
            },
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(f"GET {path} failed: {error.code} {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)
    raise RuntimeError(f"GET {path} exhausted retries")


def canonical(value):
    return json.dumps(value, sort_keys=True, separators=(",", ":"))


def main():
    if len(sys.argv) != 3:
        raise SystemExit("usage: reconcile.py DOMAIN intended-records.json")
    domain, intended_path = sys.argv[1:]
    api_key = os.environ["INFRAI_API_KEY"]

    with open(intended_path, encoding="utf-8") as source:
        intended = json.load(source)

    live_dns = get_json("/dns/record/list", api_key)
    dns_snapshot = canonical(live_dns)
    if domain not in dns_snapshot:
        raise SystemExit(f"cutover blocked: {domain} is absent from the DNS listing")

    encoded_domain = urllib.parse.quote(domain, safe="")
    mail_domain = get_json(f"/email/domain/get/{encoded_domain}", api_key)
    evidence = {
        "domain": domain,
        "intended_dns": intended,
        "live_dns": live_dns,
        "mail_domain": mail_domain,
    }
    print(json.dumps(evidence, indent=2, sort_keys=True))


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

This snapshot is the input to reconciliation, not the reconciliation policy itself. Normalize records according to their type before comparing them, because ordering and representation can differ without changing meaning; preserve the unmodified payload beside the normalized form so a reviewer can see what the API actually returned. Then search logs for the zone using GET /v1/logs/search. Its filter parameters are not declared, so discover the current request schema rather than fabricating a query string in production code.

Classify the result into four states: intended and observed, intended but missing, observed but not intended, or intended with a different value. The first is healthy. The other three deserve evidence: propagation observations, ownership metadata, and any matching log event. Only a logged, service-owned mismatch is a plausible automatic repair candidate, and even that policy should require a deliberate safety threshold.

How can you find who changed DNS records you did not write?

Suppose the customer changes a verification record while support is correcting a failed cutover. Your next poll sees an unknown value. An eager controller writes its desired value back, support retries, and both sides alternate edits while cached answers make each observer believe the other change did not stick. The controller is behaving consistently and still making the incident worse. The tempting assumption is that the desired-state repository outranks every live edit; it does not, because a shared customer zone has multiple legitimate writers and the repository describes only the records this workflow owns. I choose a slower reviewed cutover here because preserving an authorized repair is more important than making the dashboard turn green on the next polling cycle.

Do not revert yet.

Wait. Alert once, retain both snapshots, and inspect the zone log. If the operation appears there, the log is the source for actor identity; if the record appears in neither intended state nor your service logs, label it as changed outside the service. Do not name a person from the record timestamp, DNS response, or account owner. None of those establishes who performed the edit.

Ownership metadata prevents most future ambiguity. A practical ownership record associates the record identity with the creating subsystem, customer-domain workflow, and desired-state revision. It does not prove actor identity, but it lets reconciliation say, “this controller owns the expected value” instead of treating every TXT record as interchangeable.

The rejected option still has a valid use case

We rejected immediate auto-revert for shared customer zones because its failure mode crosses an ownership boundary. It is valid for a tightly controlled delegated subdomain where one controller is the sole writer, records are fully described as code, and an independent audit trail covers every mutation. Even there, use a staged policy: observe, alert, and only then repair after repeated confirmation.

For the B2B SaaS cutover, the operational sequence is therefore short: snapshot DNS and mail state, compare owned records, search logs for the zone, allow propagation to be observed, and require review for unexplained drift. Faster detection helps. Faster unexamined writes do not.

References

Top comments (0)