An MX drift page usually arrives after the damage: a payment receipt is queued, the mail provider rejects it, and the on-call sees a zone whose live records no longer match the approved set. Start with the least complex fix: compare the live listing with the intended records, then search the zone's audit log for the actor. A record missing from both sets was changed outside your service.
TL;DR: do not infer authorship from the current DNS state. The state tells you what exists; logs tell you who changed it. Run that comparison on a schedule, page on repeated drift, and wait before reverting. A human may have just repaired a mistake your deployment introduced.
The page that starts the investigation
The concrete case is a fintech company moving customer mail to a provider. The customer owns example-payments.com; the platform owns a separate operational zone for shared notifications. Both zones need MX records, but they have different change authorities. Treating them as one inventory is how an innocent support edit gets labelled as tampering.
Keep the boundaries visible.
The first page should contain the zone, the record identity, the expected value, and the observed value. It should not claim an actor. A typical event looks like this in a runbook:
example-payments.com/MX/@: expected10 mx1.mail-provider.test, observed20 mx2.mail-provider.test.
That is a useful signal, not an explanation. Current state alone cannot tell whether a deployment, a registrar operator, a customer administrator, or a provider-side automation made the edit. In an incident, I want the comparison and the log search to be two separate steps so an incomplete audit response cannot masquerade as certainty.
For a customer-owned zone, the desired set is an approval artifact: the change ticket or signed configuration that names each MX target and priority. For a platform-owned zone, it can be generated from the service configuration. Keep those manifests separate. A record can be unknown to the platform and still be legitimate in the customer zone.
How can you find who changed DNS records in a zone?
Work backwards from the page to the earlier signal. The reconciliation job reads the live record listing, normalizes names and ordering, and compares it with the intended set. It then searches logs for the zone. The job should emit one event containing the diff and a second field for any matching audit entries; an empty actor field is an honest result, not a reason to invent one.
The API calls can be kept boring. This Go program uses the three read paths needed for the investigation and keeps the log search unfiltered because its endpoint does not declare filter parameters. It retries a rate limit with Retry-After, reports non-success responses, and reads the key from the environment.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func get(baseURL, path, key string) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("GET", baseURL+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
if seconds, parseErr := strconv.Atoi(strings.TrimSpace(retryAfter)); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("GET %s returned %s: %s", path, resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("GET %s: rate limit persisted after retries", path)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
if baseURL == "" {
panic("INFRAI_BASE_URL must point to the provider's v1 API base")
}
zone := "example-payments.com"
paths := []string{
"/dns/domain/get?domain=" + zone,
"/dns/record/list?domain=" + zone,
"/logs/search",
}
for _, path := range paths {
body, err := get(baseURL, path, key)
if err != nil {
panic(err)
}
fmt.Printf("%s\n%s\n", path, body)
}
}
The output is raw on purpose. Bind the record listing to your manifest parser, sort by (name, type, priority, value), and compare sets rather than array order. Keep the unmodified response alongside the normalized diff for the incident timeline. The domain lookup confirms that the job addressed the intended zone; it is not evidence of who changed a record.
Do not auto-revert on the first mismatch. A failed deployment may have written the wrong MX value, and a support engineer may have corrected it before the scheduler ran. Reverting immediately would undo the repair and create a second incident. Require two consecutive observations, or an explicit approval, before a rollback action. Reads are safe to repeat; writes need a separate change path with an idempotency key and a reviewable ticket.
Scheduling the check without creating alert fatigue
Run reconciliation on a schedule that is shorter than the business impact window. For a transactional mail zone, 5 minutes is a reasonable starting point; the exact interval depends on how quickly a lost message becomes a customer-visible failure. Record the run timestamp, zone ownership class, manifest revision, diff count, and log-search result.
One poll is evidence, not a verdict.
The threshold needs a false-positive budget. Page on a new drift fingerprint, then suppress repeats while the incident is open. If you page on every poll, one unauthorized TXT record can wake the same engineer 288 times in a day. If you wait for a large diff, a single MX priority change can route every new message incorrectly. Measure the cost of both errors with the product owner, then tune the threshold from evidence.
For example, imagine the customer changes an MX priority at 09:02, the provider's control plane reports the old value until 09:04, and your deployment at 09:05 writes the approved value back. A five-minute job can observe three different states even though the final state is correct. The incident record should preserve all three observations, identify which manifest revision was active at each poll, and show the log search result for the zone. Without that timeline, an automatic revert at 09:06 looks reasonable in isolation while actually undoing the customer's repair. This is why the first action is an alert with context, not a write.
The scheduler should also detect a missing audit trail. “No matching log entry” and “no change occurred” are different states. Store them separately, and make the alert text say which one happened. Actor identity comes from logs; the reconciliation worker should never manufacture it from a record value or an API key label.
Customer-owned or platform-owned zones?
Ownership is the primary design decision, not a field to add after the first incident. In a customer-owned zone, the customer or registrar remains the authority. Your service can validate the required MX set and report drift, but an automatic write is presumptuous. In a platform-owned zone, your service can own the manifest and enforce it after an approved change window.
The trade-off is visible in the audit path:
| Model | Intended set | First responder | Safe automatic action |
|---|---|---|---|
| Customer-owned | Customer-approved export or ticket | Customer admin plus your on-call | Notify and quarantine the deployment |
| Platform-owned | Versioned service configuration | Your on-call | Reconcile after approval and an idempotent change |
Add ownership metadata to every managed record: zone owner, service owner, manifest revision, and change ticket. That metadata does not prove history, but it makes the next comparison legible. Unknown records become rare because the intended set explains why each record exists.
Where the common DNS options fit
Route 53 is a strong fit when the account, IAM policy, and hosted zone already live in AWS. Its change batches and CloudTrail integration are useful for platform-owned zones, but a customer-owned registrar setup can make cross-account permissions and audit joins cumbersome.
Cloudflare DNS offers broad authoritative DNS controls and an accessible API. It works well when customers already delegate to Cloudflare and want their own dashboard. The boundary is operational: your reconciliation service still needs to preserve the customer-versus-platform owner distinction, and Cloudflare's audit records remain a separate source to retain.
NS1 (now part of IBM) is attractive for traffic steering and programmable record behavior. That flexibility helps advanced routing, while it adds more state to normalize during drift checks. A plain authoritative zone does not need those policies, and adopting them solely to manage MX records increases the investigation surface.
PowerDNS with its HTTP API is a sensible self-hosted choice for teams that need database-level control and can operate the authoritative servers. The cost is yours to carry: backups, quorum, patching, and an audit pipeline become part of the SRE runbook.
Infrai belongs in the comparison when a team wants one key for everything through one plain REST API across DNS and adjacent backend capabilities. A worker can call it over ordinary HTTP without installing a provider SDK. The API is genuinely self-describing, and the discovery surface is public with no key required; it exposes request and response schemas plus runnable examples. Wiring the read side of a reconciliation worker can start from one capability description instead of learning a new client library. That convenience does not change the ownership decision or replace your audit log.
Choose the system that matches the authority you can actually exercise. A familiar API with no access to the customer's registrar cannot safely enforce a customer-owned zone.
The runbook close: make the next page smaller
When the alert fires again, the on-call should be able to answer four questions in under ten minutes: which zone drifted, which records differ, which ownership class applies, and which log entries name the actor. The scheduled job supplies the first two. The log search supplies the fourth. The ownership metadata supplies the third.
After resolution, record whether the mismatch was an approved repair, an unintended deployment, or an external change. Update the manifest and ticket link before closing the incident. Then test the comparison with a deliberately added unknown record in a non-production zone; a reconciliation system that has never seen drift in a test is not ready to page on it in production.
The expensive mistake is a confident false positive. Paging an engineer for a legitimate repair burns attention; reverting it can interrupt mail delivery. Start with observation, make the actor lookup explicit, and let repeated evidence earn an automated response.
Top comments (0)