DNS change monitoring: the seven false alarms I had to kill

The first version of any DNS checker is about ten lines. Look up a record, compare it with the value stored last time, send an email if they differ. Point that at a real domain for a day or two and it will email you about records nobody has touched. The checker behind DNS Notify started there too.

DNS change monitoring is easy to describe and annoying to get right, because DNS answers vary in ways that mean nothing. Below are the false alarms I had to design out, and what I did about each. If you want to monitor DNS changes with your own script, this should save you a week.

1. Asking a resolver instead of the source

The obvious way to look up a record is through the system resolver, which is what dig example.com MX does by default. A recursive resolver answers from cache until the TTL runs out. Public resolvers are also big anycast clusters, so two queries a minute apart can land on machines with different cache contents. After a real change you can see old, new, old, new until the TTL has run out everywhere. To a naive checker each flip is a change.

The fix is to skip resolvers and ask the domain's own nameservers. With dig:

dig example.com NS +short
dig @ns1.example-dns.net example.com MX +noall +answer

And with dnspython, which is what the checker uses:

import dns.resolver

def authoritative_answer(domain, name, rtype):
    ns_hosts = [str(r.target) for r in dns.resolver.resolve(domain, "NS")]
    ns_ips = [a.to_text() for h in ns_hosts for a in dns.resolver.resolve(h, "A")]

    r = dns.resolver.Resolver(configure=False)
    r.nameservers = ns_ips
    r.timeout, r.lifetime = 5, 10

    ans = r.resolve(name, rtype, raise_on_no_answer=False)
    return sorted(rd.to_text() for rd in ans.rrset) if ans.rrset else []

If you only want to eyeball one record this way, I put the same logic behind a free authoritative DNS lookup so you don't need a terminal.

2. Answer order

Many nameservers rotate multi-value answers on purpose. Two A records come back as .10, .20 and then as .20, .10. A string comparison calls that a change. Sort the values before comparing. That is what the sorted() in the snippet above is for.

3. TTLs

Store the full answer line, TTL included, and you are in trouble twice. From a resolver the TTL counts down, so every check differs from the last. From an authoritative server the TTL is stable, but people lower it before a migration and raise it afterwards, and that is not something anyone wants an email about. I keep the TTL for display and leave it out of the comparison.

4. The SOA serial

The SOA record carries a serial number that goes up whenever anything in the zone is edited. Some providers bump it on their own schedule too. Include the SOA in naive DNS record monitoring and you get an alert for every edit to every record, including the ones you already alert on. I mask the serial before comparing. If the serial is the only difference, the new SOA is accepted quietly. Changes to the primary nameserver or the timers still come through, and those are the ones that tell you something.

5. Timeouts that look like deletions

A query that times out returns nothing. A record that was deleted also returns nothing. Treat them the same and every network blip becomes "your MX record was removed", which is a horrible email to get at 3am when it is not true.

These need to be separate code paths. NXDOMAIN and an empty answer are real answers from the server. A timeout or SERVFAIL is a failed check and says nothing about the record. I count failures per record and only alert after four failed runs in a row. When the record comes back, the recovery goes on the timeline without a second email.

6. Formatting noise

Mail.Example.NET. and mail.example.net are the same host. A long TXT record is sent as several quoted 255-byte strings, and where the splits fall is up to the server. Pick one canonical form and stick to it. The checker strips trailing dots, rebuilds MX and SRV answers field by field, and joins TXT strings into one value before comparing. SPF and DKIM records are where this bites, because they are the long ones.

7. Changes that undo themselves

Someone fat-fingers a record, notices, and fixes it two minutes later. The first alert is fair. A second alert saying it changed again, back to what you expected, is noise. I keep two values per record, the accepted one and the current one. When current goes back to accepted, the pending flag clears and a "reverted" event is logged with no email.

The same two-value model handles planned changes. After a migration you accept the new value and it becomes the baseline. Without that step the monitor nags forever about a change you made on purpose. I wrote the whole loop up as a guide, how to monitor DNS changes, with the dig commands for each step.

What I could not fix

Some records are supposed to differ depending on who asks. Latency-based routing, geo DNS and most CDNs return different A records from different places, and sometimes from the same place. No amount of normalising makes that a stable signal. I leave those records off the watch list and monitor the CNAME that points at the CDN instead. That one should never move without someone knowing.

What was left

After all of that, the checker got boring, which was the goal. DNS change alerts are only worth having if people still open them in month six. It emails when a value at the source differs from the value someone accepted, and shows both. It waits before reporting failures. It logs reverts and recoveries without emailing them.

This is how DNS change monitoring works in DNS Notify today. The full rules, including how certificates and registration data are compared, are on the methodology page. If you have run into a false alarm I have not listed, tell me in the comments. I would like to break the checker before a customer does.