It is 2 a.m. Alerts are firing. Something is wrong with DNS, and the person on call, who may not be your DNS expert, has to figure out what is happening and what to do about it, quickly, under pressure, half awake. This is exactly the moment when tribal knowledge fails and improvisation causes second outages. It is also the moment a good runbook earns back every minute spent writing it.
DNS incidents are particularly unforgiving because DNS sits beneath everything else. A mistake made while responding, a wrong record, a hasty change with a long TTL, can extend an outage for hours after the original problem is fixed. Runbooks and on-call playbooks turn a high-pressure judgment call into a checklist, so the responder follows a known-good path instead of guessing.
This guide covers what makes a DNS runbook effective, then provides adaptable playbook structures for the most common DNS incidents. Treat these as starting templates to fill in with your own environment's specifics.
What Makes a DNS Runbook Actually Useful
A runbook that gets used at 2 a.m. has a few properties that a document written to tick a compliance box does not.
It assumes the reader is not the expert. The person on call may know little about DNS. Good runbooks spell out commands, name the systems to check, and avoid assuming context. If a step says "verify the record," it also says how and where.
It leads with triage, not theory. The first section should help the responder confirm what is happening and gauge severity, before any fix. Wrong diagnosis leads to wrong action.
It names people and places. Who owns DNS. Which registrar and DNS provider. Where credentials live. Who to escalate to and how to reach them. Generic advice is useless without these specifics filled in.
It includes rollback. Every action that changes something should say how to undo it. Under pressure, the ability to revert cleanly is worth more than the ability to fix perfectly.
It is tested. A runbook nobody has walked through is a guess. Tabletop the common scenarios so the gaps surface in calm conditions, not during an incident.
A Standard Runbook Structure
A consistent template makes every playbook faster to follow because responders know where to look. A workable structure:
INCIDENT: [name of the scenario]
SEVERITY: [how to judge how bad this is]
1. DETECT & CONFIRM
- What signals indicate this incident
- Commands / tools to confirm it is real
2. ASSESS IMPACT
- What is affected, how widely
- How to gauge scope
3. CONTAIN / MITIGATE
- Immediate steps to stop the bleeding
- What NOT to do
4. RESOLVE
- Steps to fix the root cause
- Rollback for each change
5. VERIFY
- How to confirm recovery
- Accounting for propagation / TTL
6. ESCALATE
- When to escalate and to whom
- Contact details
7. POST-INCIDENT
- What to record, review, and follow upThe playbooks below follow this shape. Fill the bracketed specifics with your own environment.
Playbook 1: Suspected DNS Hijacking or Unauthorized Change
Detect and confirm. Triggered by an alert on a record or nameserver change nobody authorized, or reports of traffic going to the wrong place. Confirm by checking the current records against your known-good baseline and comparing what authoritative nameservers return. Verify whether the change is at the record level or the nameserver level, since that determines everything downstream.
Assess impact. Identify which records changed and what they affect, web, email, or other services. A changed MX or nameserver is higher severity than a single peripheral record.
Contain. If this looks like a registrar account compromise (nameserver change, unexpected transfer notice), treat it as one: begin the registrar account takeover response, secure the account and its associated email, and contact the registrar urgently. Do not simply revert the record and move on if the account itself may be compromised, or the attacker will change it back.
Resolve and verify. Restore records from your baseline. Because propagation and caching mean changes are not instant, account for TTL when judging recovery, and consider that a long TTL on the malicious record may keep serving it to some resolvers for a while. Detecting and reacting to the broader pattern is covered in our guide to detecting DNS hijacking.
Playbook 2: DNSSEC Validation Failure
Detect and confirm. Triggered by reports that a domain is unreachable for some users but fine for others, a classic DNSSEC-failure signature, or by a validation alert. Confirm by checking DNSSEC validation status and whether the chain of trust is intact.
Assess impact. A DNSSEC failure can make a domain unreachable for everyone using validating resolvers, which is a large and growing share. Treat it as high severity by default.
Contain and resolve. Identify the cause, commonly an expired signature, a key rollover that went wrong, or a mismatch between the DS record at the parent and the keys in the zone. The fix depends on the cause: correct the DS record, complete or roll back the key change, or re-sign as needed. This is a place where a wrong move makes things worse, so follow the specific sub-steps for your provider carefully. The mechanics are explained in our DNSSEC guide.
Verify. Confirm validation succeeds from an external validating resolver, not just your own, and account for TTL on the affected records.
Playbook 3: Propagation or Stale-Record Issues
Detect and confirm. Triggered by "the change I made is not showing up" or inconsistent behavior across regions. Confirm by querying multiple resolvers and comparing answers, and by checking the authoritative record against what resolvers are returning.
Assess. Determine whether this is normal propagation delay (expected, driven by TTL) or a genuine problem such as a record set incorrectly or a secondary that has not updated. Most propagation "incidents" are simply TTL playing out and need patience, not action.
Resolve. If it is a real error, correct the record. If it is propagation, the fix is usually to wait out the TTL, and the lesson for next time is to lower TTLs before planned changes, per our TTL best practices. Resist the urge to make more changes, which often compounds confusion.
Playbook 4: Domain or Certificate Expiry Emergency
Detect and confirm. Triggered by a domain that has stopped resolving, or an expiry alert. Confirm the registration status and expiration date.
Contain and resolve. For an expired domain, renew immediately through the registrar; most offer a grace period, but act fast because that window is finite and the alternative is losing the domain. For a related certificate expiry, reissue and deploy the certificate. The broader prevention is tracked expiration monitoring, which we cover in domain expiration as a silent risk.
Post-incident. An expiry emergency is nearly always a monitoring and contact-hygiene failure. The follow-up is to ensure renewal notices reach a monitored inbox and expirations are tracked independently.
Wiring Detection Into the Runbooks
Every playbook above starts with "detect and confirm," and a runbook is only as good as the alert that triggers it. If the first anyone hears of a DNS hijack or a DNSSEC failure is a customer complaint, the runbook starts late. Continuous monitoring is what fires the initial signal that sends the on-call person to the right playbook in the first place.
This is where DNS Assistant fits into your incident process. It watches for the conditions these playbooks respond to, unauthorized record and nameserver changes, DNSSEC problems, approaching expirations, WHOIS changes, and alerts in real time across email, Slack, Microsoft Teams, SMS, and webhooks. Routed into your on-call channel, those alerts become the trigger that starts the right runbook early. DNS Assistant does not execute the runbook or make changes for you; it provides the early, specific detection that makes a calm, on-time response possible instead of a scramble that begins only once damage is visible.
Start Building
Before an incident, it helps to know your current baseline, what your records, DNSSEC status, and registration details look like when everything is healthy. Capture that with the free DNS lookup tool or a Free Domain Risk Report, so your runbooks have a known-good state to compare against.
To get the real-time alerts that trigger these playbooks early, start free at dnsassistant.com.
Top comments (0)