Article image
DNSSEC Key Rollover: Preventing Total Domain Blackouts During Cryptographic Updates
Picture the call every agency dreads: an enterprise client's website is completely dead across major networks, their email is bouncing, and API webhooks are dropping globally. Your team pings the server — it's up. You run dig against a public resolver like Google (8.8.8.8) or Cloudflare (1.1.1.1), and it comes back with a devastating response:
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 48219
But run dig example.com @8.8.8.8 +cd (Checking Disabled), and you get a clean NOERROR with the correct IP address.
This isn't hypothetical. On May 5, 2026, a routine, scheduled key rollover at DENIC — the registry that runs Germany's .de TLD, one of the largest on the internet — began publishing signatures that didn't match the published keys. Every DNSSEC-validating resolver on the planet, including Cloudflare's 1.1.1.1 and Google Public DNS, was obligated by the DNSSEC specification to reject them. Amazon.de, DHL, Deutsche Bahn's ticketing system, the N26 banking app, eBay, and German government portals all went unreachable for several hours. It's the same failure mode that broke Sweden's .se in 2009, New Zealand's .nz in 2017, and — closer to home for anyone running infrastructure — knocked every torproject.org domain offline for roughly a day in September 2025 when a KSK rotation fired earlier than scheduled.
The domain hadn't expired and the web servers hadn't crashed. The cryptographic chain of trust securing the domain broke during a key rotation, and validating resolvers worldwide treated the domain as tampered with and refused to resolve it.
For agencies handling government, financial, healthcare, or other high-compliance clients, DNSSEC is increasingly a checkbox on security questionnaires and, for U.S. federal agencies and their contractors, a standing compliance obligation under NIST's DNS security guidance. But managing cryptographic key rollovers across dozens of client domains is a massive operational blind spot for most shops. This guide covers the mechanics of DNSSEC, why key rollovers break domains, how to triage an active blackout, and how to keep the rollover schedule itself from becoming the thing that takes a client offline.
- Demystifying DNSSEC: Keys, Signatures, and the Chain of Trust DNSSEC (Domain Name System Security Extensions) adds cryptographic signatures to DNS records to protect against DNS spoofing, cache poisoning, and man-in-the-middle attacks. Rather than simply serving an A record, a DNSSEC-signed zone signs each resource record set (RRset) using asymmetric cryptography, and resolvers validate those signatures against public keys published up through the DNS hierarchy.
┌────────────────────────────────────────────────────────┐
│ TLD Registry Zone (.com) │
│ - DS Record (hash of the child zone's KSK) │
└──────────────────────────┬─────────────────────────────┘
│ validates
▼
┌────────────────────────────────────────────────────────┐
│ Child Zone (example.com) │
│ - KSK (Key Signing Key) → signs the DNSKEY set │
│ - ZSK (Zone Signing Key) → signs the A/AAAA/MX RRsets │
│ - RRSIG records → the active signatures │
└────────────────────────────────────────────────────────┘
The core cryptographic components:
ZSK (Zone Signing Key): Signs the zone's actual data records (A, AAAA, MX, TXT). ZSKs are typically shorter-lived keys, often rotated every 30–90 days.
KSK (Key Signing Key): Signs the DNSKEY set that contains the ZSK. KSKs are rotated far less often — annually or on a multi-year cycle — because rotating one requires coordinating with the parent registry.
DS (Delegation Signer) record: A cryptographic hash of the child zone's KSK, uploaded to the parent registry (.com, .org, a ccTLD, etc.). This is what links the child zone into the global chain of trust.
RRSIG (Resource Record Signature): The actual signature generated by the ZSK or KSK for an RRset, carrying explicit inception and expiration timestamps.
Because a break anywhere in this chain causes validation to fail for everything beneath it, a misconfiguration at the TLD level — as the .de incident showed — can take down millions of domains at once, regardless of who hosts them or which resolver their visitors use.
- Why DNSSEC Key Rollovers Fail A key rollover is the deliberate process of retiring an active key and introducing a new one. Because DNS responses are cached across thousands of resolvers worldwide, you cannot swap keys instantaneously. When a domain "just stops resolving," it almost always traces back to one of these execution errors:
Scenario 1: Unsynchronized DS record update (KSK rollover). If a new KSK is published and the old one removed from the DNSKEY set before the DS record at the registrar is updated to match, validating resolvers fetch the stale DS hash from the registry, fail to match it against the live DNSKEY set, and return SERVFAIL for everything under the domain.
Scenario 2: Caching and TTL violations (ZSK rollover). If a new ZSK starts signing records but the old DNSKEY set is pulled from the authoritative nameservers before its TTL has fully expired across global caches, resolvers still holding the old key data will reject the new RRSIGs as unrecognized.
Scenario 3: Stale RRSIG signatures. Signatures aren't permanent — a typical RRSIG is valid for somewhere between two and four weeks. If a signing daemon (BIND, PowerDNS, Knot DNS) crashes, stalls, or hits a permissions error and stops re-signing the zone, existing signatures simply age out and resolution fails the moment they expire. This is effectively what happened in the .de incident: the post-incident detail that's emerged is that the rollover script generated signatures over the new DNSKEY set using the previous ZSK's private key while publishing the new key as the only one in the set — the signatures and the key material no longer matched, and every validating resolver rejected the zone on the spot.
Scenario 4: Registrar and nameserver migration desync. When moving a client from one DNS provider to another (say, Route 53 to Cloudflare) without first disabling DNSSEC or matching the DS/KSK pair, the new nameservers can end up serving records signed with a key that doesn't match the DS record still live at the registrar.
- Diagnosing an Active Blackout Step 1: Confirm DNSSEC as the root cause.
Validating lookup (normal resolver behavior)
dig example.com @8.8.8.8
Non-validating lookup (+cd = Checking Disabled)
dig example.com @8.8.8.8 +cd
If the first returns SERVFAIL and the second returns NOERROR with the correct records, you're looking at a DNSSEC validation failure, not an outage at the origin.
Step 2: Trace the chain of trust.
delv @8.8.8.8 example.com +vtrace
Or inspect the DS and DNSKEY records directly:
DS record from the parent registry
dig DS example.com @a.gtld-servers.net.
DNSKEY set from the authoritative server
dig DNSKEY example.com @ns1.example.com +dnssec
Compare the key tag computed from the live DNSKEY set against the key tag in the parent's DS record — a mismatch here is the smoking gun.
Step 3: Read the Extended DNS Error (EDE), if your resolver returns one. RFC 8914 defines EDE codes that ride along with SERVFAIL to say why validation failed. Code 6 ("DNSSEC Bogus") points straight at a broken signature; other resolvers may report a less specific code that just says the upstream was unreachable, so don't treat an ambiguous EDE as proof the problem isn't DNSSEC.
Step 4: Emergency mitigation. If the fix isn't immediate:
Disable DNSSEC at the registrar by pulling the DS record. This breaks the chain of trust deliberately — resolvers will treat the zone as unsigned and resume normal resolution once the parent's DS TTL expires. It's the fastest lever an agency actually controls, and it's effectively what large resolver operators do on their own end during a registry-level incident, where it's called a Negative Trust Anchor (RFC 7646): an explicit instruction to skip validation for a specific zone. Cloudflare used this approach on 1.1.1.1 during the .de incident, restoring resolution for its users roughly three hours after the break began, well before DENIC's own fix landed.
Re-publish the old ZSK alongside the new one if you're mid-rollover, so both keys validate while caches catch up.
Two things work in your favor while you fix the real problem: DNS is heavily cached, so not every user hits the break instantly, and modern resolvers increasingly implement "serve stale" (RFC 8767) — continuing to serve expired cached answers rather than an error when the authoritative source is failing validation. That's a large part of why the .de outage, despite lasting hours at the registry level, didn't produce a flat SERVFAIL rate for every user the whole time.
-
Key Rollover Architectures: Pre-Publication vs. Double-Signature
RFC 6781 lays out the operational practices for DNSSEC key management, and RFC 7583 goes further into the precise timing math — how long to wait at each stage based on your zone's TTLs. The two mechanisms agencies actually use are Pre-Publication (for ZSKs) and Double-Signature / Double-DS (for KSKs).PRE-PUBLICATION ZSK ROLLOVER TIMELINE
Time ──►
┌──────────────────┬──────────────────┬──────────────────┬──────────────────┐
│ Stage 1 │ Stage 2 │ Stage 3 │ Stage 4 │
│ Normal state │ Pre-publish key │ Switch signing │ Retire old key │
├──────────────────┼──────────────────┼──────────────────┼──────────────────┤
│ DNSKEY: {ZSK-1} │ DNSKEY: {ZSK-1, │ DNSKEY: {ZSK-1, │ DNSKEY: {ZSK-2} │
│ │ ZSK-2} │ ZSK-2} │ │
│ Signed by ZSK-1 │ Signed by ZSK-1 │ Signed by ZSK-2 │ Signed by ZSK-2 │
└──────────────────┴──────────────────┴──────────────────┴──────────────────┘
Pre-Publication (ideal for ZSK):
Pre-publish the new ZSK (ZSK-2) into the DNSKEY set while continuing to sign everything with ZSK-1.
Wait for the DNSKEY set's TTL to expire across global caches (commonly 24–48 hours).
Switch signing over to ZSK-2. Both keys stay published.
After another TTL cycle, safely remove ZSK-1.
Double-DS / Double-KSK (ideal for KSK):
Publish the new KSK (KSK-2) into the child zone's DNSKEY set.
Generate the DS record for KSK-2 and upload it to the registrar. The parent zone now carries two DS records.
Wait for the parent's DS TTL to propagate fully.
Remove the old DS record at the registrar, then purge KSK-1 from the child zone.
The single most instructive real-world case study for this pattern is the DNS root zone itself. Its first-ever KSK rollover was originally scheduled for October 2017, but ICANN delayed it a full year after data showed a meaningful share of resolvers weren't ready — a textbook example of "be patient" beating "be on schedule." It finally completed on October 11, 2018, with minimal user impact precisely because of that caution. ICANN and Verisign are now partway through only the second root KSK rollover in DNSSEC's history: the new key (KSK-2024) was published in the root zone on January 11, 2025, and under the automated trust-update process in RFC 5011, validating resolvers had a 30-day observation window before they'd trust it automatically. As of ICANN's most recent update, more than 95% of reporting resolvers have already picked it up — but the root zone will start signing with only KSK-2024 on October 11, 2026, about five weeks from today. Any resolver that hasn't picked up key tag 38696 by then loses DNSSEC validation entirely, which for a misconfigured resolver can mean total resolution failure rather than a graceful fallback. It's a useful, concrete deadline to check against your own infrastructure and any client-managed resolvers this quarter.
- How Common Is DNSSEC, Really? It's worth being honest with clients about where DNSSEC adoption actually stands, because it shapes how much operational risk a rollover carries relative to its benefit. The numbers vary a lot depending on what's being measured, and they don't always agree:
The European Commission's Joint Research Centre put the average DNSSEC validation rate (the share of resolvers configured to validate) at 49.4% across the EU and 35.4% globally as of Q3 2025.
An analysis of Cloudflare's public Radar telemetry found that in July 2026, about 8% of DNS queries reached a domain that was DNSSEC-signed, but only roughly 0.6% of all queries were actually validated end-to-end — a large gap between zones being signed and resolvers actually checking the signatures. That said, the 0.6% figure had grown by roughly two-thirds year over year, so the trend is upward even if the absolute number is small.
A separate analysis of TLD zone files covering more than 240 million domains found only about 4.3% carried a DS record indicating DNSSEC was signed at all, as of early 2026.
These are different metrics measuring different things — resolver capability, query-weighted signing, and raw domain counts don't move in lockstep — so treat any single "X% adopted" headline with some skepticism. The consistent thread across all of them is that DNSSEC remains a minority practice globally, which is exactly why so few engineers get hands-on rollover experience before they're doing it for a client's production zone for the first time.
Agency DNS Security Management Matrix
Hosting Tier DNSSEC Automation Level Agency Operational Requirement Risk Severity
Fully Managed Cloud (Cloudflare, Route 53) Fully automated (1-click) Monitor that the registrar and DNS provider stay in sync; audit key changes quarterly Low — automated by the platform
Hybrid (external registrar + custom nameservers) Semi-automated (manual DS upload) The agency must manually upload new DS records during every KSK rollover High — significant human-error potential
Custom on-prem / BIND / PowerDNS Manual or scripted (cron) Audit RRSIG expiration timestamps, verify the signing daemon's health, track rotation cycles High — a crashed daemon silently stops re-signing until someone notices
For agencies with clients on custom or hybrid regulatory hosting (government, healthcare, financial services), it's also worth knowing that NIST published a full rewrite of its federal DNS security guidance (SP 800-81, Revision 3) in March 2026, replacing a document that had stood since 2013. It covers DNSSEC alongside encrypted DNS and Protective DNS, and while it's written for U.S. federal agencies under FISMA, it's increasingly treated as a reference point by cyber insurers and by private-sector clients under their own compliance obligations — worth a look if you're writing DNSSEC into a client SOW.Tracking Renewals Without Relying on One Engineer's Memory
The recurring theme in almost every DNSSEC incident, including the .de outage, is timing: a rollover step happened before or after it was safe to, or a rotation schedule slipped without anyone catching it in time. None of that is really a knowledge problem — the RFCs and the runbooks exist — it's an operational tracking problem. When an agency is running rollovers across dozens of client domains on different cadences, "who's supposed to upload the new DS record, and by when" is exactly the kind of thing that falls through the cracks in a spreadsheet or a Slack thread.
This is where a renewal-date and asset-ownership record like InstaRenewal earns its keep — not as a technical DNS monitor, but as the shared system of record so a rollover schedule doesn't live only in one engineer's calendar. In practice that looks like:
Logging each client domain's KSK and ZSK rotation cadence alongside its other renewal dates (SSL/TLS certs, hosting, licenses), so an upcoming KSK rollover shows up next to everything else that domain needs attention for.
Recording RRSIG expiration windows and DS-upload deadlines as scheduled reminders, with a named owner on the team responsible for each step — so "someone will handle it" becomes "Priya handles the DS upload by Thursday."
Keeping a plain record of DNSSEC status and who manages it per client (which registrar holds the account, which team member has access, when it was last verified), which matters as much during agency offboarding or a client handoff as it does during a routine rollover.
That's deliberately a record-keeping layer, not a live DNS monitor — it won't detect a bad signature the moment DENIC-style breakage happens, and it isn't a substitute for the dig/delv diagnostic workflow above or for actual uptime and DNSSEC-validation monitoring tools. What it's good for is making sure the rollover gets scheduled, assigned, and tracked to completion in the first place, which is where most of these incidents actually start.
- Conclusion DNSSEC provides real protection against DNS spoofing and cache poisoning, but the .de, Tor, .nz, and .se incidents all make the same point: the technology isn't what fails, the timing does. A rollover on even the most professionally run registries can go wrong when a script signs with the wrong key or a step happens before caches have caught up.
The fix isn't more caution on any single rollover — it's a standard operating procedure (Pre-Publication for ZSKs, Double-DS for KSKs, patience over speed at every stage), a diagnostic habit that starts with dig +cd, and a tracking system that makes sure nobody's client rollover is riding on a single person remembering a date. With the second-ever root KSK cutover landing on October 11, 2026, this is a reasonable week to check that every resolver you're responsible for is actually ready for it.
Sources referenced in this article:
Cloudflare Blog, "When DNSSEC goes wrong: how we responded to the .de TLD outage", May 2026
ICANN, "Preparing for the Root Zone KSK Rollover: What You Need to Know", July 2026
ICANN, "The Recent KSK Rollover: Summary and Next Steps", October 2018
Tor Project infrastructure status, DNSSEC outage incident report, September 2025
IETF RFC 6781 (DNSSEC Operational Practices), RFC 7583 (Key Rollover Timing), RFC 7646 (Negative Trust Anchors), RFC 8767 (Serving Stale Data), RFC 8914 (Extended DNS Errors), RFC 5011 (Automated Trust Anchor Updates)
European Commission Joint Research Centre, DNSSEC and DNS resolver adoption report, Q3 2025
NIST SP 800-81 Revision 3, Secure Domain Name System (DNS) Deployment Guide, March 2026
Top comments (0)