On 5 May 2026, .de domains were hard to reach for what DENIC puts at approximately three hours. DENIC published its final report on 11 June. It is short, and the cause comes down to one sentence:
This was therefore not a classic key tag collision, but rather three key pairs with different contents but identical metadata.
To see what that looks like from the validator's side, I wrote a check using nothing but the Python standard library. It computes the key tags of the live .de keys, matches the signing key to the DS record in the root, verifies two real signatures by hand, and then reproduces the failure mode with a throwaway key. Everything below was run on 20 October 2026.
What the report says happened
This is DENIC's own assessment of its own outage, so I am staying close to its wording. A rollover agent, an in-house component, was supposed to generate one key pair and load it into every connected HSM. A bug made it generate a separate key pair for each HSM and load each one into exactly one of them. All three carried the same identifiers, including key tag 33834. One of the three public keys was written into the zone as the ZSK, so only signatures made by the HSM holding the matching private key could be validated: "in practice, therefore, only about a third of all signatures."
The test environment had one HSM at one location, so the code path that only misbehaves with several HSMs never ran before go-live. The report also says three continuously running validation tools "detected the errors as intended; however, the generated notifications were not processed correctly." And it rules out a few things explicitly: no compromise or attack, no fault in the Knot name server, no HSM malfunction, and no classic key tag collision.
A key tag is a 16-bit checksum
RFC 4034 Appendix B defines the key tag as the sum of the DNSKEY RDATA taken as 16-bit words, with the carry folded back in once. The same appendix is blunt about what that means: "Implementations MUST NOT assume that the key tag uniquely identifies a DNSKEY RR."
There is no dig on the machine I use for these checks, so the script asks Google's DNS-over-HTTPS JSON API with do=1, which returns the RRSIG records alongside the answers. That is a resolver's view, not the authoritative servers', which is fine for this purpose. The Python blocks in this post, pasted into one file in the order shown, are the whole script. First the parsing helpers and the key tag function:
import base64, hashlib, json, secrets, struct, urllib.request
def doh(name, rrtype):
url = f"https://dns.google/resolve?name={name}&type={rrtype}&do=1"
with urllib.request.urlopen(url, timeout=10) as r:
return json.load(r)
def answers(resp, rrtype):
return [a["data"] for a in resp.get("Answer", []) if a["type"] == rrtype]
def wire_name(name):
out = b""
for label in name.rstrip(".").lower().split("."):
out += bytes([len(label)]) + label.encode()
return out + b"\x00"
def dnskey_rdata(text):
flags, proto, alg, key = text.split(" ", 3)
return struct.pack("!HBB", int(flags), int(proto), int(alg)) + base64.b64decode(key)
def key_tag(rdata):
# RFC 4034 Appendix B, for every algorithm except 1
ac = 0
for i, b in enumerate(rdata):
ac += b if i & 1 else b << 8
ac += (ac >> 16) & 0xFFFF
return ac & 0xFFFF
def rsa_public(rdata):
# RFC 3110: exponent length, exponent, modulus (after flags/proto/alg)
k = rdata[4:]
if k[0] == 0:
elen, k = int.from_bytes(k[1:3], "big"), k[3:]
else:
elen, k = k[0], k[1:]
return int.from_bytes(k[:elen], "big"), int.from_bytes(k[elen:], "big")
rsa_public follows RFC 3110: one length octet for the exponent (or a zero octet plus two length octets if it is longer than 255 bytes), then the exponent, then the modulus.
Matching the signing key to the root
zone = "de."
keys = [dnskey_rdata(t) for t in answers(doh(zone, "DNSKEY"), 48)]
by_tag = {}
for rd in keys:
flags = struct.unpack("!H", rd[:2])[0]
by_tag.setdefault(key_tag(rd), []).append(rd)
print(f"DNSKEY flags={flags} alg={rd[3]} bits={rsa_public(rd)[1].bit_length()} tag={key_tag(rd)}")
for ds in answers(doh(zone, "DS"), 43):
tag, alg, dtype, digest = ds.split()
for rd in by_tag.get(int(tag), []):
ok = hashlib.sha256(wire_name(zone) + rd).hexdigest().upper() == digest.upper()
print(f"DS in root: tag={tag} digest_type={dtype} matches DNSKEY {key_tag(rd)}: {ok}")
The DS digest is SHA-256 over the owner name in wire format followed by the DNSKEY RDATA (RFC 4034 section 5.1.4; digest type 2 is SHA-256). The first three lines of output:
DNSKEY flags=257 alg=8 bits=2048 tag=26755
DNSKEY flags=256 alg=8 bits=1024 tag=32911
DS in root: tag=26755 digest_type=2 matches DNSKEY 26755: True
So today .de publishes a 2048-bit KSK with tag 26755, which is what the root's DS points at, and a 1024-bit ZSK with tag 32911. Tag 33834 from the report is no longer in the DNSKEY set.
Verifying the signatures by hand
Checking a tag against a DS is the easy half. The part that failed on 5 May is the signature check, so the script does that too. For algorithm 8 (RSA/SHA-256), verification means raising the signature to the public exponent and comparing the result with the padded SHA-256 digest, using the DigestInfo prefix from RFC 5702. The signed data is the RRSIG RDATA without the signature field, followed by every record in the RRset in canonical order, each one carrying the original TTL from the RRSIG rather than whatever TTL the resolver returned.
SHA256_PREFIX = bytes.fromhex("3031300d060960864801650304020105000420") # RFC 5702
def rsa_sha256_verify(e, n, data, sig):
k = (n.bit_length() + 7) // 8
em = pow(int.from_bytes(sig, "big"), e, n).to_bytes(k, "big")
digest = SHA256_PREFIX + hashlib.sha256(data).digest()
return em == b"\x00\x01" + b"\xff" * (k - 3 - len(digest)) + b"\x00" + digest
TYPES = {"soa": 6, "dnskey": 48}
def soa_rdata(text):
mname, rname, *nums = text.split()
return wire_name(mname) + wire_name(rname) + struct.pack("!5I", *map(int, nums))
def signed_data(rrsig_text, owner, rdatas):
# RFC 4034 3.1.8.1: RRSIG RDATA without the signature, then the RRset in canonical order
covered, alg, labels, ttl, exp, inc, tag, signer, _ = rrsig_text.split(" ", 8)
head = struct.pack("!HBBIIIH", TYPES[covered], int(alg), int(labels), int(ttl),
int(exp), int(inc), int(tag)) + wire_name(signer)
rrs = b""
for rd in sorted(rdatas):
rrs += wire_name(owner) + struct.pack("!HHIH", TYPES[covered], 1, int(ttl), len(rd)) + rd
return head + rrs, int(tag)
def check(rrtype, rdatas_from):
resp = doh(zone, rrtype)
rdatas = rdatas_from(resp)
for sig in answers(resp, 46):
data, tag = signed_data(sig, zone, rdatas)
raw = base64.b64decode(sig.rsplit(" ", 1)[1])
for rd in by_tag.get(tag, []):
e, n = rsa_public(rd)
print(f"RRSIG over {rrtype} names tag {tag}: verifies = {rsa_sha256_verify(e, n, data, raw)}")
return resp
check("DNSKEY", lambda r: [dnskey_rdata(t) for t in answers(r, 48)])
soa = check("SOA", lambda r: [soa_rdata(t) for t in answers(r, 6)])
The script only knows how to rebuild SOA and DNSKEY records, which is all it needs. Output:
RRSIG over DNSKEY names tag 26755: verifies = True
RRSIG over SOA names tag 32911: verifies = True
Both verify. The KSK (flags 257, Secure Entry Point bit set) signs the DNSKEY set, and the ZSK (flags 256) signs the SOA.
Reproducing "different contents, identical metadata"
Now the failure mode. The Key Tag field in an RRSIG is a number the signer writes. A validator reads it, looks for DNSKEY records in the zone with that owner, algorithm and tag, and tries them. RFC 4035 section 5.3.1 says that when several keys match, the validator "MUST try each matching DNSKEY RR until either the signature is validated or the validator has run out of matching public keys to try."
On 5 May there was exactly one matching DNSKEY to try, and for two of the three HSMs it was the wrong public key. The tag cannot tell a validator that, because it identifies a public key that is in the zone, not the private key that produced the signature.
So the last part of the script generates a second RSA key, signs the real .de SOA RRset with it, and writes the published ZSK's tag into the RRSIG fields:
# The 5 May failure mode: a second private key signs, and the RRSIG carries the published tag.
def probable_prime(bits): # Fermat test: good enough for a throwaway demo key
while True:
p = secrets.randbits(bits) | (1 << bits - 1) | 1
if all(pow(a, p - 1, p) == 1 for a in (2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37)):
return p
e = 65537
while True:
p, q = probable_prime(512), probable_prime(512)
phi = (p - 1) * (q - 1)
if p != q and phi % e:
break
n2, d2 = p * q, pow(e, -1, phi)
k2 = (n2.bit_length() + 7) // 8
other_rdata = struct.pack("!HBB", 256, 3, 8) + b"\x03" + e.to_bytes(3, "big") + n2.to_bytes(k2, "big")
sig_text = answers(soa, 46)[0]
zsk_tag = int(sig_text.split()[6])
data, _ = signed_data(sig_text, zone, [soa_rdata(t) for t in answers(soa, 6)])
digest = SHA256_PREFIX + hashlib.sha256(data).digest()
em = b"\x00\x01" + b"\xff" * (k2 - 3 - len(digest)) + b"\x00" + digest
other_sig = pow(int.from_bytes(em, "big"), d2, n2).to_bytes(k2, "big")
print(f"second key's own tag: {key_tag(other_rdata)}; RRSIG written with tag {zsk_tag}")
print(f" verifies against the second key: {rsa_sha256_verify(e, n2, data, other_sig)}")
for rd in by_tag[zsk_tag]:
pe, pn = rsa_public(rd)
print(f" verifies against published DNSKEY {zsk_tag}: {rsa_sha256_verify(pe, pn, data, other_sig)}")
second key's own tag: 31959; RRSIG written with tag 32911
verifies against the second key: True
verifies against published DNSKEY 32911: False
The second key is random, so its own tag changes from run to run (31959 in the run shown here). The signature is valid under the key that made it, and it carries the published tag, so a validator finds a candidate DNSKEY straight away. It just fails, and with no other key carrying tag 32911 there is nothing left to try. That RRset is bogus.
The report does not explain how three different keys ended up carrying the same tag, and I am not going to guess. The demonstration only shows the validator's side: a matching tag tells you which published key to try, and nothing more.
Why unsigned .de domains broke too
The report also says that domains with no DNSSEC at all failed to resolve. The reason is the proof that a delegation has no DS record. Here is what a validating resolver gets back when it asks for the DS of example.de, which is a real, unsigned delegation:
curl -s "https://dns.google/resolve?name=example.de&type=DS&do=1" | python3 -c '
import json, sys
r = json.load(sys.stdin)
print("Status", r["Status"], "AD", r["AD"], "answers", len(r.get("Answer", [])))
for a in r["Authority"]:
if a["type"] == 50:
print("NSEC3 ", a["name"], a["data"])
elif a["type"] == 46 and a["data"].startswith("nsec3"):
print("RRSIG ", " ".join(a["data"].split()[:8]))
'
Status 0 AD False answers 0
NSEC3 a0d5d1p51kijsevll74k523htmq406bk.de. 1 1 0 - A0D5F6VETKD4HE1UG3NJGF20U4QMCIAQ NS SOA RRSIG DNSKEY NSEC3PARAM
NSEC3 cjkj4kvicg7rjbjst1mqfmeb6q28thd0.de. 1 1 0 - CJKJR41VLNHBFMDL2HTKSS0TBVM1GB73 NS DS RRSIG
RRSIG nsec3 8 2 7200 1793603732 1792388732 32911 de.
RRSIG nsec3 8 2 7200 1793603732 1792388732 32911 de.
No answer, and two NSEC3 records in the authority section: one for the zone apex and one whose range covers the hash of example.de (SHA-1, no salt, zero extra iterations per the NSEC3 parameters: cjkjelihnuqdpp2fevo6dvn8qdtj47ki, which sorts between that record's owner and its next-hashed name). Both have the opt-out flag set (the first field, 1). RFC 5155 section 3.1.2.1 says an NSEC3 with opt-out set "covers zero or more unsigned delegations." The resolver accepts that example.de is unsigned only after validating these records, and both are signed by the ZSK. If those signatures had been among the two thirds made by the wrong HSM, the "no DS here" proof would itself be bogus, and the resolver could not resolve example.de even though example.de never signed anything. The report describes the same mechanism and adds that non-validating resolvers kept serving .de without trouble.
What I took from running it
If you sign your own zone, a key tag that lines up with the DS or with an RRSIG proves very little. The useful post-rollover check verifies the signatures against the published DNSKEY set from outside your signer, the way the script does, on more than one record type. DENIC's own list of measures includes a partial validation of the zone before deployment, which is that check placed before publication rather than after.
The other lesson is in the report's own sentence about monitoring. The three validation tools found the fault. The alerts were not processed in time. DENIC published all of this, including the parts that reflect badly on its own process, and that is why anyone outside can learn from it.
Written with AI assistance from my own notes and test results. I checked every fact and command before publishing.
Top comments (0)