A tenant DNS record was deleted, nobody knows its old value, and linden.example.net no longer reaches the property portal. Recover it from change logs only when they prove the exact owner name, type, and previous record set; otherwise, stop and escalate rather than guessing from a neighboring tenant.
TL;DR: Treat a DNS deletion as a state-reconstruction problem. Correlate the alert time with an immutable control-plane audit event, recover the previous record set from retained change data, check that no later authorized change superseded it, and submit restoration through the normal reviewed path. If the logs contain only “delete succeeded” without the old value, they prove who changed state but do not prove what to restore.
How can we recover a deleted DNS record nobody knows?
The first page should identify the affected tenant hostname, queried type, observation point, and first failed check. “DNS is down” is too broad. For a property platform that automatically gives every tenant a subdomain, an actionable page looks more like this hypothetical signal: tenant=linden, name=linden.example.net, type=CNAME, expected=present, observed=absent, first_failed_at=14:07Z. It should link the operator to the relevant audit search, not to a generic DNS dashboard.
The on-call now has two clocks to reconcile. The data-plane clock says when resolvers stopped returning the expected answer. The control-plane clock says when a person or automation changed the desired record set. Those timestamps will not necessarily match, so the investigation window needs room on both sides; a fixed five-minute equality test is brittle. Search by canonical owner name and type first, then narrow by zone, tenant identifier, actor, request identifier, and action. The recovery gate is intentionally strict: a delete event must match the affected name and type, and some retained evidence must show the complete previous record set. A ticket saying “it used to work” does not clear that gate. Neither does a log that discarded values for privacy or volume reasons. This approach is limited to mutations visible inside the retained audit boundary; if an administrator changed a customer-owned zone through an unobserved control plane, the platform log cannot establish the missing value, and the right next step is to request the customer's authoritative history or restore from a separately verified zone snapshot. That trade-off is inconvenient during an outage, but fabricating certainty is worse.
No proof, no write.
Logs are evidence, not truth.
Work backward from absence to the last known state
Start with the failed synthetic check and preserve its raw result. Confirm that the question is scoped correctly: an absent A answer does not establish that a CNAME, TXT, or a delegated child zone was deleted. Then inspect control-plane events newest-first. The important sequence is not merely the first deletion you find; it is the latest ordered history for that record key.
Suppose the audit trail shows an approved create, a later delete, and then another approved create. Replaying the old value would overwrite newer intent. Conversely, if the deletion is the terminal mutation and the event carries a trustworthy before image, that image is a restoration candidate. Use a zone snapshot as a second source when available, but compare its capture time with the event sequence before trusting it.
This small Go program demonstrates the reconstruction rule over newline-delimited JSON audit events. The input is deliberately generic. In production, the event producer should authenticate and durably retain the fields on which this decision depends.
package main
import (
"bufio"
"encoding/json"
"fmt"
"os"
"sort"
"time"
)
type RecordSet struct {
Name string `json:"name"`
Type string `json:"type"`
TTL uint32 `json:"ttl"`
Values []string `json:"values"`
}
type Event struct {
At time.Time `json:"at"`
Action string `json:"action"`
Zone string `json:"zone"`
TenantID string `json:"tenant_id"`
Actor string `json:"actor"`
Request string `json:"request_id"`
Before *RecordSet `json:"before"`
After *RecordSet `json:"after"`
}
func keyMatches(r *RecordSet, name, rrtype string) bool {
return r != nil && r.Name == name && r.Type == rrtype
}
func main() {
const targetName = "linden.example.net"
const targetType = "CNAME"
var events []Event
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
var event Event
if err := json.Unmarshal(scanner.Bytes(), &event); err != nil {
fmt.Fprintf(os.Stderr, "invalid event: %v\n", err)
os.Exit(1)
}
if keyMatches(event.Before, targetName, targetType) ||
keyMatches(event.After, targetName, targetType) {
events = append(events, event)
}
}
if err := scanner.Err(); err != nil {
fmt.Fprintf(os.Stderr, "read audit log: %v\n", err)
os.Exit(1)
}
sort.Slice(events, func(i, j int) bool { return events[i].At.Before(events[j].At) })
if len(events) == 0 {
fmt.Println("manual review: no matching history")
return
}
last := events[len(events)-1]
if last.Action != "delete" || !keyMatches(last.Before, targetName, targetType) {
fmt.Println("manual review: terminal event is not a recoverable delete")
return
}
b, err := json.MarshalIndent(last.Before, "", " ")
if err != nil {
fmt.Fprintf(os.Stderr, "encode candidate: %v\n", err)
os.Exit(1)
}
fmt.Printf("candidate from request %s by %s:\n%s\n", last.Request, last.Actor, b)
}
The program emits a candidate, not a mutation. That distinction matters. Before applying it, verify the event's zone and tenant mapping, confirm the actor and request are authentic, look for changes arriving after the exported log window, and compare the candidate with the application's current desired state. Restoration should generate a fresh, reviewable change with a new request identifier; editing history would destroy the evidence needed for the incident review.
Instrument the mutation path, not just the resolver
The earlier signal should have come from the change path. Every create, replace, and delete needs an audit event that can answer six questions later: when, who, which zone, which owner name and type, what state existed before, and what state existed after. Add the tenant identifier and request identifier because a property platform may create many similar subdomains concurrently. Normalize names and types at ingestion so capitalization or a trailing dot does not split one history into several apparent keys.
Retention deserves capacity planning rather than a vague promise to “keep logs.” Estimate daily mutation count, average serialized event size, replication overhead, and required investigation horizon. A simple planning model is mutations/day × bytes/event × retained days × copies. Measure the event-size distribution instead of using one friendly sample; complete TXT values and multi-value record sets can be much larger than a delete marker.
Keep the before image. Without it, the logging system answers attribution but cannot support deterministic restoration. Sensitive values may require tighter access control and separate retention policy, yet silently dropping them changes the recovery guarantee and should be reflected in the runbook and SLO.
For platform-owned zones, the team controls this event contract and can reject unlogged mutations. Customer-owned zones move the boundary. The platform may know what it requested but cannot assume it sees changes made directly by the customer's administrators, registrar, or other automation.
| Decision area | Platform-owned zone | Customer-owned zone |
|---|---|---|
| Authoritative change history | Platform can require one audited mutation path | Customer evidence and access determine completeness |
| Automatic restoration | Possible only after conflict and policy checks | Usually needs customer confirmation and delegated authority |
| On-call scope | Record, automation, and zone are one operational boundary | Diagnosis crosses organizational and control-plane boundaries |
| Lock-in concern | Audit schema should remain exportable and replayable | Integration should tolerate different event formats and retention |
| Build versus buy test | Compare storage, integrity controls, review flow, and on-call load | Compare portability, customer setup burden, and evidence gaps |
This is where the roadmap decision gets uncomfortable. Building an event ledger buys schema control and portability, but the platform team then owns durability, access review, retention, indexing, and restore tooling. A managed audit system can reduce that operational surface, while export format, retention guarantees, and the ability to retrieve before images remain contract questions. I would choose the smallest design that can meet the stated recovery SLO with the people actually available for on-call, then test export and reconstruction before trusting it. The correct choice follows the recovery SLO and staffing model, not a feature-count spreadsheet.
Restore narrowly and test the result
A restoration plan should contain one record-set key, the candidate value and TTL, its evidence event, the current observed state, an approver, and a rollback condition. Apply it through the same policy checks used for ordinary changes. Then test from more than one relevant observation point and check the application path for the affected tenant; a DNS answer alone does not establish that routing and certificates are correct. The limitation is sharp: log reconstruction is unsuitable when event integrity cannot be established, when retention ended before the suspected mutation, or when a later writer may not yet appear in the exported window. In those cases, freeze automated writes, compare independent snapshots and authoritative state, and require manual review.
Email-related TXT records need extra restraint. DMARC policy is published in DNS, and RFC 7489 defines the policy record location beneath _dmarc. Restoring a remembered TXT string at the tenant's web hostname would be the wrong operation even if an email alert occurred at the same time. Match the exact owner name, type, and complete value from evidence.
Close the incident only after desired state, authoritative state, and external observation agree, or after the remaining disagreement is explicitly accepted. The post-incident action should improve the first missing link in the chain: detection, event completeness, retention, correlation, approval, or validation. “Be more careful” has no measurable owner.
The false-positive budget is part of the design
Paging on every delete would catch the relevant event early and exhaust the on-call team quickly, especially where tenant offboarding legitimately removes records in batches. Paging only after a tenant complains preserves attention but misses the recovery window the system was meant to protect. The useful alert combines a deletion with expected-state evidence: an active tenant mapping, a protected record class, an unexpired lease, or a failed synthetic check.
Set thresholds from observed legitimate mutation patterns and review them like any other SLO signal. Track pages that required action, benign pages, missed deletions found through support, and time from mutation to verified repair. A threshold that produces frequent unactionable pages is not conservative; it is training operators to distrust the channel.
There is no universal number. A platform with ten planned offboarding changes each week has a different noise profile from one performing thousands of automated tenant transitions, and inventing a shared cutoff would conceal the capacity question. Start with a narrow protected set, record the outcomes, and expand only when the team can explain the added page volume and response obligation.
The final rule is plain: recover the smallest state the evidence proves, and let uncertainty stop automation. That yields slower restoration in the poorly logged case, but it prevents an incident response from becoming a second unauthorized DNS change.
Further reading
- RFC 7489, Domain-based Message Authentication, Reporting, and Conformance (DMARC): https://datatracker.ietf.org/doc/html/rfc7489
Top comments (0)