TL;DR: after a DNS record is deleted, treat the current zone as evidence of what remains, not evidence of what belonged there. Reconstruct the last intended record from your own append-only audit events, verify that the tenant and zone ownership still match, then apply one narrowly scoped change. For a marketplace that assigns every tenant a subdomain, platform-owned zones can support automated repair; customer-owned zones should usually stop at a reviewed change proposal because control belongs to the customer.
That ownership split is the first decision, not an implementation detail. A log can tell you that a record existed. It cannot grant authority to recreate it.
How can you recover a deleted DNS record when nobody knows which one?
A useful investigation joins three timelines: the tenant's desired hostname, the DNS mutations attempted by your control plane, and the resulting status recorded by that control plane. Start with the tenant identifier rather than a guessed hostname. In a marketplace, names can be renamed, released, or attached to a different zone; searching only for shop.example.test risks finding the right text in the wrong lifecycle.
The minimum useful event contains a stable event ID, tenant ID, zone ID, record name, type, value, action, timestamp, actor, and request ID. Preserve the submitted TTL too, but do not confuse it with proof that a resolver still has an old answer. The recovery target is the latest successful desired state before the deletion, followed by every later event for the same record key.
Be strict about success. A submitted mutation and an accepted mutation are different states in your application model, so the event that drives recovery should be whatever your system records only after its normal success condition. If your logs lack that distinction, mark the candidate uncertain and require review. Do not manufacture confidence.
Debug the timeline, not the memory of whoever noticed the outage.
Rebuild the candidate before debating the cause
Here is a small, runnable reducer for newline-delimited JSON audit events. The sample is intentionally local: it demonstrates the state calculation without depending on a provider API. It also makes duplicate delivery harmless by deduplicating event IDs, which matters when an audit pipeline retries a write.
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
@dataclass(frozen=True)
class Record:
tenant_id: str
zone_id: str
name: str
record_type: str
value: str
ttl: int
event_id: str
occurred_at: datetime
def record_key(event: dict) -> tuple[str, str, str]:
return (
event["zone_id"],
event["name"].rstrip(".").lower(),
event["record_type"].upper(),
)
def rebuild(path: Path) -> dict[tuple[str, str, str], Record]:
events = []
seen = set()
for line in path.read_text(encoding="utf-8").splitlines():
event = json.loads(line)
if event["event_id"] in seen or event["status"] != "succeeded":
continue
seen.add(event["event_id"])
event["parsed_time"] = datetime.fromisoformat(
event["occurred_at"].replace("Z", "+00:00")
)
events.append(event)
state: dict[tuple[str, str, str], Record] = {}
for event in sorted(
events, key=lambda item: (item["parsed_time"], item["event_id"])
):
key = record_key(event)
if event["action"] == "DELETE":
state.pop(key, None)
elif event["action"] in {"CREATE", "UPSERT"}:
state[key] = Record(
tenant_id=event["tenant_id"],
zone_id=event["zone_id"],
name=event["name"],
record_type=event["record_type"],
value=event["value"],
ttl=int(event["ttl"]),
event_id=event["event_id"],
occurred_at=event["parsed_time"],
)
return state
def prior_candidate(path: Path, deleted_event_id: str) -> Record | None:
raw = [
json.loads(line)
for line in path.read_text(encoding="utf-8").splitlines()
]
deletion = next(
event for event in raw if event["event_id"] == deleted_event_id
)
cutoff = datetime.fromisoformat(
deletion["occurred_at"].replace("Z", "+00:00")
)
target = record_key(deletion)
eligible = [
event
for event in raw
if record_key(event) == target
and event["status"] == "succeeded"
and event["action"] in {"CREATE", "UPSERT"}
and datetime.fromisoformat(
event["occurred_at"].replace("Z", "+00:00")
) < cutoff
]
if not eligible:
return None
event = max(
eligible,
key=lambda item: (item["occurred_at"], item["event_id"]),
)
return Record(
tenant_id=event["tenant_id"],
zone_id=event["zone_id"],
name=event["name"],
record_type=event["record_type"],
value=event["value"],
ttl=int(event["ttl"]),
event_id=event["event_id"],
occurred_at=datetime.fromisoformat(
event["occurred_at"].replace("Z", "+00:00")
),
)
This reducer answers a narrow question: what value did the application last record before the successful delete? It does not decide whether that value is safe now. Keep those jobs separate. The script also orders equal timestamps by event ID, but that is only deterministic, not causal; a production event schema should carry a per-record revision or another ordering field issued by the writer.
One limit is absolute: this method cannot recover a value that was never captured. If retention dropped the create event, redaction removed the value, or a customer changed its own zone outside the platform, the reducer must refuse to guess. That trade-off favors an auditable recovery candidate over a fast but potentially wrong write. It is not suitable as a substitute for a desired-state store or an authoritative zone snapshot.
Run the reconstruction in an eval harness before attaching any write path. Feed it shuffled events, duplicated events, failed mutations, two tenants with similar names, and a create-delete-create sequence. The expected output should be a structured candidate or an explicit refusal reason. This is the same habit that keeps an AI workflow honest: test the state transition, not the fluency of its explanation.
Which zone owns the repair?
The platform-owned case is the cleaner branch. The marketplace controls the zone and can compare the candidate with its current tenant registry, reserved-name rules, and record revision. Even then, automation should require an exact zone ID and record key; a broad replay of historical events can resurrect intentionally removed tenants.
A customer-owned zone changes the contract. Your audit trail may show the value your platform requested, while the customer may have edited the record independently afterward. Produce a proposed name, type, value, and supporting request IDs for review. Do not represent that proposal as the authoritative zone state, and do not attempt a write unless the customer's current authorization and the application's ownership policy permit it.
| Check | Platform-owned zone | Customer-owned zone |
|---|---|---|
| Authority source | Current marketplace zone registry | Current customer authorization |
| Default output | Guarded repair candidate | Reviewed change proposal |
| Primary conflict | Tenant lifecycle changed | Customer made an out-of-band edit |
| Required scope | Exact zone and record key | Exact delegated or authorized scope |
This distinction becomes especially important for mail-related records. DMARC is published as a DNS TXT record at a defined location and expresses a domain owner's requested message-handling policy. Restoring an old TXT value without checking current ownership can therefore restore an old policy, not merely a routing hint. RFC 7489 is the source of truth for the DMARC mechanism; the audit log remains evidence only of what your application previously wrote.
Why not copy an answer from a resolver cache?
A cached answer is useful corroboration, but it is a poor desired-state database. It may be absent, stale relative to a later intentional edit, or detached from the tenant lifecycle that authorized the name. Likewise, a screenshot proves what somebody observed, not which mutation won.
Logs have their own traps. Redaction may remove a TXT value. Retention may begin after the original create. Two workers may record events in an order that does not match the writer's revision order. A delete event may identify only a name while the name held multiple record types. Each gap should lower the recovery confidence and push the operation toward manual review.
No evidence, no write.
This is also where prompt cost enters the design. An AI assistant can summarize a long event trail for an operator, but the deterministic reducer should select the candidate first. Send the small evidence packet to the model: record key, relevant revisions, ownership classification, actor, request IDs, and refusal reasons. Do not spend tokens asking a model to infer state that ordinary code can compute and an eval can verify.
Make the next deletion boring
Finish the repair as a compare-and-apply operation. Re-read the tenant registry and ownership immediately before the change, compare the current record revision with the revision observed during investigation, and abort on a mismatch. Apply only the selected record. Then record a new audit event linked to the deletion and recovery request IDs, and verify through the same application path used for ordinary provisioning.
Operationally, the durable checklist belongs in prose because every item is connected. Keep audit events append-only and assign stable IDs. Store success status and a writer-issued revision beside the full normalized record data. Set retention from the recovery objective rather than from dashboard convenience. Alert on deletion volume and on deletions of protected names, but route alerts by zone ownership so customer-owned records do not trigger unauthorized automation. Finally, keep a fixture corpus for the reducer and run it in CI; six carefully chosen transition cases are more useful than a notebook that once returned the expected hostname.
The key move is modest: recover intent before recovering data. When the evidence is complete and the platform owns the zone, a narrow repair can be routine. When ownership or ordering is uncertain, the correct result is a reviewable proposal. Stop there.
Top comments (0)