Put an evidence gate in front of every zone teardown, and make that gate bind the tenant, exact zone identifier, observed mail state, and approval to one expiring deletion plan. TL;DR: a pipeline should first quarantine the namespace, then collect deliverability evidence, and only later accept a separately authorized commit; an empty dashboard or a missing DMARC report is never sufficient proof that a tenant subdomain is unused.
That distinction matters for developer tools that automatically give every tenant a subdomain. Provisioning is cheap to repeat. Deletion is asymmetric: the pipeline can remove the DNS namespace that routing, verification, and mail authentication depend on, while its control plane still looks healthy. The useful design question at 3am is not whether the teardown job was green. It is: what page fired before the commit became irreversible?
How can an automated pipeline prevent accidental zone deletion?
DMARC aggregate reports describe authentication results observed by participating receivers. RFC 7489 defines the reporting mechanism, the policy record under the _dmarc label, and alignment between a message's visible From domain and authenticated identifiers. Those reports are valuable evidence that a namespace has been used for mail and that receivers have evaluated it. They do not turn non-observation into proof of non-use.
Silence is ambiguous. A reporting interval may not have closed, a receiver may not send reports, traffic may be rare, or the telemetry path may be unhealthy. Even a long quiet window answers only “what did this evidence collector observe?” It does not answer “can this name be deleted?” Treat positive evidence as a veto and missing evidence as unknown unless an independent lifecycle signal closes the gap.
Stop there.
Consider a tenant called acme whose readable name still resolves to one zone while an earlier offboarding plan carries another zone ID. The tenant reactivated after quarantine, the provisioning path created a replacement namespace, and the old approval remained in a queue. A worker that searches for acme and deletes whatever currently matches can destroy the replacement even though every individual API call succeeds. A bound plan refuses instead: the lifecycle revision changed, the opaque ID differs, and the approval digest no longer describes the live object. Now add a mail collector that missed its expected cutoff. That is a separate refusal, not a reason to relax the identity check. This example has two target identities and several independent predicates on purpose; collapsing them into one boolean such as safe_to_delete makes the eventual incident record nearly useless. The page should name the mismatch before any commit, while an operator can still cancel the queued plan and leave DNS untouched.
This is where dashboards fail the operator. A smooth line compresses collection health, report coverage, and tenant activity into one picture; the deletion decision needs the underlying timestamps and identities. No page, no trust.
Build a deletion plan, not a delete button
Use a state machine with at least active, quarantined, and deletion-approved states. Quarantine must stop new tenant configuration while preserving the zone. During that interval, the pipeline records the tenant lifecycle decision, the immutable provider-side zone identifier, the expected DNS name, mail evidence, collector health, and an approval that was issued for the same plan digest. The final worker receives the digest, not a free-form domain.
The exact quiet interval is an operating-policy choice; RFC 7489 does not prescribe a safe deletion delay. Pick it from the system's reporting cadence and risk tolerance, then keep it configurable and visible in the plan. A low-volume tenant deserves the same refusal behavior as a busy one because weak evidence becomes more dangerous as traffic becomes sparse.
Unknown means no.
| Input | Safe interpretation | Guard action |
|---|---|---|
| Recent aligned or unaligned mail observations | The namespace is still relevant to mail investigation | Refuse and page the owning team |
| DMARC policy record still expected by tenant state | Lifecycle systems disagree | Refuse teardown |
| No reports, collector healthy, quarantine incomplete | Evidence window is unfinished | Wait |
| No reports, collector unhealthy | Activity is unknown | Refuse and page telemetry ownership |
| Window complete, lifecycle closed, identities match, approval matches | Preconditions agree | Permit a single commit attempt |
The approval should cover a canonical plan, including the opaque zone ID. Names are human-readable but can be recreated; an ID mismatch is the guard doing its job. Require the worker to read the target again immediately before commit and compare both fields. If either changed after approval, expire the plan rather than trying to infer intent.
Make the destructive worker boring
The worker below is deliberately small. The surrounding systems decide how evidence is collected and who may approve; this boundary only validates a complete snapshot and returns a permit tied to one target. Durations and thresholds are policy inputs, not magic constants hidden in the deletion client.
package deletion
import (
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"time"
)
type Plan struct {
TenantID string
ZoneID string
ZoneName string
QuarantinedAt time.Time
ObservedThrough time.Time
MailObservations int
CollectorHealthy bool
LifecycleClosed bool
ApprovalDigest string
ApprovalExpiresAt time.Time
}
type Permit struct {
ZoneID string
ZoneName string
Digest string
}
func digest(p Plan) string {
canonical := fmt.Sprintf("%s\n%s\n%s\n%s\n%s\n%d\n%t\n%t",
p.TenantID,
p.ZoneID,
p.ZoneName,
p.QuarantinedAt.UTC().Format(time.RFC3339Nano),
p.ObservedThrough.UTC().Format(time.RFC3339Nano),
p.MailObservations,
p.CollectorHealthy,
p.LifecycleClosed,
)
sum := sha256.Sum256([]byte(canonical))
return hex.EncodeToString(sum[:])
}
func Authorize(now time.Time, quietWindow time.Duration, p Plan) (Permit, error) {
if p.TenantID == "" || p.ZoneID == "" || p.ZoneName == "" {
return Permit{}, errors.New("target identity is incomplete")
}
if !p.LifecycleClosed {
return Permit{}, errors.New("tenant lifecycle is still open")
}
if !p.CollectorHealthy {
return Permit{}, errors.New("mail evidence collector is unhealthy")
}
if p.MailObservations != 0 {
return Permit{}, errors.New("mail evidence vetoes deletion")
}
if p.ObservedThrough.Before(p.QuarantinedAt.Add(quietWindow)) {
return Permit{}, errors.New("quarantine evidence window is incomplete")
}
want := digest(p)
if p.ApprovalDigest != want {
return Permit{}, errors.New("approval does not match this plan")
}
if !now.Before(p.ApprovalExpiresAt) {
return Permit{}, errors.New("approval has expired")
}
return Permit{ZoneID: p.ZoneID, ZoneName: p.ZoneName, Digest: want}, nil
}
There is a subtle trap in that example: MailObservations == 0 is accepted only alongside a healthy collector, a completed quarantine window, and a closed lifecycle. Removing any one of those checks changes “corroborating evidence” into “absence of evidence.” Keep the conditions conjunctive.
The trade-off is explicit: offboarding takes longer, but stale approval cannot silently outrun current evidence.
The actual deletion adapter should also be narrow: fetch by immutable ID, compare the returned name with the permit, issue one delete, and record the provider response against the plan digest. Do not let it search by suffix, enumerate zones and pick a match, or retry with a newly resolved target. Automation that gets “helpful” at this boundary enlarges the blast radius.
What should page when teardown is requested?
Page on a blocked destructive commit when human action is required, not on every ordinary wait. An identity mismatch, recent mail evidence after quarantine, or unhealthy evidence collection at the commit deadline deserves an actionable event containing the tenant ID, zone ID, plan digest, failed predicate, and owner. A quarantine window that is merely still running belongs in workflow status.
The alert must answer four questions without requiring a dashboard tour: what operation was prevented, which exact object was targeted, which predicate failed, and when the approval expires. The pipeline's success counter is secondary. I distrust any view that cannot expose the raw observation cutoff and collector-health decision used by the guard, because a green aggregate can hide stale inputs.
Keep an append-only decision record for both refusal and permission. Store the canonical inputs and outcome, but do not put credentials or message content in that record. The practical trade-off is extra operational state and slower offboarding in exchange for a reviewable answer when someone asks why a namespace disappeared. For a destructive operation, that delay is the feature.
Verify first; roll back while rollback still exists
Test the guard with state transitions, not only happy-path mocks. At minimum, exercise a recreated zone name with a different ID, an approval made before the latest evidence snapshot, a collector outage, mail observed one instant before the cutoff, an expired approval, and two workers attempting the same plan. The expected result for ambiguity is refusal.
Before the commit, rollback means cancelling the plan, reopening lifecycle state, and lifting quarantine while the zone remains intact. After the commit, there is no honest DNS rollback primitive in this design; recovery is a new provisioning operation whose identity and prior contents must come from controlled state. Calling recreation “rollback” masks the loss of continuity and encourages a runbook that has never been tested.
Run a dry-run adapter in continuous delivery that performs every read and comparison but cannot call delete. In production, canary the policy against scheduled teardowns, inspect refusals, then enable commits for a tightly bounded tenant cohort. The decision rule stays plain: positive mail evidence vetoes deletion; missing evidence never overrides unhealthy telemetry or incomplete lifecycle state.
That is the standard worth carrying into an incident review. The deletion happened only when independent lifecycle, identity, observation, health, timing, and approval signals agreed; otherwise, a specific page named the failed predicate while the namespace still existed.
References
- RFC 7489, Domain-based Message Authentication, Reporting, and Conformance (DMARC): https://datatracker.ietf.org/doc/html/rfc7489
Top comments (0)