TL;DR: Treat a DNS cutover as three scheduled, idempotent changes: lower the existing TTL early enough to matter, publish the SPF, DKIM, and DMARC target state during the change window, then restore the original TTL only after verification. For a logistics platform, the decisive design choice is zone ownership: platform-owned zones can use direct automation, while customer-owned zones need an explicit handoff, evidence, deadlines, and a stop condition.
Do not schedule one giant "cut over DNS" job. Give each phase its own due time, expected prior state, retry policy, and audit record. That separation is the difference between a runbook that can be resumed and a script that can only be rerun with crossed fingers.
Why does zone ownership change the runbook?
A platform-owned zone has one control plane. The platform can read the current record and TTL, compare them with an approved plan, apply a change, and verify the resulting state through the same generic DNS interface. This is the straightforward path, but it still needs change fencing: a delayed worker must not overwrite a newer operator decision merely because its scheduled time arrived.
A customer-owned zone has two control planes: the platform knows the mail-authentication target, while the customer controls publication. Pretending that this is direct automation creates a dangerous ambiguity. A successful notification is not a successful DNS change, and a screenshot is not machine-readable state. The runbook should therefore emit a change packet containing the record name, type, intended value, temporary TTL, original TTL, due time, and verification command; it should then wait for observed DNS state rather than marking the phase complete when the packet is sent.
That distinction also shapes the SLO. For platform-owned zones, measure scheduled-phase completion and verified publication. For customer-owned zones, separate platform processing from customer action time, or the metric will hide which control plane consumed the change window. Either way, mail delivery is the outcome; job completion is only an internal signal.
| Decision | Platform-owned zone | Customer-owned zone |
|---|---|---|
| Change executor | Automated DNS adapter | Customer or delegated automation |
| Completion evidence | Observed record and TTL | Observed record and TTL |
| Primary failure mode | Stale job overwrites newer intent | Handoff accepted but never published |
| Rollback authority | Platform operator | Named customer contact |
| On-call cost | Automation and adapter maintenance | Coordination and deadline management |
| Lock-in pressure | Provider-specific write APIs | Portal-specific instructions and manual state |
This is the buy-versus-build boundary I care about. Buying a managed DNS control plane can reduce adapter maintenance; building a thin scheduler preserves the domain rules, audit model, and ownership workflow. Neither choice removes the need to verify public state.
There is a hard limitation: this scheduler is not suitable when the platform cannot observe authoritative state or when nobody has explicit authority to restore the prior records. In that case, choose a manual, two-person change with a named DNS owner and postpone the mail cutover if the evidence never arrives. Direct automation also trades lower coordination time for adapter upkeep and a larger credential boundary; customer execution makes that boundary smaller but transfers timing risk into the handoff. Those are operational costs, even when no invoice names them.
How should a scheduled pre-change TTL lowering step work?
A long-running process that sleeps between phases has poor failure semantics. A deployment, restart, or worker replacement can lose its place, while a blind retry may apply an obsolete change. Persist the plan instead. Each phase should be independently claimable and should carry both the state it expects and the state it intends to create.
State survives restarts.
The example below deliberately leaves provider writes behind an interface. Its sample plan uses a temporary TTL of 300 seconds and restores 3,600 seconds; those are example configuration values, not universal recommendations. Capacity planning starts with inventory: count zones, records per tenant, verification queries, retry amplification, and the number of changes that can land in one dispatch window. Then size worker concurrency against the DNS write boundary you actually operate.
package main
import (
"context"
"errors"
"fmt"
"time"
)
type Phase string
const (
LowerTTL Phase = "lower_ttl"
Publish Phase = "publish_authentication"
Restore Phase = "restore_ttl"
)
type Change struct {
Zone string
Phase Phase
DueAt time.Time
ExpectedTTL uint32
DesiredTTL uint32
Revision uint64
RecordSetHash string
}
type DNS interface {
Observe(ctx context.Context, zone string) (ttl uint32, recordSetHash string, err error)
Apply(ctx context.Context, change Change) error
}
func Execute(ctx context.Context, dns DNS, now time.Time, change Change) error {
if now.Before(change.DueAt) {
return fmt.Errorf("phase %s is not due", change.Phase)
}
ttl, hash, err := dns.Observe(ctx, change.Zone)
if err != nil {
return fmt.Errorf("observe %s: %w", change.Zone, err)
}
if ttl == change.DesiredTTL && hash == change.RecordSetHash {
return nil // An acknowledged retry has nothing left to change.
}
if ttl != change.ExpectedTTL {
return errors.New("observed TTL differs from the approved prior state")
}
if err := dns.Apply(ctx, change); err != nil {
return fmt.Errorf("apply phase %s: %w", change.Phase, err)
}
return nil
}
The Revision belongs in the durable claim and audit record even though the generic adapter above does not prescribe a storage system. Before applying a phase, the worker should confirm that the claimed revision is still current. This blocks an old restore job from undoing a later, approved TTL policy. A practical plan stores the three due times together but claims them separately: the lowering step expects the normal TTL, publication expects the temporary TTL and the old record hash, and restoration expects both the temporary TTL and the verified target hash. If any precondition differs, the worker stops instead of trying to be helpful. That pause may cost a change window, but an unplanned overwrite costs control of the zone.
Keep the mail records in one approved target set. DMARC belongs in that set rather than being treated as an unrelated follow-up: RFC 7489 defines DMARC policy publication in DNS and explains how receivers use authenticated identifiers and published policy. The operational implication is plain. If SPF, DKIM, and DMARC are reviewed separately but released as one mail change, the scheduler must preserve that reviewed bundle and its hash.
Verification is a gate, not a courtesy
After the publish phase, query through the same observation contract used for preconditions and compare the complete approved set. A green worker status is insufficient. Record the observed TTL and record-set hash with a timestamp, then require that evidence before the restore phase becomes eligible.
Short checks catch expensive mistakes. Confirm that the customer zone is the intended zone, that the planned revision is still current, and that the observed authentication set matches the approved set. For DMARC, verify the policy record at the location defined by RFC 7489 and retain aggregate-reporting configuration as part of the reviewed policy. Do not infer delivery health from publication alone; DNS evidence proves publication, while mail telemetry must prove the service outcome.
The alerting model should follow the phase boundaries. Page on a platform-owned change that misses its verification window or conflicts with observed state. For a customer-owned change, notify before the agreed handoff deadline, escalate when the deadline passes, and stop the dependent cutover rather than guessing. This keeps an external dependency from becoming an unexplained internal SLO burn.
No heroics.
Roll back forward, then restore deliberately
Rollback is another approved state transition, not an ad hoc deletion. Preserve the pre-change record set and its hash before lowering the TTL. If authentication verification fails, publish that prior set through the same guarded adapter, verify it, and only then schedule TTL restoration. If the observed state no longer matches either the planned or prior hash, stop: another actor has changed the zone, and automatic rollback could erase valid work.
Customer-owned rollback needs the same rigor in a different envelope. The change packet should include the prior values, the condition that triggers rollback, the person authorized to approve it, and the deadline after which the logistics mail cutover is postponed. The platform can prepare and observe; it cannot manufacture control it does not own.
Restore the normal TTL as its own visible phase after the chosen verification interval, not as cleanup hidden in a defer. Failed cleanup is still failed production work. Track outstanding temporary TTLs as inventory, expose their age, and make the restore job idempotent so an operator can retry it without replaying the mail-record publication.
A credible runbook ends with evidence: current revision, observed record-set hash, observed TTL, phase timestamps, executor identity, and the reason for any stop. That is enough to resume safely during an on-call handoff and enough to distinguish DNS publication from actual mail-delivery health.
References
- RFC 7489, Domain-based Message Authentication, Reporting, and Conformance (DMARC): https://datatracker.ietf.org/doc/html/rfc7489
Top comments (0)