A tenant subdomain cannot move faster than the cached answer already held by resolvers. For a gaming platform that assigns one subdomain to every tenant, the useful control is therefore not a hurried record update; it is a scheduled, durable workflow that lowers the TTL before the migration window, waits for the old cache horizon, changes the target, verifies the authoritative result, and restores the normal TTL only after the new target is stable.
Short answer: model pre-change TTL lowering, DNS cutover, and TTL restoration as separate idempotent states, each with its own deadline, evidence, and retry policy.
That distinction matters because a DNS write succeeding is not the same event as every recursive resolver forgetting the previous answer. It also prevents an awkward operational ambiguity: if a worker restarts after applying a change but before recording success, the next run must be able to prove what happened and continue without creating a second, conflicting transition.
This is a control-plane problem.
How should a scheduled pre-change TTL step govern DNS cutover and restore?
Start with the desired cutover time and work backward. The lowering step needs a not_before time, the cutover needs a separate not_before time, and the interval between them must be at least the previous TTL plus whatever safety margin the operator has chosen. The margin is a policy input, not a universal constant; I'm not sure any fixed value can be defended without resolver observations from the actual tenant population.
For example, consider tenant arcade-17, served at arcade-17.play.example. The illustrative plan below begins with a normal TTL of 3,600 seconds, temporarily uses 300 seconds, and changes an address from 192.0.2.10 to 192.0.2.44. Those addresses are documentation values, and the timings are example policy values rather than measured recommendations. The workflow has four checkpoints: lower, wait, switch, restore. A worker may visit a checkpoint repeatedly, but only one transition may be committed for the same plan version.
The subtle case is a late plan. Suppose the lowering job was scheduled for 01:00 UTC and the cutover for 02:00 UTC, but the scheduler did not begin until 01:47. It must not compress the cache-drain interval and pretend the original cutover is still valid. It should apply the lower TTL if the plan is still authorized, compute a new earliest cutover from the observed application time, and mark the original target time as superseded in the audit record. Fast execution is less important than an explainable sequence. Don't trade correctness for a clock on a dashboard.
The restore is deliberately asymmetric. Lowering the TTL is preparation; restoring it is a commitment that the new destination should remain. A restore job should therefore require both an authoritative observation of the intended record and a stabilization decision from the application owner. If either is absent, the low TTL can remain temporarily, subject to an explicit expiry and an alert. It should never remain low merely because nobody encoded the final state.
Treat the plan as a small ledger
A scheduled job table is insufficient if it stores only run_at and done. The control plane needs an immutable plan identity, a monotonically increasing version, the expected record before and after each mutation, timestamps for eligibility and observation, and a result digest returned by the DNS adapter. That is the minimum information needed to distinguish a safe retry from an operator editing a plan that is already in flight.
The state machine can stay compact:
| State | Required evidence | Next state |
|---|---|---|
planned |
approved plan version | ttl_lowered |
ttl_lowered |
observed temporary TTL and recorded cache horizon | cutover_ready |
cutover_ready |
old horizon elapsed | target_changed |
target_changed |
intended authoritative answer and application approval | ttl_restored |
ttl_restored |
intended normal TTL observed | complete |
Each command should carry an idempotency key such as planID:version:state. The DNS adapter first reads the record and compares it with the expected precondition. If the desired value is already present, it returns an already_applied result with evidence; if the current value matches neither the expected old value nor the desired value, it returns a conflict for human review. Blind overwrites erase the very evidence needed during reconciliation.
Exactly-once execution is not a realistic promise across a scheduler, a database, and an authoritative DNS service.
Retries are normal.
Exactly-once effects are the better target — repeated delivery is acceptable when preconditions, idempotency keys, and durable transition records make the mutation converge on one intended record set. The audit entry should be written in the same database transaction that advances local state, while the remote observation is stored as evidence attached to that transition. This is familiar territory for payment systems because the accounting question is the same: what did we intend, what did the external system accept, and what can we prove afterward?
Keep email policy records outside this generic tenant-host workflow. DMARC policies are published in DNS and evaluated by mail receivers under RFC 7489, so a tenant application cutover must not casually include _dmarc or other organizational mail records in its mutation set. DNS automation does not by itself establish domain authorization, regulatory approval, or change-control compliance — those remain separate controls.
A Go worker with explicit preconditions
The useful abstraction is a narrow DNS interface rather than provider-shaped calls spread throughout business logic. The following Go example omits storage and transport details, but the transition rules are complete enough to show where scheduling, idempotency, and evidence belong. Apply is expected to perform a compare-and-set operation against the record view supplied in Expected; Observe provides the authoritative view used for reconciliation.
package cutover
import (
"context"
"errors"
"fmt"
"time"
)
type State string
const (
Planned State = "planned"
TTLLowered State = "ttl_lowered"
CutoverReady State = "cutover_ready"
TargetChanged State = "target_changed"
TTLRestored State = "ttl_restored"
Complete State = "complete"
)
type Record struct {
Name string
Value string
TTL time.Duration
Serial string
}
type Plan struct {
ID string
Version int
State State
Original Record
Destination Record
TemporaryTTL time.Duration
LowerNotBefore time.Time
MoveNotBefore time.Time
RestoreAfter time.Time
ApprovedStable bool
}
type Evidence struct {
Observed Record
At time.Time
Result string
}
type DNS interface {
Apply(ctx context.Context, key string, expected, desired Record) (Evidence, error)
Observe(ctx context.Context, name string) (Evidence, error)
}
var ErrNotDue = errors.New("transition is not due")
var ErrNeedsApproval = errors.New("restore requires stability approval")
func Step(ctx context.Context, dns DNS, now time.Time, p Plan) (State, Evidence, error) {
key := func(target State) string {
return fmt.Sprintf("%s:%d:%s", p.ID, p.Version, target)
}
switch p.State {
case Planned:
if now.Before(p.LowerNotBefore) {
return p.State, Evidence{}, ErrNotDue
}
desired := p.Original
desired.TTL = p.TemporaryTTL
ev, err := dns.Apply(ctx, key(TTLLowered), p.Original, desired)
return TTLLowered, ev, err
case TTLLowered:
if now.Before(p.MoveNotBefore) {
return p.State, Evidence{}, ErrNotDue
}
ev, err := dns.Observe(ctx, p.Original.Name)
return CutoverReady, ev, err
case CutoverReady:
expected := p.Original
expected.TTL = p.TemporaryTTL
desired := p.Destination
desired.TTL = p.TemporaryTTL
ev, err := dns.Apply(ctx, key(TargetChanged), expected, desired)
return TargetChanged, ev, err
case TargetChanged:
if now.Before(p.RestoreAfter) {
return p.State, Evidence{}, ErrNotDue
}
if !p.ApprovedStable {
return p.State, Evidence{}, ErrNeedsApproval
}
expected := p.Destination
expected.TTL = p.TemporaryTTL
ev, err := dns.Apply(ctx, key(TTLRestored), expected, p.Destination)
return TTLRestored, ev, err
case TTLRestored:
ev, err := dns.Observe(ctx, p.Destination.Name)
return Complete, ev, err
default:
return p.State, Evidence{}, fmt.Errorf("unsupported state %q", p.State)
}
}
The caller must persist the returned state only when err is nil and must retain the evidence with the transition. In production, it should also lease a plan version before calling the adapter, place bounded retries around transient transport outcomes, and route precondition conflicts to reconciliation rather than retrying them as though they were timeouts. A response lost after a successful mutation is precisely why the next attempt begins with observation and comparison.
Notice what the worker does not infer: it does not decide that elapsed wall time proves propagation, and it does not treat an application health check as proof of an authoritative DNS value. Those are different evidence streams. Combining them into one boolean makes post-cutover investigation needlessly vague.
Roll out by proving recovery, not by maximizing speed
Before enabling automatic tenant migrations, run the state machine against a delegated test zone and inject a worker restart after every external mutation but before every local commit. Replaying the same idempotency key should produce the same intended record, while a manually altered record should stop with a conflict. Also test a delayed lowering step, a cancelled plan, a newer plan version arriving during a lease, and a restore approval that never arrives. These tests exercise the boundaries where a scheduler demo usually looks convincing and an operational system usually fails.
Use a small tenant cohort first. For every transition, retain the plan version, idempotency key, expected value, desired value, authoritative observation, actor, approval, and timestamps. Alert on plans that remain in a nonterminal state beyond their policy deadline, but don't advance them merely to clear the alert. Reconciliation should classify the difference before any new write: already applied, still pending, superseded, or conflicting.
This method is not suitable when the team cannot automate the authoritative zone, cannot obtain tenant authorization, or cannot observe the record after mutation. In those cases, stick with a manually approved change window or terminate tenant traffic behind a stable reverse-proxy address, accepting that application routing rather than DNS becomes the cutover control. A low TTL also cannot guarantee a simultaneous global switch; if the requirement is atomic session movement, preserve compatibility at both destinations during the overlap instead of assigning that job to DNS.
The final rollout criterion is intentionally dull: a plan can be interrupted at any instruction, resumed without an ambiguous write, and explained later from durable evidence. Cutover speed still matters for a game tenant awaiting its subdomain, but a five-minute target is meaningless if the platform cannot show why one resolver or one application session followed the old destination. Schedule the cache horizon, preserve overlap, and restore the normal TTL only after the destination has earned it.
References
- RFC 7489, Domain-based Message Authentication, Reporting, and Conformance (DMARC): https://datatracker.ietf.org/doc/html/rfc7489
Top comments (0)