TL;DR: Treat the apex A record and the www CNAME as one desired-state transaction, but never pretend DNS has an atomic commit. Write both, assign them one operation ID, observe both through the same resolver set, and declare the gaming tenant ready only after every required answer matches for a full stability window. If either answer diverges, keep traffic on the previous routing state and retry reconciliation.
The page arrives as tenant-domain-cutover SLO burn: a newly onboarded studio says www.guild.example reaches the game portal, while guild.example does not. The on-call view shows a green API write and a red readiness check. That combination matters. A successful control-plane request is evidence that an instruction was accepted; it is not evidence that two independently cached names have converged.
The practical answer is a small converger around a paired desired state. It gives both records the same generation, makes repeated writes harmless, and separates publish success from cutover readiness. Propagation delay then affects when the gate opens, not whether half of the tenant hostname pair is exposed as ready.
No half-ready tenants.
How should we publish an apex record and WWW CNAME together?
The late signal watched the wrong boundary. It measured the publishing call, while the user-visible contract contained two names and multiple observations over time. The writer could finish before recursive resolvers stopped returning an older answer, and the two names could become observable at different moments. From the player or studio operator's perspective, the pair is still one feature.
Model the operation as a generation rather than two unrelated jobs. For tenant guild-042, generation 1847 might describe an apex A value of 192.0.2.44 and a www CNAME target of edge-042.example.net.. The generation number is application metadata; it does not go into the DNS answer. It lets workers, logs, and checks agree about which intent they are discussing.
A reconciler compares normalized desired and observed record sets before writing. On a retry, it may discover that the apex is already correct and only repair the alias. On another retry, both may already be correct, so it records progress without generating needless updates. Do not turn a timeout into a blind rollback: a timed-out request may require observation before the next decision.
Use four states: pending, observing, ready, and degraded. Only ready can change routing, and only after both answers remain correct for the configured stability window. The DNS updates remain separate operations, but the application exposes one readiness decision.
That's the unit.
Build the convergence loop around evidence
This Go sketch keeps the provider behind a generic interface. Ensure must be idempotent for its key, while Lookup represents the resolver observations selected by the platform team.
The state machine does not depend on the orchestration runtime. A Node.js worker can persist the same generation and invoke the same two interface operations; the example stays in Go so the concurrency, deadline, and cancellation boundaries are visible in one compact block. The important behavior is that both records converge together before promotion, rather than which SDK performs the writes. Keep the operation ledger outside an individual worker process, claim work with bounded leases, and make the generation key survive restarts. If a worker disappears after publishing the A record, its successor observes that result, publishes or verifies the CNAME, and resumes the stability window. If the older worker returns late, the generation check prevents it from promoting stale intent. This is also why two independent queue messages are a weak transaction boundary: each can succeed while the application loses the knowledge that the pair belongs to one cutover.
package cutover
import (
"context"
"errors"
"fmt"
"time"
)
type Record struct {
Name, Type, Value string
}
type DNS interface {
Ensure(context.Context, string, []Record) error
Lookup(context.Context, string, string) ([]string, error)
}
type Pair struct {
TenantID string
Generation uint64
Apex, WWW Record
}
type Policy struct {
PollEvery, StableFor, Deadline time.Duration
RequiredChecks int
}
func Converge(ctx context.Context, dns DNS, pair Pair, p Policy) error {
if pair.Apex.Type != "A" || pair.WWW.Type != "CNAME" {
return errors.New("expected an apex A and www CNAME pair")
}
key := fmt.Sprintf("%s:%d", pair.TenantID, pair.Generation)
if err := dns.Ensure(ctx, key, []Record{pair.Apex, pair.WWW}); err != nil {
return fmt.Errorf("publish generation %s: %w", key, err)
}
deadline := time.NewTimer(p.Deadline)
ticker := time.NewTicker(p.PollEvery)
defer deadline.Stop()
defer ticker.Stop()
var stableSince time.Time
checks := 0
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-deadline.C:
return fmt.Errorf("generation %s did not converge before deadline", key)
case now := <-ticker.C:
apexOK := exact(ctx, dns, pair.Apex)
wwwOK := exact(ctx, dns, pair.WWW)
if !apexOK || !wwwOK {
stableSince, checks = time.Time{}, 0
continue
}
if stableSince.IsZero() { stableSince = now }
checks++
if checks >= p.RequiredChecks && now.Sub(stableSince) >= p.StableFor {
return nil
}
}
}
}
func exact(ctx context.Context, dns DNS, want Record) bool {
got, err := dns.Lookup(ctx, want.Name, want.Type)
return err == nil && len(got) == 1 && got[0] == want.Value
}
A real adapter must normalize names, address representations, and answer ordering according to its interface contract. If multiple A values are valid, compare sets. Keep normalization in tests because a trailing dot or reordered answer should not manufacture an incident.
There is a capacity question hiding in the loop. With 20,000 tenant cutovers, four resolver vantage points, two names, and a five-second interval, a naive design attempts 32,000 lookups per second. That is workload arithmetic, not a benchmark. Queue generations, add bounded concurrency and jitter, and budget observation load before selecting the interval. Faster polling does not make caches expire faster.
Measure first.
Instrument the signal that should fire first
The useful early warning is the age of the oldest non-ready generation, partitioned by state and failure class. Pair it with a histogram from generation creation to readiness, a counter for divergent pairs, and structured logs containing tenant ID, generation, record name, expected value, observed value, and observation point. Do not put arbitrary tenant-controlled hostnames into an unbounded metric label.
One SLO can describe the user contract: the proportion of valid domain generations that become ready before the platform's declared cutover objective. The objective must come from measured resolver behavior and the published TTL policy; there is no defensible universal number here. A second internal SLO can cover reconciliation availability, but it cannot substitute for external observation.
Fire an alert on sustained burn of the readiness objective or when the oldest generation threatens it, not on every slow sample. The dashboard should move from symptom to cause: readiness latency, split pairs, writer errors, lookup errors, and queue depth. A control-plane success counter is context, not the user outcome.
Test one awkward sequence explicitly: the apex reaches the desired value, the alias still returns its previous target, and then the apex temporarily returns its previous value at one vantage point. A single successful sample opens the gate too early. Resetting the stability window on any mismatch turns that sequence into delayed readiness rather than an oscillating cutover.
Choose the ownership boundary, then test it
The buy-versus-build decision is about on-call surface and control, not a feature checklist.
| Operating model | Team owns | Cutover trade-off | Lock-in pressure |
|---|---|---|---|
| Managed DNS API | Intent ledger, retries, observation, readiness | Less authoritative operation; API semantics remain a boundary | Adapter and change model |
| Self-operated authority | Serving path plus converger | Maximum control; highest capacity and paging burden | Data model and tooling |
| Delegated tenant zone | Delegation lifecycle and child-zone policy | Smaller tenant blast radius; another boundary converges | Zone layout and workflow |
Test the contract with a fake adapter that exposes intermediate states deterministically. Then use an isolated authoritative zone for partial acceptance, duplicate requests, timeout after acceptance, stale answers, conflicting generations, canceled contexts, and a worker restart between writes. The invariant is stronger than “both calls eventually ran”: only the newest fully observed generation may become ready.
Deploy the observer in shadow mode, recording what it would mark ready without changing routing. Compare those decisions with the current process, investigate disagreements, and then enable a small tenant cohort. A rollback disables promotion of new generations; it does not erase DNS state that may already be visible.
The threshold can create its own incident
A stability window that is too short produces false readiness. One favorable observation promotes a pair that another resolver still sees differently. A window that is too long produces false pages and holds otherwise healthy tenants behind the gate.
Choose the threshold from a latency distribution gathered across the actual resolver set, and review it when TTL policy or traffic geography changes. Track how many alerts led to operator action. If most pages resolve without intervention while the generation remains inside its objective, the threshold is consuming on-call capacity without protecting the SLO.
The durable design is modest: one intent record, two DNS records, idempotent reconciliation, external observation, and one guarded readiness transition. It does not promise atomic DNS. It gives the platform an auditable cutover decision despite propagation, which is the mechanism a multi-tenant gaming service needs.
Top comments (0)