The page says custom domain activation failures above SLO, and the on-call sees 38 customer-support tenants stuck in verifying during a registrar migration. Registrar APIs and a DNS interface expose overlapping controls, but they do different jobs. The tempting response is to retry every DNS write. That's dangerous: some zones belong to the platform, while others belong to customers whose registrars and authoritative DNS services sit outside the platform's control.
TL;DR: a registrar API changes registration state; a DNS interface changes records in an authoritative zone. The surfaces overlap around nameservers and DNSSEC delegation, but they are not interchangeable. During a registrar migration, split the workflow into four boundaries: registration, delegation, authoritative data, and application verification. Automate only the boundaries the platform owns, observe the rest, and make every transition resumable.
For an internal customer-support console, the ownership rule should be explicit. Platform-owned zones may permit managed record changes. Customer-owned zones should produce exact record instructions and verification evidence, never an attempted mutation through credentials the platform does not control. That distinction is more useful than a universal adapter pretending every domain operation is the same.
How are registrar APIs and a DNS interface different jobs?
The alert fired on activation because activation is where the customer noticed the problem. The earlier signal was a stalled control-plane transition: registration transferred, delegation had not converged to the intended authoritative service, or the expected record was absent from the authoritative answer. A single domain_failed counter collapses these states and sends the on-call toward the wrong system.
Work backward from what the application can prove. A hostname being registered does not prove that the intended nameservers are delegated. Correct delegation does not prove that the zone contains the application record. A record visible from one recursive resolver does not prove that every cached answer has expired. Keep those claims separate in state, metrics, and the admin console.
This is also where email records deserve caution. DMARC is published in DNS and defines policy plus reporting for message authentication. Moving a domain without preserving the relevant DNS data can therefore affect more than the web endpoint used by the support console. The migration inventory must include records the application does not directly consume.
The useful page identifies the boundary and the owner: delegation_pending for a customer-owned zone is a guided customer action; authoritative_write_failed for a platform-owned zone is an operator action. Same hostname, different response.
Stop there.
Four control boundaries, one customer-facing workflow
A registrar handles the relationship around a registered domain, including registration lifecycle data and the delegation submitted for the domain. An authoritative DNS service answers from zone data. Some organizations obtain both through one provider and one console, which makes the boundary easy to miss until migration day. Shared screens do not create shared semantics.
| Boundary | Desired proof | Who may change it | Failure treatment |
|---|---|---|---|
| Registration | Domain is in the expected account and lifecycle state | Registrant or authorized registrar operator | Stop the cutover; preserve current service |
| Delegation | Parent points at the intended authoritative nameservers | Domain owner through the registrar surface | Wait and verify; do not rewrite zone data |
| Authoritative data | Required records exist in the intended zone | Zone operator through the DNS surface | Retry idempotently only when platform-owned |
| Application verification | The support hostname resolves to the expected target | Platform verifier observes; owner fixes upstream state | Report the failed boundary and evidence |
The table is deliberately asymmetric. An internal console can observe all four boundaries, but permission to mutate one says nothing about permission to mutate another. For customer-owned zones, verification is a product capability in its own right: show the exact owner-visible value, the last observation, and which boundary remains incomplete.
For platform-owned zones, the control plane can do more, but it still should not hide ordering. Create and verify the destination zone before changing delegation. Retain the old serving path during the planned overlap. Remove old state only after the rollback window closes and the application SLO remains healthy. The precise waiting period belongs in an operational policy derived from the records' TTLs and the migration's risk, not in a hard-coded sleep.
Instrument the transition, not the button click
The instrumentation change is to model each domain as a small state machine and emit observations at boundary crossings. Request counts around an admin-console button tell you that somebody asked for a change. They do not tell you whether the externally visible state matches intent.
The following Go sketch keeps ownership and observation explicit. It does not call a registrar or DNS vendor; adapters can supply observations without changing the decision rule.
package domains
import "time"
type Ownership string
const (
CustomerOwned Ownership = "customer"
PlatformOwned Ownership = "platform"
)
type Observation struct {
Domain string
Ownership Ownership
RegistrationReady bool
DelegationReady bool
AuthoritativeReady bool
ApplicationReady bool
ObservedAt time.Time
}
type Action string
const (
HoldMigration Action = "hold_migration"
RequestOwnerAction Action = "request_owner_action"
ReconcileZone Action = "reconcile_zone"
Activate Action = "activate"
)
func NextAction(o Observation) Action {
if !o.RegistrationReady || !o.DelegationReady {
return HoldMigration
}
if !o.AuthoritativeReady {
if o.Ownership == CustomerOwned {
return RequestOwnerAction
}
return ReconcileZone
}
if !o.ApplicationReady {
return HoldMigration
}
return Activate
}
Emit the state as low-cardinality metrics, and put domain-level evidence in structured logs or traces rather than a metric label. Track time spent in each state, transition attempts, and the age of the last authoritative observation. Alert on exhausted error budgets or a sustained queue of domains beyond the migration objective; a momentary mismatch during an expected transition is evidence, not automatically a page.
Short polling looks responsive but creates noise and load without changing DNS caching behavior. Backoff should be bounded by the user-facing objective, and retries must not turn a read-only customer-owned workflow into an accidental write path.
Migration gates that survive partial failure
Treat the cutover as a journaled operation, because the process can stop after any external change. Each step records its intent, an idempotency key, the observed result, and the safe next action. On restart, reconcile observed state instead of assuming the previous request failed merely because its response was lost.
Before execution, snapshot the registration data relevant to the move, the current delegation, and the complete zone data available to the operator. Validate the destination zone independently. During execution, prevent two workers from advancing the same domain, but do not use a global lock that turns one slow customer into a fleet-wide outage. After execution, verify from more than one observation point if the SLO depends on broad reachability, while remembering that recursive caches can legitimately disagree during TTL expiry.
Rollback also crosses ownership boundaries. Restoring a platform-owned record is an automated control-plane operation. Asking a customer to restore delegation is a coordinated procedure whose elapsed time the platform cannot guarantee. Capacity planning should therefore count manual customer transitions separately from automatic reconciliations; the same queue length implies very different on-call load.
A buy-versus-build decision follows from those operational boundaries.
| Option | On-call load | Lock-in surface | Best fit |
|---|---|---|---|
| Build registrar and DNS adapters | Highest: lifecycle differences and retries remain yours | Adapter contracts and stored state are controlled internally | Stable, narrow provider set with staff to own migrations |
| Use a managed abstraction | Lower integration effort, but incident diagnosis still needs boundary evidence | Provider's domain model and supported operations | Broad provider coverage where reduced integration work outweighs dependency risk |
| Keep customer changes manual, automate verification | Human coordination dominates; mutation risk is lower | Minimal write-path dependency | Customer-owned zones with modest migration volume |
No row wins universally. Estimate peak concurrent migrations, expected manual interventions, verification query volume, and pages per error-budget period before choosing. The cheapest implementation to launch can be the most expensive one to operate if every ambiguous failure wakes a human.
This design has limits, and the trade-off is deliberate. A four-state model is unsuitable when registry-specific lifecycle states determine whether a transfer can proceed; those states need an additional registrar adapter and separate evidence rather than another overloaded boolean. Manual customer changes also stop scaling once their arrival rate exceeds the support team's review capacity, while a managed abstraction can narrow integration work at the cost of depending on its supported operation set. Conversely, building adapters is a poor choice for a team that cannot fund ongoing conformance tests and on-call ownership. The decision should change when those constraints change.
No shortcut fixes ownership.
Set the alert where an operator can act
The final threshold should reflect the state machine and the ownership model. Page immediately for a sustained failure to serve platform-owned authoritative data when it threatens the application SLO. Ticket a customer-owned delegation that remains pending, with escalation tied to the promised activation objective. Record transient resolver disagreement for analysis unless it consumes that objective.
False positives have a direct capacity cost. A threshold that pages on every failed observation trains the team to retry externally cached transitions, interrupts unrelated incident work, and can prompt unsafe writes at the wrong boundary. A threshold that waits only for the final activation SLO hides a growing migration queue. The defensible middle is multi-window alerting on actionable states, backed by a dashboard that exposes registration, delegation, authoritative data, and application verification separately.
Keep the closing rule blunt: observe every boundary, mutate only owned state, and page only when the on-call has a safe action.
Top comments (0)