Short answer: keep DNS as the stable naming boundary, and add a service registry only where endpoint churn is faster than DNS caching and deployment controls can safely absorb. For a logistics company moving mail to a new provider, publish the intended MX set in authoritative DNS, validate it from independent recursive resolvers, and alert on drift between that intent and the observed answer. A registry does not replace public MX publication. It solves a different problem: rapidly changing internal service membership.
The page arrives as mail-routing-drift: the declared MX targets for freight.example differ from the answers seen through two recursive resolvers. The on-call sees expected and observed sets, answer TTLs, authoritative nameservers, and the first mismatch time. That is actionable. A vague email unhealthy page is not, because SMTP delivery, DNS delegation, and the mail provider are separate failure domains.
What should have fired before mail routing drift became a page?
The late signal is a delivery complaint. The earlier signal is control-plane divergence: the desired MX resource-record set no longer equals the published set. RFC 5321 says an SMTP client looks up MX records and uses their preference values; if the DNS lookup returns no MX record but does return an address record, implicit MX handling may apply. That fallback is protocol behavior, not a migration plan. An explicit, verified MX set makes intent inspectable.
Model the comparison as sets of (preference, exchange) tuples, after normalizing case and the trailing root dot. Do not compare answer order. DNS does not make record ordering your deployment contract, while MX preference is meaningful. Also verify that each exchange resolves to an address: RFC 2181 states that an MX target must not be an alias.
The first alert should be a ticket or deployment failure when an authoritative observation disagrees with declared intent. Page only when the mismatch persists through a deliberately chosen window and is visible through more than one recursive path. One sample is weak evidence because caches legitimately retain data for the TTL supplied with the record; RFC 1035 defines TTL as the interval a resource record may be cached before it should be discarded.
Three minutes is not a universal threshold. It is an example evaluation interval. The correct window comes from the zone's TTL, the change process, and the mail-routing SLO; an alert evaluated faster than the relevant cache lifetime mostly measures cache age.
Instrument the intent-to-answer path
Store desired records in the same reviewed change that authorizes the mail cutover. Probe from at least two resolver perspectives and retain the raw tuples, TTL, response code, and observation time. Querying only a workstation's resolver hides whether the discrepancy lives in a local cache, a recursive resolver, or authoritative data.
This Go sketch compares already-collected answers. Collection stays behind an interface, so tests can use fixed data and production can use a standards-aware DNS client without coupling policy to one provider.
package mxdrift
import (
"fmt"
"strings"
)
type MX struct {
Preference uint16
Exchange string
}
func key(mx MX) string {
exchange := strings.ToLower(strings.TrimSuffix(mx.Exchange, "."))
return fmt.Sprintf("%d %s", mx.Preference, exchange)
}
func Drift(want, got []MX) (missing, unexpected []string) {
expected := make(map[string]struct{}, len(want))
observed := make(map[string]struct{}, len(got))
for _, mx := range want { expected[key(mx)] = struct{}{} }
for _, mx := range got { observed[key(mx)] = struct{}{} }
for record := range expected {
if _, ok := observed[record]; !ok { missing = append(missing, record) }
}
for record := range observed {
if _, ok := expected[record]; !ok { unexpected = append(unexpected, record) }
}
return missing, unexpected
}
Expose dns_mx_drift{domain,resolver} as a binary gauge, plus observation age. Keep exchange names out of labels; target names multiply time series during migrations, while an alert can fetch them from structured probe output. For capacity planning, query load is approximately domains multiplied by resolver perspectives divided by interval. With 300 domains, two perspectives, and a five-minute interval, that averages two queries per second before retries. Jitter probes, bound retries, and budget for timeout bursts.
A deployment gate and runtime monitor catch different faults. The gate catches a bad desired change before publication. The monitor catches out-of-band edits, delegation mistakes, partial propagation, and later drift. Record desired and observed state, or the dashboard can say only that DNS changed, not whether the change was authorized.
Should internal endpoints use DNS or a service registry when deployments accelerate?
DNS is a distributed, cached naming system. A service registry maintains a more immediate view of service instances, commonly paired with health or lease semantics. The decision compares endpoint lifetime, acceptable stale-resolution time, client behavior, and the operational system the team is willing to own.
Neither mechanism is universally suitable.
For public mail routing, the answer is fixed: SMTP senders discover the route through DNS MX records. An internal registry may describe a mail-processing worker or a private submission service, but external senders will not consult it. Keep those namespaces and failure budgets separate.
For an internal shipment-rating service deployed a few times per week behind a stable load-balancer address, DNS is usually enough because instance churn is hidden behind that endpoint. For tasks created and removed many times inside one DNS TTL, a registry can reduce stale membership, provided clients actually watch or refresh it and registration is reliable. Lowering TTL shifts query traffic toward resolvers and authoritative servers; it does not guarantee every client refreshes on your schedule.
DNS has a concrete limitation for rapidly replaced instances: a correct cached answer can still name an endpoint that has already left service, and an operator can't revoke that answer from caches already holding it. A registry has the opposite operational trade-off. Its fresher view depends on leases being renewed, unhealthy registrations expiring, clients reacting correctly to watch interruption, and the registry control plane remaining available. Choose DNS with a stable front door when bounded cache staleness fits the SLO; choose a registry for controlled internal clients when direct membership changes faster than that bound, then pay explicitly for lifecycle testing and control-plane ownership. For externally delivered mail, neither preference nor deployment frequency changes the protocol boundary: the MX route still has to be in DNS.
| Decision pressure | DNS boundary | Registry boundary |
|---|---|---|
| Consumers | Broad, including external systems | Controlled internal clients |
| Change pattern | Stable names or a load balancer | Rapid instance membership |
| Staleness control | TTL and client caching | Leases, watches, or polling |
| Failure concern | Resolver and authoritative availability | Registry quorum and registration lifecycle |
| Team cost | Automation and DNS monitoring | New control plane, upgrades, and on-call ownership |
My buy-versus-build threshold would be operational: if DNS plus a stable endpoint meets the stale-resolution objective, a registry creates another stateful dependency without improving the user-visible SLO. If instance lifetime is routinely shorter than tolerable DNS staleness, evaluate managed and self-hosted options against the same requirements: failure behavior, consistency model, lease expiry, multi-region operation, client support, exportability, and on-call load.
Deployment order is part of correctness
An MX migration should expand capacity before removing the old path. Validate that each new exchange is configured to receive mail for the domain. Publish the combined intended set, observe it through authoritative and recursive paths for the relevant cache horizon, then remove old records after acceptance checks and rollback criteria are satisfied. DNSSEC validation, where deployed, belongs in the probe path; RFC 4035 defines how security-aware resolvers authenticate DNS responses.
Stop the rollout on unexpected NXDOMAIN, SERVFAIL, an empty explicit MX set, an alias used as an MX target, or persistent tuple drift. Treat a transient timeout as a probe failure first, not proof that the zone is wrong. The monitor needs match, mismatch, and indeterminate states; collapsing network failure into mismatch produces noisy pages and hides the actual dependency.
Rollback is another DNS change, so cached answers constrain it. Keep the previous mail path able to accept traffic during overlap instead of assuming a reverted record instantly restores the old world. Define the maximum time published routing may differ from reviewed intent, and separately define availability for mail-acceptance endpoints. One alert cannot represent both objectives cleanly.
Keep it boring.
Pages are expensive.
Tune the page around error budget, not impatience
Page when a missing intended MX tuple is observed through two recursive perspectives for longer than the expected cache horizon, while sending a lower-severity notification immediately when a deployment gate detects drift. The exact duration must come from configured TTLs and the organization's mail SLO, not an article.
False positives consume the attention needed for real routing failures. A threshold below cache lifetime pages on expected propagation. Requiring every resolver to disagree can miss a regional fault. Two perspectives are a defensible starting rule only if they are genuinely independent and the coverage model says which users they represent.
Measure the monitor too: probe success rate, observation age, mismatch duration, and alert-to-acknowledgment time. Exercise it with a reviewed canary record that carries no production mail. A detector that has never observed a controlled transition is an untested dependency. The goal is bounded, visible staleness and a page that tells an operator what changed, where it was observed, and which safe action comes next.
Top comments (0)