Treat mail-domain provisioning as reconciliation, not a sequence of successful API calls: store one normalized intent, read the published RRsets, and keep retrying idempotent upserts until observation matches intent. The decisive state is what authoritative DNS publishes, not what the provisioning job last attempted. For an edtech company moving staff mail, that distinction keeps a harmless retry from becoming a second change while also exposing the dangerous case: the workflow says “done,” but the MX records still point somewhere else.
TL;DR: use three explicit states—desired, observed, and verified—and give each reconciliation run the same input. Replace an entire owned RRset rather than appending individual answers; preserve unrelated records; compare normalized values; and do not mark the domain ready until an independent DNS read agrees. A retry should narrow drift or do nothing. It should never create a new interpretation of the request.
How should an idempotent domain provisioning retry handle an upsert?
The bounded incident scenario is ordinary: a school changes the company mail route, the control plane times out after submitting the MX update, and the job retries. There are two plausible realities behind the same timeout. The first write may have failed before reaching the DNS authority, or it may have succeeded while the response was lost. A workflow built around “run step two after step one returned 200” cannot distinguish them.
I would not accept a green provisioning dashboard as evidence here.
What page fired?
The useful alert is sustained disagreement between declared mail intent and independently observed authoritative answers, scoped to domains that are expected to be active. An alert on every timeout mostly reports transport noise; an alert on one failed attempt pages before the retry policy has had a chance to do its job.
This framing also prevents a subtler error. Mail policy is not one scalar value. An MX RRset can contain multiple exchanges with preferences, while DMARC is published as a DNS TXT record at _dmarc and expresses receiver policy for the organizational domain. Those records have different owners and meanings, so a reconciler should compare typed RRsets rather than concatenate “DNS settings” into one mutable bag. RFC 7489 also defines DMARC alignment in terms of authenticated identifiers, which is why publishing a TXT record is not by itself proof that the intended mail path is correctly authenticated.
One attempt failed. That is not yet an incident.
Model intent, observation, and proof separately
The desired state should be small enough to review and stable enough to replay. For this case, it contains the domain, the complete MX RRset the company intends to own, and the DMARC TXT value approved through the normal policy process. The observed state comes from DNS reads.
Verification expires.
The verified state records that observation matched a particular revision of intent; it must become stale when that revision changes.
| State | Source | Question it answers | Safe retry behavior |
|---|---|---|---|
| Desired | Versioned configuration | What should exist? | Reuse the same revision |
| Observed | Authoritative DNS lookup | What is published now? | Read again after delay |
| Verified | Comparison result | Did this revision converge? | Recompute, never assume |
Do not use a request timestamp as part of identity. If the same logical request receives a new key or a newly generated record set on every attempt, it is not idempotent; it is a series of mutations that happen to look related. A useful operation identity is derived from the domain plus an immutable intent revision, while correctness still comes from comparing state. The key deduplicates work. It does not prove DNS convergence.
Normalization deserves explicit code because superficial differences can manufacture drift: DNS names may be presented with or without a trailing dot, MX answer order is not intent, and preference is part of each MX value. The comparison below deliberately owns only MX. TXT parsing and DMARC policy validation should be separate typed functions rather than clever extensions to this one.
package dnsstate
import (
"cmp"
"slices"
"strings"
)
type MX struct {
Preference uint16
Exchange string
}
func normalizeName(s string) string {
return strings.ToLower(strings.TrimSuffix(strings.TrimSpace(s), ".")) + "."
}
func normalizeMX(in []MX) []MX {
out := make([]MX, len(in))
copy(out, in)
for i := range out {
out[i].Exchange = normalizeName(out[i].Exchange)
}
slices.SortFunc(out, func(a, b MX) int {
if n := cmp.Compare(a.Preference, b.Preference); n != 0 {
return n
}
return cmp.Compare(a.Exchange, b.Exchange)
})
return slices.Compact(out)
}
func sameMX(want, got []MX) bool {
return slices.Equal(normalizeMX(want), normalizeMX(got))
}
Make the write path converge
The preventative path is a level-triggered loop: read, compare, replace the owned RRset if different, then read again later. “Replace” here is an interface contract for the complete RRset under management; an adapter may implement it through a standards-based DNS update mechanism or through a provider API. The reconciliation logic must not depend on whether the previous call returned success.
package dnsstate
import (
"context"
"errors"
"time"
)
var ErrNotConverged = errors.New("published MX does not match desired MX")
type Authority interface {
LookupMX(context.Context, string) ([]MX, error)
ReplaceMX(context.Context, string, []MX, string) error
}
type Intent struct {
Domain string
Revision string
MX []MX
}
func ReconcileMX(ctx context.Context, dns Authority, in Intent) error {
got, err := dns.LookupMX(ctx, in.Domain)
if err != nil {
return err
}
if sameMX(in.MX, got) {
return nil
}
operationID := normalizeName(in.Domain) + ":" + in.Revision
if err := dns.ReplaceMX(ctx, in.Domain, normalizeMX(in.MX), operationID); err != nil {
return err
}
timer := time.NewTimer(2 * time.Second)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
}
got, err = dns.LookupMX(ctx, in.Domain)
if err != nil {
return err
}
if !sameMX(in.MX, got) {
return ErrNotConverged
}
return nil
}
The two-second delay is an example scheduler interval, not a DNS guarantee and not a recommended constant. Production code should put retry timing outside this function, honor cancellation, use bounded backoff with jitter, and retain the same desired revision across attempts. A non-converged result belongs back on the queue. It should not be translated into “create another MX record.”
There is a sharp ownership boundary: replacing an RRset is safe only when this controller owns the whole RRset. If another team or system legitimately manages some MX values at the same owner name, define a merge contract or move ownership before automating. Blind read-modify-write is vulnerable to a concurrent change between the read and write, while unconditional replacement can erase someone else’s intent. Neither deserves a quiet retry loop.
Choose adapters by semantics, not screenshots
The adapter decision is less about a polished zone editor than about whether the underlying mechanism can express the controller’s contract. Evaluate it with failure injection: lose the write response, replay the same operation, race two revisions, return answers in a different order, and leave the published RRset temporarily unchanged.
Then inspect DNS, not the job log.
Three mechanism families are common enough to compare without turning this into a vendor ranking:
| Mechanism | Useful boundary | Operational trade-off |
|---|---|---|
| Dynamic Update defined by RFC 2136 | Changes DNS data through prerequisite and update sections | Requires careful authorization and server support; prerequisites can protect concurrent changes |
| Provider-specific API | Can expose an atomic RRset replacement operation | Idempotency and concurrency semantics vary, so the adapter must test and document them |
| Declarative zone deployment | Reviews a complete desired zone before publication | Broad changes may have a larger blast radius, and convergence depends on the deployment pipeline |
The acceptance test stays the same for all three. Given identical intent, repeated reconciliation must settle on the same published RRset. Given a newer revision racing an older one, the system needs an explicit ordering or concurrency rule so the stale worker cannot win silently. Logs should carry domain, intent revision, operation ID, observed digest, and outcome; metrics should count drift age and reconcile results, without placing full email addresses or sensitive TXT payloads in labels.
This is also where dashboards tend to lie by aggregation. A 99.9% success panel can hide one school domain that has disagreed with intent for hours. Page on actionable, sustained drift after the allowed convergence window, and route first-attempt errors to logs or a ticket unless they consume the retry budget. The exact window is an operational choice informed by the authoritative service and the change process; guessing a universal number would create false confidence.
Where does this pattern stop applying?
Do not use automatic replacement when the controller cannot prove ownership of the RRset, when a human approval is required for each policy change, or when the desired state is intentionally partial. In those cases, detect and report drift, but stop before mutation. The same restraint applies during a disputed domain transfer or any period when two control planes may write the same owner name.
DMARC needs another caution. RFC 7489 specifies policy discovery, alignment, and reporting behavior, but a domain team still has to choose policy based on its authenticated mail sources. A generic provisioner should validate and publish approved intent; it should not invent a stricter policy during a retry. Reconciliation preserves a decision. It does not make the decision sound.
The final readiness condition for the edtech mail move is therefore plain: the approved MX intent is versioned, authoritative observation matches its normalized RRset, the DMARC record matches separately approved policy, and verification refers to the current revision. If a retry can produce any other interpretation, the workflow is not ready for the pager.
Top comments (0)