Short answer: for DNS provisioning, use create when a duplicate is the failure you want to expose, and use a narrow upsert when retries should converge on a fully specified record. The choice matters during a healthtech MX cutover because a conflict can reveal a successful first write, while an apparently successful write still leaves mail waiting on propagation.
The page usually fires late. A notification queue is retrying, a verification message has not arrived, and the deployment dashboard is green. Someone asks whether the DNS upsert or create call failed. That is the wrong starting point: first decide which failure your provisioning retry must make visible.
1. Define the failure before choosing create or upsert
create asserts that the name does not exist. A retry that receives a conflict has learned something useful: the first request may have reached the control plane. Read the existing record, compare owner name, type, TTL, and every value, then either adopt the matching state or stop on a mismatch.
upsert asserts that a desired state should exist after the operation. It is convenient for reconciliation, but a broad payload can overwrite an operator's temporary MX target. That is not a harmless retry; it is an unreviewed policy change.
Three words help: unknown is not failed.
Give every change an operation key containing the zone, owner, type, and deployment revision. Persist the intended RRset beside that key. When transport returns a timeout after the request was sent, retry with bounded backoff and perform a read-before-write or read-after-write check. A 2xx response only confirms API acceptance. It does not prove that recursive resolvers have discarded their cached answer.
2. How should DNS upsert and create shape provisioning retries?
Treat the retry path as a small state machine. For create, classify a conflict as a branch: matching state means the operation can be adopted; differing state means human review. For upsert, require the same post-write comparison, because convergence is only safe when the payload describes the complete RRset and the controller is its owner.
I once expected a two-second API response to make a test cutover ready. The response was fine. Resolver probes still returned the previous MX set in two regions, so enabling notification traffic at that point would have created a false green. The useful record in the incident log was not “request succeeded”; it was the operation key, the authoritative revision, each probe location, and the age of the answer. That detail let the on-call distinguish a transport retry from normal cache lifetime instead of sending a second, conflicting change.
Keep two clocks in the runbook: control-plane convergence and DNS visibility. They rarely finish together.
Do not retry validation failures. Retry an unknown transport result, cap the attempts, then reconcile. If ownership cannot be established, leave the RRset unchanged and page the zone owner.
3. What does a safe healthtech MX cutover verify?
Stage the change. Capture the exact MX RRset in version control, including preference ordering, TTL, and name normalization. Lowering TTL ahead of a migration can reduce the later cache window, but TTL is a cache hint, not a global stopwatch; recursive resolvers have their own behavior and policy.
The authoritative check answers “did the zone publish this revision?” Resolver checks answer “can the mail workers in their actual regions see it yet?” Use both. A practical alert trace works backward from the page:
- Delivery latency crosses the service objective for one domain.
- Queue retries rise while other domains remain normal.
- Regional resolver probes show the old MX answer beyond the accepted observation window.
- The controller records the authoritative revision and resolver answer age.
That fourth signal is the instrumentation change. Thresholds need care. Paging on the first stale answer creates noise during ordinary caching; waiting indefinitely hides a delegation mistake. I am not sure one universal window exists, so derive it from the notification workflow, resolver locations, and the time the incident team can tolerate before failing over.
Here is a deliberately boring comparison function for a controller. It keeps preference order visible instead of sorting away a potentially meaningful mistake.
type MXRecord struct {
Name string
Value string
Pref uint16
TTL uint32
}
func sameRRset(want, got []MXRecord) bool {
if len(want) != len(got) {
return false
}
for i := range want {
if want[i] != got[i] {
return false
}
}
return true
}
Normalize DNS names consistently before comparison, but do not silently reorder records unless the policy explicitly says order is irrelevant.
4. Where do drift, conflicts, and ownership boundaries belong?
Drift is a signal, not automatically a defect. An incident commander may add a temporary destination, or a compliance review may require a hold. Record who changed the RRset, why, and which revision it superseded. A narrow upsert fits a declared whole-record policy; create fits a unique migration marker where a duplicate should stop the rollout.
The catch is that a controller should not silently take over a customer-managed zone. When a registrar or another team is authoritative, produce a plan, verify the resulting answers, and ask that owner to apply the change. Stick with a plan-and-verify workflow when ownership is unclear.
The recommended pattern is unsuitable when several independent writers intentionally manage different members of the same RRset. In that case, use an ownership model that can merge those contributions, or keep the automation read-only; a blind upsert will erase context that the retry cannot recover.
5. Which tests expose the real provisioning failure?
Test the ugly paths, not just 2xx. Include a timeout after send, duplicate create, partial RRset input, stale control-plane reads, resolver disagreement, and equal values presented in a different order. Add a trailing-dot case so string normalization does not masquerade as a DNS change.
Log a compact, searchable code such as DNS_CONFLICT or DNS_PROPAGATION_PENDING with the operation key. Then make the decision table part of the runbook:
| Situation | Prefer | Failure you want | Follow-up |
|---|---|---|---|
| New unique migration marker | create |
Conflict | Read and compare; stop on mismatch |
| Full RRset owned by one controller | Narrow upsert
|
Convergence delay | Verify authoritative and recursive views |
| Customer-owned or shared zone | Neither blindly | Ownership ambiguity | Produce a plan for the zone owner |
| Retry after unknown timeout | Same verb plus reconciliation | Duplicate or stale state | Use key and revision |
The trade-off is operational, not fashionable. upsert can shorten the control-plane path but cannot shorten resolver caches. create adds conflict handling to the happy path, yet that conflict may be the evidence that prevents duplicate or destructive provisioning. For a healthtech mail domain, I would use a unique create marker, a narrow upsert for the complete MX set, and a separate propagation gate before enabling notification traffic.
Top comments (0)