Short answer: make an idempotent upsert the default for tenant-subdomain provisioning, but only after you have decided who owns the zone and what an existing record is allowed to mean. Use create-only for a claim you must never overwrite; use update-only for an operator-approved change.
The page usually fires after the damage. A tenant cannot reach pay.example.com, a certificate check is waiting on a TXT record, or a retry has created two contradictory values. The on-call sees a DNS timeout and a provisioning job with three attempts. The signal that should have fired earlier was simpler: the write did not converge to the intended record within its deadline.
That distinction matters.
For a fintech platform that gives every tenant a subdomain, this is a policy problem before it is an API problem. The policy has to survive retries, deploys, and a handoff between the team that owns example.com and a customer that owns customer-bank.com.
What should create, update, and upsert mean for idempotent DNS provisioning?
Treat a DNS change as a small state transition with an explicit expected value. A create operation says the name must be absent. An update says the name must exist and match a precondition. An upsert says the desired state is authoritative, so repeating the same request produces the same final record. None of these words tells you what to do when an unrelated value is already present; your contract must.
For platform-owned zones, an upsert is normally the least surprising default when the record name is allocated from a durable tenant identifier. The worker can replay a message after a timeout without turning a retry into a duplicate. It should still compare type, name, routing policy, and value before declaring success. A successful HTTP response is not proof that recursive resolvers have the new answer yet.
For customer-owned zones, defaulting to overwrite is dangerous. A customer may have a CNAME, TXT, or verification record at the same label for a reason your control plane cannot see. In that boundary, create-only plus an explicit conflict result is safer. Ask the customer to delegate a label or provide an ownership proof, then write only inside that authority. The catch is operational friction: the first setup takes longer, and support needs a clear handoff checklist.
I once traced a noisy page back to a much less dramatic change than the alert suggested: a tenant migration had left an old TXT value at the delegated label, while a replayed provisioning message carried the new value. The worker treated a timeout as permission to try again, so the queue showed healthy throughput while the desired state never became true. The useful fix was not a larger retry count. We recorded the zone owner and change version with the intent, classified the old value as a conflict, and made the alert include the exact name and type. That gave the on-call a decision they could make in minutes: restore the previous intent, or ask the customer to remove the unmanaged value. Your mileage may vary on the timeout itself, but the ownership and version fields are still useful when the provider's dashboard is unavailable.
Here is the decision table I keep beside the runbook:
| Zone ownership | Operation default | Existing value | Retry behavior | Escalate when |
|---|---|---|---|---|
| Platform-owned | Upsert | Replace only a record previously managed by this tenant | Replay until the desired state is observed | Ownership token or type differs |
| Customer-owned, delegated label | Create, then update by version | Reject an unmanaged value | Retry the same write with an idempotency key | Delegation or proof is missing |
| Customer-owned, shared label | Update-only | Require an operator-approved precondition | Stop on conflict; do not guess | Any third-party value is present |
How does an alert-to-action workflow prevent duplicate DNS writes?
Start with an intent record in your database: tenant ID, fully qualified name, type, desired value, zone owner, and a monotonically increasing change version. The queue message carries the intent ID, not a fresh copy invented by each retry. A worker reads the current intent, obtains a provider change token if one exists, and submits one desired-state operation. It then polls the authoritative view or provider change status until the result is applied, conflict, or deadline_exceeded.
The instrumentation change is to measure convergence, not just request latency. Emit dns_change_intent_total, dns_change_conflict_total, and a histogram for dns_convergence_seconds, all labeled by zone ownership and record type. Page on a sustained rise in deadline expirations or on a tenant's change version stuck past its service objective. A single slow resolver should create a trace annotation, not a second write.
False positives have a cost. If the threshold is shorter than the provider's normal propagation window, the worker will page the same engineer who just approved the change, and an eager retry can amplify the queue. I'm not sure there is one universal timeout; measure the authoritative-to-recursive spread in your environment, then leave room for maintenance windows.
A small worker contract in Go
The interface below keeps provider details outside the reconciliation loop. It also makes the dangerous case visible: an existing value that this tenant does not own is a conflict, not an invitation to overwrite.
package dnsreconcile
import "context"
type Record struct {
Name string
Type string
Value string
}
type ChangeResult string
const (
Applied ChangeResult = "applied"
Conflict ChangeResult = "conflict"
)
type DNS interface {
Create(ctx context.Context, record Record, key string) (ChangeResult, error)
Upsert(ctx context.Context, record Record, key string) (ChangeResult, error)
Update(ctx context.Context, record Record, expected string, key string) (ChangeResult, error)
}
func Reconcile(ctx context.Context, api DNS, record Record, owner string, key string) (ChangeResult, error) {
switch owner {
case "platform":
return api.Upsert(ctx, record, key)
case "delegated-customer":
return api.Create(ctx, record, key)
default:
return api.Update(ctx, record, "managed-by-tenant", key)
}
}
The key must be stable across retries of one intent and different for a later change version. Persist the result before acknowledging the queue message. If the process dies after the provider accepts the write but before the acknowledgement, the next delivery uses the same key and desired value. That is the idempotency reflex that prevents a recovery drill from becoming a duplicate-delivery incident.
Node.js services can apply the same contract even if the reconciliation worker is written in Go: keep the record shape and conflict semantics in the service boundary, and test them with a fake provider that returns timeout, conflict, and success in that order. Do not test only the happy path.
Where the model does not fit
Upsert is a poor fit when a label is shared by several independent writers, when the value is customer content, or when deleting an old value has regulatory impact. Use an approval workflow and update-only semantics there. Create-only is also a bad default for a platform-owned name that can be reclaimed after a tenant migration; a stale record will make every retry look like a collision.
DNS itself is eventually consistent. A record can be correct at the authoritative server while a recursive cache still serves the previous TTL. The worker should report those states separately so the support team does not roll back a correct change. For DMARC and related policy records, preserve the customer's existing policy unless the contract explicitly grants ownership; RFC 7489 describes how those records are interpreted, but it does not grant your platform permission to replace them.
Top comments (0)