Defaulting DNS Record Writes to Upsert for Provisioning Idempotency

TL;DR

Use upsert as the default DNS record write for automated tenant subdomain provisioning, but make the desired record set explicit, verify ownership before writing, and reserve create-only writes for control-plane invariants that should page on drift.

The page usually doesn't say "your DNS mutation strategy was wrong." It says a fintech tenant cannot receive callbacks at ledger-17.payments.example.com, or a customer support thread says login links stopped resolving after an account rename. The runbook starts at the visible symptom, but the earlier signal should have been a control-plane event: the provisioner tried to converge a record, the record already existed with a different value, and the system treated that as either success or a fatal conflict without checking intent.

I've been paged by missed jobs and duplicate deliveries; DNS provisioning has the same shape. A retry after a timeout must be boring. A second worker must not create a second answer by accident. A manual fix in a customer-owned zone must not be silently overwritten unless the customer explicitly delegated that control.

Small rule, big blast radius.

What does the on-call page tell you?

The alert should be tied to a tenant state transition, not just a DNS API response. A useful page says the tenant subdomain has been in provisioning_dns for 15 minutes, the expected record is CNAME ledger-17.payments.example.com -> ingress-a.platform.example.net, and the last reconcile attempt ended with a write conflict or a verification mismatch. That gives the responder a record name, an expected value, and a next action.

A weak page says "DNS write failed." That burns time because DNS writes are only one step in a longer chain: tenant creation, domain policy, record publication, propagation checks, certificate issuance, routing table update, and customer-visible activation. If the alert fires at the final symptom, the customer finds the broken hostname before the operator sees the bad transition.

For a fintech system, I would model tenant DNS as desired state. The provisioning database owns an intent row: tenant id, zone ownership mode, record name, record type, desired values, TTL, and a monotonic revision. The DNS writer is a reconciler. It does not ask, "can I create this?" on every retry. It asks, "does the published record set match the intent for this revision?"

That distinction matters because DNS record sets are shared objects. Some providers expose one call that overwrites the whole record set. Some expose changes as add and delete operations. Some normalize values before returning them. The article question says Node.js, but the language is not the decision point; the same write contract applies in Node.js, Go, or a shell job. The hard part is deciding what a retry is allowed to change.

Should DNS record writes use create, update, or upsert for provisioning idempotency?

For tenant subdomain provisioning, default to upsert when the platform owns the zone and the record name is fully generated by your control plane. Use update after a successful read when you need compare-and-set behavior. Use create when existence itself means someone violated an invariant.

Here is the decision table I use in design reviews:

Write shape	Good default for	Bad fit when	Pager meaning
Create	One-time reservations, ownership tokens, unique control rows	Retries after ambiguous network results	"This name should never have existed"
Update	Editing a known record with a version, ETag, or prior read	First-time provisioning with racing workers	"The record changed since we read it"
Upsert	Converging generated records toward desired state	Customer-managed records with manual edits	"The desired state could not be verified"

Upsert is the least surprising default for platform-owned tenant subdomains because it makes retries idempotent. If worker A writes the desired CNAME and then loses the response, worker B can send the same desired record set and land in the same state. If the operation returns a 409-style conflict because another reconcile moved the revision, the worker can read, compare, and decide whether the published state already matches the current intent.

The catch is customer-owned zones. If a customer controls customer.example and asks you to use pay.customer.example, your platform should usually publish instructions and verify the target record, not keep mutating their zone. An upsert against a customer-owned zone can erase a deliberate local decision, especially around mail-related TXT records. DMARC is a useful reminder here: RFC 7489 defines domain-based email policy records that domain owners publish in DNS, and those records express policy. A SaaS control plane should not treat that kind of domain policy as scratch space.

So the default is conditional, not religious. Platform-owned generated hostname? Upsert. Customer-owned zone without delegated automation? Verify and report drift. Existing record with security meaning? Read first and require an explicit migration.

A safer write contract for tenant subdomains

The write path needs one contract that the queue worker can retry without remembering which attempt won. This is the shape I prefer: compute the desired record set from tenant state, apply it with idempotent semantics when the platform owns the zone, then verify by reading the record set back through the same abstraction. Don't mark the tenant active on the write response alone.

package dnsprovisioning

import (
    "context"
    "errors"
    "fmt"
    "sort"
)

type RecordSet struct {
    Name   string
    Type   string
    TTL    int
    Values []string
}

type ZoneMode string

const (
    PlatformOwned ZoneMode = "platform_owned"
    CustomerOwned ZoneMode = "customer_owned"
)

type DNSWriter interface {
    UpsertRecordSet(ctx context.Context, zone string, record RecordSet) error
    ReadRecordSet(ctx context.Context, zone, name, recordType string) (RecordSet, error)
}

func ReconcileTenantDNS(ctx context.Context, writer DNSWriter, zone string, mode ZoneMode, desired RecordSet) error {
    if mode != PlatformOwned {
        return errors.New("customer-owned zones should be verified, not mutated by default")
    }

    normalize(&desired)

    if err := writer.UpsertRecordSet(ctx, zone, desired); err != nil {
        return fmt.Errorf("apply desired DNS record set: %w", err)
    }

    observed, err := writer.ReadRecordSet(ctx, zone, desired.Name, desired.Type)
    if err != nil {
        return fmt.Errorf("read back DNS record set: %w", err)
    }
    normalize(&observed)

    if !sameRecordSet(desired, observed) {
        return fmt.Errorf("verify DNS record set: desired %v, observed %v", desired.Values, observed.Values)
    }

    return nil
}

func normalize(record *RecordSet) {
    sort.Strings(record.Values)
}

func sameRecordSet(a, b RecordSet) bool {
    if a.Name != b.Name || a.Type != b.Type || a.TTL != b.TTL || len(a.Values) != len(b.Values) {
        return false
    }
    for i := range a.Values {
        if a.Values[i] != b.Values[i] {
            return false
        }
    }
    return true
}

Two details are doing most of the work. First, the function refuses to mutate customer-owned zones by default. That is not because automation is bad; it is because the ownership boundary changes the safety model. If the customer delegated a subzone or gave explicit automation rights, you can treat that delegated scope like platform-managed desired state. If they only copied a verification record, stay in read-and-report mode.

Second, activation waits for read-back verification. DNS propagation still needs its own checks, and caches can keep old answers around for the TTL period, but the control plane should at least prove that the authoritative record set now matches the intended state before it advances the tenant. Your mileage may vary on the exact timeout. The useful threshold depends on the registrar path, queue latency, certificate flow, and how much false paging the team can tolerate.

Observability before the customer notices

The earlier signal is not "tenant cannot resolve hostname." By then, the workflow has already leaked. The earlier signal is "desired DNS record set has not converged after N reconcile attempts or M minutes." That metric should carry labels for zone ownership mode, record type, operation shape, and terminal reason. Keep tenant id out of high-cardinality metrics unless your metrics system can handle it; put tenant id in structured logs and traces.

The runbook should separate four cases. No desired state means the app workflow is wrong. Desired state exists but no record is published means the writer or credentials need attention. A record exists with a different value means either drift or a racing revision. A record exists and matches desired state but the product is still waiting means the next dependency, often certificate issuance or routing activation, owns the delay.

I would also count duplicate reconcile attempts. Not as a vanity metric. Duplicate delivery is how the queue tells you the write path will be tested under pressure, and DNS upsert should make that boring. If duplicate attempts correlate with different desired values for the same record name, the problem is no longer DNS; the tenant state machine is emitting competing intents.

One practical alert:

type DNSConvergenceEvent struct {
    TenantID      string
    ZoneMode      ZoneMode
    RecordName    string
    RecordType    string
    DesiredHash   string
    Attempt       int
    ElapsedSeconds int
    Reason        string
}

func ShouldPage(event DNSConvergenceEvent) bool {
    if event.ZoneMode == CustomerOwned {
        return event.ElapsedSeconds >= 1800 && event.Reason == "customer_record_not_verified"
    }
    return event.ElapsedSeconds >= 900 && event.Attempt >= 3
}

Those numbers are examples, not universal law. The important part is having two thresholds. Customer-owned DNS deserves more patience because a human or a separate automation path may be involved. Platform-owned DNS should converge faster because the same control plane owns intent, credentials, and records.

Get the threshold wrong and the cost is real. Too low, and the on-call gets paged during normal DNS cache behavior or while a customer is still copying a record. Too high, and the first reliable signal is a failed webhook, a blocked onboarding flow, or a support ticket with a screenshot. Neither teaches the system to converge; both just move pain around.

References

• https://datatracker.ietf.org/doc/html/rfc7489