Go Reconciliation of Intended and Current DNS Configuration State

Short answer: treat DNS as a control loop, not a sequence of successful API calls. For a fintech platform that gives every tenant a subdomain, the intended state is the complete, versioned record set the platform is responsible for; the current state is a fresh, normalized observation from authoritative DNS. Reconcile only that owned record set, preserve everything outside it, and declare convergence only after authoritative observation matches the intent. Customer-owned zones need a verification boundary and slower, more defensive automation than platform-owned zones.

This distinction matters most during ordinary changes. A deployment can successfully request acme.pay.example and still leave an old verification record, an unexpected mail-policy record, or a value that has not become observable yet. The write succeeded. The system did not necessarily converge.

The incident-shaped lesson is bounded but familiar: imagine a tenant moving from a platform-owned subdomain to a delegated customer-owned name while an earlier reconciliation is still queued. Two individually valid writes can arrive out of order. If each worker treats its payload as a command, the stale job may restore the old answer. If the worker instead reloads the latest desired version, observes DNS, and computes a fresh plan, the stale job becomes harmless. The invariant is simple: a worker reconciles state, never history.

How should intended and current DNS configuration state converge?

Convergence is equality within an explicitly owned scope. It does not mean that every record visible at a name matches a platform database, because a customer may legitimately manage records that the platform neither understands nor owns. It also does not mean that a provider accepted a mutation. Acceptance is an input to the next observation cycle, not proof of the final condition.

For each tenant, keep four things separate:

• desired records, including type, owner name, value, and policy-relevant metadata;
• an ownership boundary that says which exact records the controller may change;
• observed authoritative records, normalized before comparison;
• reconciliation status, including desired version, last observation, last successful match, and retry state.

That model makes drift precise. Drift is a difference between desired and observed state inside the ownership boundary. A record outside the boundary is neither drift nor cleanup work. This sounds pedantic until a controller deletes a customer's unrelated record because it compared whole zones rather than managed subsets.

For DMARC, the boundary deserves special care. RFC 7489 defines DMARC policy records in DNS and describes policy discovery using a specific owner name. A fintech platform should therefore model a DMARC-related record as policy-bearing configuration, not as an arbitrary text blob to overwrite during subdomain provisioning. The exact desired policy remains an organizational decision; the controller's job is to preserve declared ownership and detect divergence, not invent policy.

The ownership boundary changes the control loop

Platform-owned and customer-owned zones may expose the same record types, yet they have different risk envelopes.

Decision	Platform-owned zone	Customer-owned zone
Authority	The platform controls the zone and its change process	The customer controls the zone or delegates a bounded name
Safe write scope	Explicit records created for tenant routing and policy	Only records covered by verified delegation or explicit authorization
Drift response	Automated repair is usually reasonable after validation	Report first; repair only where authorization is unambiguous
Deletion	Allowed after version and dependency checks	Conservative, because absence may reflect a customer's deliberate change
SLO emphasis	Reconciliation latency and stale-answer duration	Verification freshness, non-interference, and clear failure reporting

The table drives a practical rule: use one state machine, but different permissions and retry policies. In a platform-owned zone, the controller can generally close drift automatically within its managed record set. In a customer-owned zone, proof of control can expire or delegation can change, so every destructive plan needs a current authorization check.

No write should cross that line.

A controller also needs an explicit answer for partial delegation. If a customer delegates only tenant.customer.example, authority over that name does not imply permission to alter sibling names or the parent zone. Encode the authorized suffix and exact managed record keys in data. Do not infer them from a convenient credential.

A Go reconciler that rejects stale work

The preventative path is small enough to review. The interfaces below deliberately separate authoritative observation, desired-state storage, planning, and mutation. They also make the desired version part of the write precondition, which prevents an old worker from applying a plan after intent changes.

package dnscontrol

import (
    "context"
    "errors"
)

var ErrDesiredStateChanged = errors.New("desired DNS state changed")

type Record struct {
    Name  string
    Type  string
    Value string
}

type DesiredState struct {
    TenantID string
    Version  uint64
    Owned    []Record
}

type Change struct {
    Before *Record
    After  *Record
}

type Store interface {
    LoadDesired(context.Context, string) (DesiredState, error)
    StillCurrent(context.Context, string, uint64) (bool, error)
}

type Authority interface {
    Observe(context.Context, DesiredState) ([]Record, error)
    Apply(context.Context, DesiredState, []Change) error
}

type Planner interface {
    Diff(desired DesiredState, observed []Record) ([]Change, error)
}

func Reconcile(ctx context.Context, tenantID string, store Store, dns Authority, planner Planner) error {
    desired, err := store.LoadDesired(ctx, tenantID)
    if err != nil {
        return err
    }

    observed, err := dns.Observe(ctx, desired)
    if err != nil {
        return err
    }

    changes, err := planner.Diff(desired, observed)
    if err != nil || len(changes) == 0 {
        return err
    }

    current, err := store.StillCurrent(ctx, tenantID, desired.Version)
    if err != nil {
        return err
    }
    if !current {
        return ErrDesiredStateChanged
    }

    return dns.Apply(ctx, desired, changes)
}

The planner must normalize values according to the record type before comparison, reject changes outside Owned, and produce deterministic output. The Apply return value still does not establish convergence. A later pass must observe authoritative DNS again and find no diff.

There is another race between StillCurrent and Apply. In a real implementation, close it with a provider-side concurrency mechanism when one exists, or serialize mutation per ownership scope and check the version again before issuing each change. The important property is monotonic intent: once version 42 exists, version 41 must never be allowed to restore its records.

Retries require the same discipline. Retry observations freely with bounded backoff. Retry a mutation only after recomputing its plan against the latest desired and observed states. A persisted list of old DNS operations is dangerous because its assumptions age while it waits.

Operate the loop as an SLO-bearing service

A DNS controller is a production dependency, so its useful indicators describe state rather than request volume. Track the age of unconverged desired versions, the number of tenants outside their reconciliation objective, observation failures by authority boundary, rejected stale plans, and changes attempted outside the owned set. A dashboard of successful write calls can remain green while tenants point at yesterday's target.

Set the objective around what users experience: the interval from an accepted desired-state change to an authoritative observation that matches it. Keep the measurement honest by separating controller queue time, provider mutation time, and the time until the new answer is observed. Those components have different owners and different remediation paths.

Alert on burn rate, not on every mismatch. A brief mismatch is the control loop doing its job; a growing population of old mismatches means capacity, authorization, or authority lookup is failing. Capacity planning follows from the same data. Size workers for the peak rate of tenants requiring observation and repair, then reserve headroom for a zone-wide event that makes many tenants drift at once. Average mutation traffic is a poor sizing input.

Testing should attack the invariants. Feed the planner reordered equivalent observations and expect no changes. Insert unrelated customer records and prove they survive. Run version 7 and version 8 jobs in reverse order and prove the result represents version 8. Simulate an observation error after an accepted write and verify that the next pass observes before mutating. For customer-owned names, revoke authorization between planning and application and require a safe refusal.

Buying the DNS path or building the controller

The relevant choice is not a feature checklist. It is who owns the state model, authorization boundary, and pager.

Concern	Managed control plane	In-house controller
State model	Faster start if its ownership model matches yours	Exact fit for tenant versions and internal workflows
On-call load	Less infrastructure, but external failure boundaries remain	Full responsibility for queues, storage, retries, and mutation safety
Lock-in	Data and workflows may follow a specific API model	Interfaces can remain generic, though DNS authorities still differ
Customer-owned zones	Useful only if authorization can be narrowly represented	More control, with more security review and support burden
Evidence	Require authoritative observations and auditable changes	Must be designed, stored, and operated by the platform team

Build when the tenancy and authorization model is genuinely differentiating and the team can own a long-lived control plane. Buy when a managed system expresses the required boundary and exports enough state to measure the SLO. A hybrid is often coherent: retain desired state, ownership rules, and audit decisions internally while delegating authoritative mutation behind a narrow interface.

Do not decide from nominal record volume alone. The hard cost is exception handling: customer delegation mistakes, policy conflicts, stale jobs, and evidence for support investigations. Count on-call load and exit cost beside implementation work.

Where this advice stops

This control-loop approach has explicit limitations and trade-offs. It is not suitable as a way to grant authority: if a customer owns the zone and has not delegated a name or authorized a change, the correct reconciler result is blocked, not forced convergence. It also does not decide the organization's DMARC policy; security and mail owners must supply that intent. Teams unable to operate versioned state, authoritative observations, and per-scope serialization should choose a managed control plane that exposes those controls instead of building a partial reconciler.

For short-lived development names with no customer traffic or policy records, a lighter workflow may be sufficient. Production fintech tenant names deserve the stricter model because identity, routing, and policy can share the same namespace, while ownership is split across teams and organizations.

The durable rule is narrow: store versioned intent, observe authority, diff only owned records, and verify again. Convergence is an observed condition, not a successful command.

Sources

• https://datatracker.ietf.org/doc/html/rfc7489