Short answer: model each tenant's MX records as one complete, versioned desired set, reconcile that set through an idempotent provider interface, and alert on sustained drift between configuration and observed DNS state. For a fintech platform, keep platform-owned zones on the automatic path; require an explicit delegation and verification workflow for customer-owned zones.
The page should say which tenant is exposed, which desired version failed to converge, and whether ownership is platform or customer controlled. A page that says only MX update failed arrives too late and leaves the on-call guessing.
Keep the blast radius small.
What should a declarative Node.js MX records upsert do with configuration priorities?
Treat configuration as the whole answer for one DNS name, not as a stream of imperative additions. The reconciler reads the desired set, normalizes it, compares it with the observed set, and writes only when they differ. The same contract fits a Node.js worker, but the runnable example below is Go so the state machine and cancellation behavior stay explicit.
For this example, a tenant named ledger-labs receives mail at ledger-labs.mail.example.net. Its configuration contains two exchanges with explicit priorities. Those numbers are application data: validation makes duplicate record entries and invalid names deployment errors before any provider call. The generation value gives operators a stable identifier to put in logs, metrics, and a change record.
{
"tenant": "ledger-labs",
"zoneOwnership": "platform",
"name": "ledger-labs.mail.example.net",
"generation": 42,
"mx": [
{"priority": 10, "exchange": "inbound-a.mail.example.net"},
{"priority": 20, "exchange": "inbound-b.mail.example.net"}
]
}
Do not let array order define behavior. Sort by priority and then exchange before comparing or hashing a set. Otherwise, a harmless reorder in a configuration serializer can look like drift, trigger a write, and reset every convergence timer tied to that resource. This is the sort of tiny control-plane mistake that turns a quiet deploy into an unnecessary page.
The upsert boundary should accept the complete desired set. It should not expose add one record to the scheduler, because a crash between two additions can leave a half-applied route. Whether the underlying DNS system offers a replacement primitive or needs another transactional mechanism is an adapter concern; the scheduler's contract remains one desired set in, one observed set out.
Work backward from the page
Suppose the on-call receives TenantMailRouteDrift for ledger-labs, generation 42, after two reconciliation windows. The immediate runbook action is to compare three things: the deployed configuration generation, the normalized desired hash, and the normalized observed hash. Ownership belongs on the page too. With a platform-owned zone, the team can reconcile directly. With a customer-owned zone, the same page should point to the delegation or verification state instead of pretending the platform can mutate somebody else's DNS.
That page is the end of a chain. Before it fires, one reconciliation attempt should emit a structured result with a bounded status vocabulary such as converged, changed, invalid, or ownership_blocked. The signal that should fire earlier is a growing age of unconverged desired state, not a raw count of provider calls. A call count confuses harmless periodic reads with an actual routing risk.
A practical event can look like this:
{
"event": "mx_reconcile",
"tenant": "ledger-labs",
"generation": 42,
"ownership": "platform",
"result": "converged",
"desiredHash": "6f87c9d1",
"observedHash": "6f87c9d1"
}
Avoid putting record names or exchanges into metric labels. Those values grow with tenant count and belong in logs or traces. A counter can use the small result vocabulary; a gauge can track the age of the oldest unconverged generation by environment and ownership class. The page then links to the detailed event using tenant and generation as search fields.
There is a hard operational distinction between invalid and unconverged. Invalid configuration should fail the release gate and never enter the retry loop. Unconverged state was valid but has not yet matched observation, so it deserves retry with cancellation, a deadline, and eventually an alert. Mixing the two creates noisy retries for data that no amount of waiting can repair.
Implement one idempotent reconciliation pass
The following program is deliberately provider-neutral. DNSStore is the adapter boundary; production adapters can map it to an authoritative DNS control plane without leaking provider-specific operations into scheduling logic. The in-memory adapter keeps the example runnable and makes the second-pass no-op visible.
package main
import (
"context"
"fmt"
"sort"
"strings"
)
type MX struct {
Priority uint16
Exchange string
}
type Desired struct {
Tenant string
Name string
Generation int
Records []MX
}
type DNSStore interface {
ReadMX(context.Context, string) ([]MX, error)
ReplaceMX(context.Context, string, []MX) error
}
type MemoryStore map[string][]MX
func (m MemoryStore) ReadMX(_ context.Context, name string) ([]MX, error) {
return append([]MX(nil), m[name]...), nil
}
func (m MemoryStore) ReplaceMX(_ context.Context, name string, records []MX) error {
m[name] = append([]MX(nil), records...)
return nil
}
func normalize(records []MX) ([]MX, error) {
out := append([]MX(nil), records...)
seen := make(map[string]struct{}, len(out))
for i := range out {
out[i].Exchange = strings.ToLower(strings.TrimSuffix(out[i].Exchange, ".")) + "."
if out[i].Exchange == "." {
return nil, fmt.Errorf("empty exchange")
}
key := fmt.Sprintf("%d/%s", out[i].Priority, out[i].Exchange)
if _, exists := seen[key]; exists {
return nil, fmt.Errorf("duplicate MX %s", key)
}
seen[key] = struct{}{}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Priority == out[j].Priority {
return out[i].Exchange < out[j].Exchange
}
return out[i].Priority < out[j].Priority
})
return out, nil
}
func equal(a, b []MX) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
func reconcile(ctx context.Context, store DNSStore, desired Desired) (string, error) {
want, err := normalize(desired.Records)
if err != nil {
return "invalid", fmt.Errorf("tenant %s generation %d: %w", desired.Tenant, desired.Generation, err)
}
got, err := store.ReadMX(ctx, desired.Name)
if err != nil {
return "unconverged", err
}
got, err = normalize(got)
if err != nil {
return "unconverged", err
}
if equal(want, got) {
return "converged", nil
}
if err := store.ReplaceMX(ctx, desired.Name, want); err != nil {
return "unconverged", err
}
return "changed", nil
}
func main() {
store := MemoryStore{}
desired := Desired{
Tenant: "ledger-labs", Name: "ledger-labs.mail.example.net", Generation: 42,
Records: []MX{{Priority: 20, Exchange: "inbound-b.mail.example.net"}, {Priority: 10, Exchange: "inbound-a.mail.example.net"}},
}
first, err := reconcile(context.Background(), store, desired)
if err != nil {
panic(err)
}
second, err := reconcile(context.Background(), store, desired)
if err != nil {
panic(err)
}
fmt.Println(first, second)
}
go run main.go
changed converged
The second result is the property to preserve during retries. If a scheduler loses its acknowledgement after the first pass and runs again, it observes the desired set and does no write. Don't infer success from the write response alone; read-after-write observation, performed by this pass or a later one, is what closes the control loop.
The example returns errors because callers need to choose policy. A production worker should attach a context deadline, record the generation and result, and retry only errors its adapter classifies as retryable. I'm not sure one universal retry interval is defensible: the right value depends on the DNS control plane's documented behavior and the business tolerance for delayed tenant onboarding. Resolve that uncertainty with the provider contract and an end-to-end staging measurement, not a guessed constant copied into every environment.
Choose zone ownership before automating writes
Customer-owned and platform-owned zones need different runbooks. The choice is architectural, not a checkbox hidden inside the DNS adapter.
| Decision | Platform-owned zone | Customer-owned zone |
|---|---|---|
| Write authority | Platform reconciliation worker | Customer or delegated automation |
| Fast path | Validate, reconcile, observe | Publish required intent, verify delegation, observe |
| Primary failure boundary | Internal configuration versus observed state | Ownership or delegation versus requested state |
| Offboarding | Remove the tenant set through the same reconciler | Stop asserting intent and verify the customer-side change |
Automatic writes are a good fit when the platform owns the zone and can enforce one change path. The catch is customer policy: some regulated tenants require direct control of their DNS. In that case, don't route around ownership with credentials copied into a central worker. Keep the customer-owned workflow, provide an exact desired record set, verify what is observable, and make the responsible party explicit in the alert.
Platform ownership has a cost too. It expands the set of tenant routing state carried by one control plane, so authorization, audit records, configuration review, and rollback discipline matter more. Customer ownership distributes that authority but introduces a coordination boundary. Neither choice removes reconciliation; it changes who may perform the write and what actionable means to the on-call.
Mail authentication policy is adjacent to this workflow, not a substitute for it. DMARC is defined by RFC 7489 as Domain-based Message Authentication, Reporting, and Conformance. Treat its policy configuration as a separately reviewed concern rather than silently coupling it to an MX routing change.
Deploy the signal before the automation
Start with a dry-run reconciler that reads desired and observed sets, normalizes both, and emits results without writing. Feed it known fixtures: records presented in a different order, exchanges with and without a trailing dot, an empty exchange, a duplicate pair, and a tenant whose zone is customer-owned. Then enable writes for one platform-owned test zone and require the first pass to report changed and the next to report converged.
Promotion should be based on behavior, not elapsed time. A release gate can reject invalid configuration; a canary can compare the new reconciler's desired hash against the current worker's desired hash; and the deployment can stop if those disagree. This catches normalization changes before they become DNS changes. Rollback means restoring the prior desired generation and letting the same reconciler converge to it. No special delete script.
Alert only after the worker has had enough opportunities to observe convergence under the documented control-plane behavior.
Page on impact.
Send individual transient attempts to logs. Set the threshold too low and routine convergence produces false positives; repeated pages train responders to discount the alert, which is exactly what a mail-routing signal cannot afford. Set it too high and tenant onboarding can remain wrong without human attention. Your mileage may vary, so calibrate with staging observations and revisit the threshold in postmortems whenever the page was early, late, or unactionable.
The final runbook question is blunt: can the responder act? If ownership is customer-side, route the notification to the team that can change the zone and keep platform responders informed. If ownership is platform-side, include desired generation, both hashes, last successful convergence, and the deployment identifier. A page without those fields is an invitation to improvise under pressure.
Top comments (0)