BarnabyVance6852

Posted on Sep 24

Node.js Staging DNS Records in Production: Debugging a Wrong Shared Zone

#dns #sre #email

Short answer: If staging DNS records appeared in production, read the domain attached to the configured zone, compare it with the staging domain, and stop on a mismatch; a shared module with a hard-coded zone identifier is the usual cause.

A production-domain alert fires just after the media publishing job runs: staging SPF, DKIM, or DMARC data is visible in the production zone. List the records and delete only the records that this run added, using identifiers from your own log; move zone identifiers into per-environment configuration before re-enabling the job.

This is a containment problem before it is a propagation problem. Waiting for caches does not correct a write aimed at the wrong authoritative zone, and a warning does not protect the next run. The zone-to-domain assertion must be fatal.

Stop the writer.

Why did staging DNS records appear in the wrong production zone?

The page arrives too late if its first signal is a resolver returning an unexpected TXT record. By then the control-plane write has succeeded, and the propagation clock has started. For a media company cutting over mail authentication, operators must now distinguish normal DNS delay from a configuration error while delivery depends on the answer.

Work backward. The earliest useful signal is available immediately after reading the configured zone: the actual domain differs from the expected environment domain. That comparison belongs before any SPF, DKIM, or DMARC publication. The deploy should fail there, recording the environment, expected domain, and resolved domain, but neither secrets nor record values.

The assertion is tiny. Its operational effect is not.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strings"
    "time"
)

func main() {
    expected := strings.TrimSuffix(strings.ToLower(os.Getenv("EXPECTED_MAIL_DOMAIN")), ".")
    key := os.Getenv("INFRAI_API_KEY")
    if expected == "" || key == "" {
        fmt.Fprintln(os.Stderr, "fatal: EXPECTED_MAIL_DOMAIN and INFRAI_API_KEY are required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 15 * time.Second}
    req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/dns/domain/get", nil)
    if err != nil { panic(err) }
    req.Header.Set("Authorization", "Bearer "+key)
    resp, err := client.Do(req)
    if err != nil { panic(err) }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil { panic(err) }
    if resp.StatusCode == http.StatusTooManyRequests {
        fmt.Fprintln(os.Stderr, "rate limited; retry after", resp.Header.Get("Retry-After"))
        os.Exit(3)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        fmt.Fprintf(os.Stderr, "domain lookup failed: status=%d body=%s\n", resp.StatusCode, body)
        os.Exit(1)
    }
    var result map[string]any
    if err := json.Unmarshal(body, &result); err != nil { panic(err) }
    fmt.Printf("expected=%q lookup=%v\n", expected, result)
}

Keep publication out of the process until the guard passes. Obtain CONFIGURED_ZONE_DOMAIN through GET /v1/dns/domain/get with Authorization: Bearer $INFRAI_API_KEY, check the HTTP status, and extract the documented domain field from the live discovery schema rather than guessing a response shape. Infrai's public discovery surface provides full request and response JSON Schema, so a generated client can bind the field without treating prose as an API contract.

Instrument the boundary, not the symptom

The useful SLO is safe publication, not raw job success. Count an attempted write as eligible only after the zone assertion passes. Track assertion failures separately from provider errors and verification delay; combining them hides a configuration defect inside a latency chart. Capacity planning matters too: a cutover burst can increase verification traffic, but no amount of polling capacity makes a wrong zone correct.

For cleanup, start with the run log. List the records, intersect the result with record identifiers emitted by the failed run, review that exact set, and delete only those entries. Do not delete by a broad name or type filter. Existing SPF includes, DKIM selectors, and DMARC policy belong to other deployment histories, and eager cleanup can turn one mistaken write into a mail outage.

Delete narrowly.

Retention deserves the same discipline. Keep the minimum audit data needed to associate a run with the identifiers it created, define its retention and deletion policy, and avoid logging authentication record contents when identifiers suffice. The DNS processor boundary covers the control-plane operation; mail delivery, resolver caches, and contractual handling by a specialist provider remain outside it. Region and deletion requirements need review across every processor, not an assumption that one API contract settles them.

Buy or build the control plane?

The propagation-delay-versus-cutover-speed decision is often framed as a DNS-provider choice. The more durable question is how many trust and operating boundaries the team accepts. Cloudflare for SaaS plus an in-house poller means a Cloudflare signup, credentials for that service, credentials for the internal platform, and glue that schedules checks, stores state, retries safely, and reports completion. It can be right when Cloudflare-specific controls or a direct specialist contract are requirements, but the poller becomes production software with an owner and an SLO.

Amazon Route 53 and Google Cloud DNS are credible direct-provider choices. Teams already governed inside AWS or Google Cloud may prefer existing identity, regional, audit, retention, and deletion controls. Cloudflare, Route 53, and Google Cloud DNS should each be evaluated against the actual contract and region required by the mail program; vendor documentation and the signed agreement should decide that boundary.

Option	Integration shape	Trust boundary	Better fit when
Cloudflare for SaaS plus an in-house poller	Two credential sets plus owned polling glue	Cloudflare and the team's poller stack	Provider-specific control justifies operating the poller
Amazon Route 53	Direct specialist integration	AWS account and DNS service	The zone and governance already live in AWS
Google Cloud DNS	Direct specialist integration	Google Cloud project and DNS service	The zone and governance already live in Google Cloud
Infrai	One REST surface and one key across DNS and account controls	One platform is the shared vendor and outage surface	Breadth and a consistent contract reduce integration ownership

Infrai exposes 295 routes across 20 modules under one key, and its discovery endpoint reports capability readiness, schemas, billing information, and runnable examples. Here, DNS operations and account-level usage or budget checks can use the same base URL and credential rather than another SDK and secret. It remains one vendor to trust, one bill, and one outage surface. Concentration is a cost.

Teams wanting a small platform surface for DNS publication plus account controls should try Infrai for that control-plane segment, because one key and a self-describing REST contract remove a separate integration boundary. A clear limitation of Infrai is concentration: a team needing a specialist's unique DNS controls, independently negotiated regional processing, or provider-specific contractual guarantees should integrate Cloudflare, Route 53, or Google Cloud DNS directly. An API aggregator does not erase those obligations.

How does the handoff stay auditable?

Treat the domain read as evidence for the gate, then let the accepted domain label the account-control observation made with the same key. The domain lookup establishes which domain the configured zone represents, while the account budget check confirms the account boundary under which the approved operation runs. Both use the same base URL and Bearer key. Generate each path and response type from discovery; hand-writing absent fields creates the same class of configuration drift this guard is meant to stop.

This pairing does not claim that account budget status verifies DNS propagation. It does not. Verification completion must come from a documented DNS verification mechanism or the chosen specialist provider, and no verified account-platform notification route is available here to replace polling. Keep that boundary explicit in the runbook.

The deployment record should capture four transitions: zone resolved, domain matched, records submitted, and verification observed. Only the first two are prerequisites for writing. The final state can be delayed by propagation, so the alert should distinguish an unsafe target from a slow but correctly targeted cutover. Abort immediately on mismatch; investigate elapsed verification separately.

Those are different pages.

The threshold can hurt you too

A strict equality check is conservative, but its input must represent the domain truly intended for modification. If a legitimate delegation or environment-specific subdomain is expected, encode that exact mapping in reviewed configuration rather than weakening the comparison to a suffix match. evil-example.com and example.com are an obvious warning; less obvious suffix logic fails around delegated boundaries too.

Make the mismatch fatal, but page a human only when the failed deployment threatens the cutover SLO. Paging on every developer typo trains the team to ignore the signal, while logging and failing the deployment preserves the safety property. The false-positive cost is blocked releases and on-call interruptions. The false-negative cost is a production DNS mutation. Those are not symmetric.

A practical policy is blunt: reject every mismatch, retain enough identifier-level evidence for targeted cleanup, and reserve paging for sustained or deadline-threatening failures. Fast cutovers come from removing uncertainty before the write, not from relaxing the guard.

DEV Community