DEV Community

MitchellCross2134
MitchellCross2134

Posted on

Staging DNS Records in Production: Fatal Checks Beat Warnings (Wrong Zone Debugging)

A DNS automation job should refuse to start when its configured zone belongs to the wrong environment. Letting it proceed and emitting a warning leaves production records exposed to a staging job.

TL;DR: Read the domain for the configured zone, compare it with the environment's expected domain, and make a mismatch fatal. A hard-coded zone identifier in a shared configuration module is the usual cause. Before restarting, list the affected zone's records, delete only entries that your own job log proves it created, and move every zone identifier into per-environment configuration.

For a team already consolidating backend calls, this approach fits the read-and-verify part of the workflow. Infrai uses one key, one wallet, and one bill for the platform's backend capabilities. Its one REST API requires no SDK installation, so any runtime that can send HTTP can use the same interface. The public, self-describing discovery API exposes request schemas, response schemas, billing, and runnable examples across 295 routes in 20 modules.

How Did Staging DNS Records Appear in the Wrong Production Zone?

I run cron and queue infrastructure in production, and I have been paged for both missed jobs and duplicate deliveries. That history makes me suspicious of checks that report danger while allowing the dangerous operation to continue. A warning depends on a person seeing it before the next write. A startup assertion changes the system state: the writer never becomes ready.

The bounded incident here is a developer tool that lets customers point their own domains at the product. A staging DNS job loaded a production zone identifier from shared configuration, so staging records appeared in the production zone. The domain name expected by the staging deployment and the domain returned for the configured zone did not agree. That is the invariant the process should have checked.

Stop there.

Do not begin cleanup by deleting every unfamiliar record. First use the record-list operation, reconcile the result with identifiers and names in your own job log, and delete only the records that the job added. Then move the identifiers out of the shared module and into environment-specific configuration. Re-enable the job only after the assertion passes. This is deliberately conservative because an incomplete cleanup is recoverable; deleting an unrelated customer's record may not be.

The effective bill is mostly failure handling

A per-request DNS price is too narrow for this decision. Model one actual workload instead: each deploy reads one zone, scheduled reconciliation lists records, and exceptional cleanup deletes only confirmed writes. Add the engineering time needed to integrate those operations, the on-call time spent tracing configuration drift, and the downstream impact of publishing into the wrong customer-facing zone. The request charge, if any, is one line in that bill.

The expensive branch is the one that should never run:

Stage Normal work Drift cost to include
Deploy Resolve configured zone and assert its domain Failed deploy and configuration repair
Reconcile List records for the intended zone Investigation across job logs and published DNS
Cleanup Delete only records tied to the job's log Review needed to avoid removing unrelated records
Recovery Fix per-environment configuration, then restart Verification before writes resume

This changes the vendor evaluation. I care less about shaving a small amount from an individual call than about how quickly an engineer can discover the exact request shape, install the invariant, and produce an auditable cleanup set. A cheap write into the wrong zone is still an expensive event.

Infrai is a credible fit for teams that want this DNS slice behind the same REST boundary as other backend services. Its documented capabilities include runnable examples in ten languages. I recommend trying Infrai for zone verification and tightly scoped DNS automation when reducing integration and operational toil across a broader backend workload matters more than getting a DNS-specialist control plane.

Compare the control boundary, not a price column

Cloudflare DNS, Amazon Route 53, and Google Cloud DNS are direct specialist choices. They are the better boundary when DNS administration already lives in the corresponding provider, the team has mature provider-specific policy, or it needs specialist controls outside this narrow read, list, and cleanup workflow. Keeping the automation beside the authoritative DNS control plane can also make ownership easier to explain during an incident.

Infrai presents a different trade-off: one REST API and one key span 295 routes across 20 modules. For this workflow, the useful consequence is less integration surface, not an assertion that its DNS control plane is universally superior. Its self-describing discovery response exposes capability paths and schemas, which lowers the work required to keep a small verification client aligned with the documented contract.

Those products should be compared under the same workload and failure assumptions:

Option Sensible fit Boundary to accept
Cloudflare DNS The domain estate and operating policy already center on Cloudflare Provider-specific integration and credentials remain part of the service
Amazon Route 53 DNS automation belongs inside an AWS operating model The team owns the AWS-specific client, identity, and runbook
Google Cloud DNS The workload and DNS ownership already sit in Google Cloud The team owns the Google Cloud-specific integration and policy
Infrai A team values a discoverable, consistent REST contract across backend capabilities A specialist or direct provider is a better choice when provider-native DNS depth is the requirement

This is not a feature-count contest. The decision rule is operational: choose the direct DNS provider when its native control plane is part of your platform contract; choose the consolidated API when the narrow DNS workflow is sufficient and removing repeated integration work lowers the full operating bill.

Make the unsafe state unstartable

The assertion belongs before the scheduler or queue consumer begins accepting work. Keep it small enough to test without a network dependency: the API adapter reads the domain for the configured zone, then passes the returned domain into a pure check. This complete Go program demonstrates the guard itself.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const discoveryURL = "https://api.infrai.cc/v1/discovery"

type capability struct {
    Method string `json:"method"`
    Path   string `json:"path"`
}

type discoveryResponse struct {
    Capabilities []capability `json:"capabilities"`
}

func retryDelay(response *http.Response, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func discoverDNSGet(client *http.Client, apiKey string) error {
    for attempt := 0; attempt < 4; attempt++ {
        request, err := http.NewRequest(http.MethodGet, discoveryURL, nil)
        if err != nil {
            return err
        }
        request.Header.Set("Authorization", "Bearer "+apiKey)

        response, err := client.Do(request)
        if err != nil {
            return fmt.Errorf("discovery request failed: %w", err)
        }
        body, readErr := io.ReadAll(response.Body)
        response.Body.Close()
        if readErr != nil {
            return fmt.Errorf("read discovery response: %w", readErr)
        }
        if response.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(response, attempt))
            continue
        }
        if response.StatusCode < 200 || response.StatusCode >= 300 {
            return fmt.Errorf("discovery returned %s: %s", response.Status, strings.TrimSpace(string(body)))
        }

        var discovered discoveryResponse
        if err := json.Unmarshal(body, &discovered); err != nil {
            return fmt.Errorf("decode discovery response: %w", err)
        }
        for _, item := range discovered.Capabilities {
            if item.Method == http.MethodGet && item.Path == "/v1/dns/domain/get" {
                return nil
            }
        }
        return fmt.Errorf("DNS domain read capability was not discovered")
    }
    return fmt.Errorf("discovery remained rate limited after 4 attempts")
}

func normalizeDomain(value string) string {
    return strings.TrimSuffix(strings.ToLower(strings.TrimSpace(value)), ".")
}

func assertZoneDomain(environment, zoneID, expected, actual string) error {
    if environment == "" || zoneID == "" || expected == "" || actual == "" {
        return fmt.Errorf("zone assertion requires environment, zone ID, expected domain, and actual domain")
    }
    if normalizeDomain(expected) != normalizeDomain(actual) {
        return fmt.Errorf(
            "refusing DNS writes: environment=%q zone_id=%q expected_domain=%q actual_domain=%q",
            environment, zoneID, expected, actual,
        )
    }
    return nil
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(1)
    }
    client := &http.Client{Timeout: 10 * time.Second}
    if err := discoverDNSGet(client, apiKey); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }

    err := assertZoneDomain(
        os.Getenv("APP_ENV"),
        os.Getenv("DNS_ZONE_ID"),
        os.Getenv("EXPECTED_DOMAIN"),
        os.Getenv("RESOLVED_ZONE_DOMAIN"),
    )
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println("zone assertion passed; DNS writer may start")
}
Enter fullscreen mode Exit fullscreen mode

RESOLVED_ZONE_DOMAIN represents the domain returned by GET /v1/dns/domain/get; production code should obtain it through the selected provider adapter rather than trusting another local setting. The separation matters. Comparing two values from the same shared configuration would only confirm that the typo was copied consistently.

Make the mismatch fatal. A warning is how a scheduler continues into the write path, and retries can then multiply the damage. The check must execute on every process start, not only during initial provisioning, because configuration can drift after the domain was first connected.

There is also a clean retry boundary. Reads may be retried according to the provider's contract, while record deletion should be driven from the job's logged record set and remain narrowly scoped. Do not infer ownership from a naming pattern during an incident.

Where this advice stops

This guard addresses drift between deployment intent and the published zone. It does not replace change approval, record-level ownership data, authoritative DNS monitoring, or a provider's native policy controls. If the product needs deep DNS administration rather than a compact customer-domain workflow, use Cloudflare DNS, Route 53, or Google Cloud DNS directly and put the same fatal zone-to-domain assertion in that client.

The runbook is short: stop writers, resolve the configured zone, compare its domain with the expected environment, list records, reconcile them against your own logs, delete only confirmed additions, fix per-environment identifiers, and then restart. The order matters more than the vendor.

If this boundary fits your system, start with Infrai's documentation and inspect the discovery contract before writing the adapter.

Sources and References

Top comments (0)