Short answer: monitor the outcome the logistics company needs — mail accepted and hostnames resolving to the intended targets — then inspect published DNS records to explain an alert. A record-presence check alone can stay green while a caching resolver or a provider check disagrees, so it is evidence for diagnosis, not proof of service.
That distinction determines what page should fire. The page should say that mail acceptance or name resolution failed; a record mismatch can open a lower-urgency drift alert before it becomes a ticket. Don't reverse those priorities.
What a DNS configuration incident actually teaches
Consider a bounded production scenario. A logistics company has pointed its mail at a new provider by publishing the required MX records. The desired configuration is clear, and a control-plane listing shows the expected entries. At 02:17, however, an external check receives NXDOMAIN from the resolver it uses. The useful alert is not “MX row missing,” because that claim may be false; the useful alert is “mail resolution failed from this observation point,” accompanied by the records that were published when the check ran.
No invented outage is needed to see the operational invariant: measure the contract at its boundary, and use configuration as explanatory context. For mail, that contract is provider verification or acceptance. For an application hostname, it is resolution to the intended target. The DNS record set helps answer why the contract failed, but it cannot establish that every relevant cache, delegation, and downstream provider agrees.
This also changes severity. A successful outcome with record drift deserves investigation, especially because slow drift is often the warning that arrives before a user report, but it isn't the same event as rejected mail. An outcome failure with apparently correct records deserves the page because something outside the row comparison is wrong. An outcome failure plus drift gives the responder an immediate lead.
What page fired?
If the answer is only “configuration differs,” the monitor hasn't yet told the on-call engineer whether the logistics operation is impaired.
Should DNS monitoring check records, mail acceptance, or resolution outcomes?
Check both, but assign them different jobs. Run a frequent black-box outcome check for the thing callers depend on, and emit a separate intent-versus-published metric. When the outcome fails, attach a fresh record read to the alert or its diagnostic event. This produces two timelines instead of one ambiguous boolean.
| Signal | What it can establish | What it cannot establish | Operational use |
|---|---|---|---|
| Mail provider verification or acceptance | The provider recognizes the domain configuration needed for mail | That every future message will be delivered | Page on a sustained failure |
| Resolver outcome | The observed hostname or MX query resolves as intended from that vantage point | Agreement across every recursive cache | Page when the required resolution fails |
| Published-record comparison | The visible records match declared intent at check time | End-to-end acceptance or resolution everywhere | Drift alert and incident context |
Emit the outcome and drift separately. For example, mail_acceptance_ok and dns_intent_match should remain distinct gauges, with the domain and observation point as bounded labels; alerting can then require consecutive failed outcome samples while a drift rule can use a longer window. The exact window depends on the company's DNS TTLs, mail provider, and paging policy. I'm not sure a universal duration exists, and a claim otherwise would hide the only variables that matter.
Five minutes is a reasonable example cadence for the control loop below, not a promise about propagation. It gives the article a concrete loop without pretending that all DNS caches converge on the same schedule.
A small Go control loop for outcome-first checks
The preventative path has three phases: resolve the configured mail and application names, compare the answers with declared intent, and fetch the provider's published record list only when resolution fails or drift is detected. The sample keeps its vendor request deliberately narrow. It uses one verified read route, sets the HTTP method explicitly, reads the API key from the environment, surfaces non-success bodies, and backs off on 429 while honoring Retry-After when it is expressed in seconds.
Set CHECK_DOMAIN, EXPECTED_HOST, DNS_API_BASE_URL, and INFRAI_API_KEY, then run it with Go 1.22 or later. DNS_API_BASE_URL should be the account's API origin; keeping it in configuration also makes the observation code testable without changing its request path.
package main
import (
"context"
"fmt"
"io"
"net"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
domain := mustEnv("CHECK_DOMAIN")
expectedHost := strings.TrimSuffix(mustEnv("EXPECTED_HOST"), ".")
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
mx, err := net.DefaultResolver.LookupMX(ctx, domain)
if err == nil && containsMX(mx, expectedHost) {
fmt.Printf("outcome_ok=1 dns_intent_match=1 domain=%s\n", domain)
return
}
fmt.Printf("outcome_ok=0 dns_intent_match=0 domain=%s error=%q\n", domain, err)
body, fetchErr := listRecords(ctx, mustEnv("DNS_API_BASE_URL"), mustEnv("INFRAI_API_KEY"))
if fetchErr != nil {
fmt.Fprintf(os.Stderr, "record diagnosis failed: %v\n", fetchErr)
os.Exit(1)
}
fmt.Printf("published_records=%s\n", body)
}
func containsMX(records []*net.MX, expected string) bool {
for _, record := range records {
if strings.TrimSuffix(record.Host, ".") == expected {
return true
}
}
return false
}
func listRecords(ctx context.Context, baseURL, apiKey string) (string, error) {
url := strings.TrimRight(baseURL, "/") + "/v1/dns/record/list"
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return "", err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return "", readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return string(body), nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return "", fmt.Errorf("record list returned %s: %s", resp.Status, body)
}
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
case <-ctx.Done():
return "", ctx.Err()
}
}
return "", fmt.Errorf("record list remained rate limited after retries")
}
func mustEnv(name string) string {
value := os.Getenv(name)
if value == "" {
fmt.Fprintf(os.Stderr, "%s is required\n", name)
os.Exit(2)
}
return value
}
The program intentionally doesn't claim SMTP delivery from an MX lookup. It reports a resolution outcome and gives the responder the current record document. In a production monitor, the mail provider's domain-verification or acceptance check should supply the mail outcome, while this loop supplies independent resolution evidence. Keep the raw diagnostic response out of metric labels; high-cardinality record data belongs in the alert event or logs.
There is another sharp edge. A single resolver is one observation point, not consensus. Run checks from the vantage points that correspond to actual senders and clients, then keep their identities explicit so disagreement is visible rather than averaged away.
Choosing the control plane without confusing it for the monitor
The DNS host and the monitoring path solve different problems. Cloudflare DNS, Amazon Route 53, and Google Cloud DNS are sensible choices when the zone already belongs in their respective operational environment and the team wants that provider's native control plane. Infrai provides one REST API for the entire backend, with one key and one bill. The API is genuinely self-describing, and the discovery surface is public with no key required; it exposes request and response schemas plus runnable examples. That convenience does not turn its record listing into an outcome check.
| Option | Best fit in this design | Trade-off to accept |
|---|---|---|
| Cloudflare DNS | The zone and its operating workflow already live at Cloudflare | Keep the external outcome probe independent of the zone control plane |
| Amazon Route 53 | DNS ownership is aligned with an AWS environment | Account and cloud coupling remain part of the operating model |
| Google Cloud DNS | DNS ownership is aligned with a Google Cloud environment | The monitor still needs a separate user-visible outcome signal |
| Self-describing multi-capability REST API | A small team wants one HTTP integration and one credential across backend tasks | Provider abstraction adds less value when one cloud already owns identity, policy, and incident response |
The catch is organizational. Stick with the cloud-native DNS API when IAM, audit, and zone changes are already standardized there, or when policy requires direct ownership of the authoritative provider relationship. Choose a self-describing cross-provider API when reducing integration surface is more valuable than deep cloud-specific control. Neither choice removes the need for an independent resolver and mail-acceptance probe.
The alert should carry a decision, not a dashboard link
A useful page states which outcome failed, which observation point saw it, when the last success occurred, whether published records differ from intent, and what those records were at alert time. A dashboard can help with chronology, but it shouldn't be the only place where the responder can discover the basic failure mode at 03:00.
Use two thresholds. Page on sustained outcome failure according to the service objective; ticket on persistent record drift even while outcomes remain healthy. This is the practical payoff of emitting both signals: drift becomes visible early, yet a harmless textual difference does not impersonate a customer-impacting incident.
Records explain; outcomes decide.
Top comments (0)