Hwpgsd503817

Posted on Sep 20

Player Email DNS Automation: Guarding Destructive Changes with Recorded Intent

#dns #sre #email

The page says transactional mail for a live game is failing authentication. Password resets still leave the sender, but recipient systems no longer have the DNS evidence used to evaluate them. The immediate action is to stop further DNS mutation, identify the affected domain, and restore the last reviewed SPF, DKIM, and DMARC records. The durable fix is earlier: a deletion controller must require an exact domain allowlist match, a separately supplied destructive flag, and a durable intent record before it can change anything.

That is the short answer. Treat a mail-authentication domain as production state, not cleanup residue. A successful DNS API response is weak evidence; the useful evidence is that resolvers can retrieve the intended records, messages carry aligned authentication results, and DMARC reports show the expected disposition. The guard protects that evidence chain from a broad cleanup job.

What should have fired before the delivery page?

The late signal is a delivery SLO breach: accepted mail stops reaching the population the service is meant to contact, or authentication-related rejection rises beyond the team's error budget. By then, a player requesting account recovery is already paying for an infrastructure mistake.

A better sequence starts with change intent. Any request to retire a zone or remove the records that establish mail identity should produce an auditable pending action before mutation. The action names one canonical domain, the requested operation, the actor, a reason, and a change identifier. The controller then evaluates policy and performs fresh DNS reads. Only after those checks may it call the provider's delete primitive.

The pre-change alert should fire when declared intent and observed prerequisites disagree. Examples include an unapproved domain, a missing destructive flag, an empty change identifier, or evidence that the domain still publishes mail-authentication records. This alert has close to zero customer distance: it occurs while the operation is still reversible because nothing has happened.

Nothing changed. Good.

Do not reduce that check to strings.HasSuffix(candidate, approvedSuffix). notexample.com has the suffix example.com; textual resemblance is not DNS ancestry. Normalize a name by lowercasing it, removing one trailing dot, converting internationalized labels to their ASCII form, and then compare the result against exact allowlist entries. For a deletion boundary, I prefer exact membership. Permitting every descendant is a separate policy decision and deserves a separate rule.

How should a DNS automation guard authorize a destructive operation?

A destructive request should pass three gates. The allowlist answers where automation has authority. The explicit flag answers whether this invocation knowingly enables destruction. The intent log answers what an operator meant to do and which review trail supports it. None substitutes for another.

A flag alone is dangerous in a reusable shell profile or CI variable. An allowlist alone still lets a faulty loop delete every approved domain. A log written after the API call cannot prove that the recorded request preceded the mutation. Write the pending intent first, require that write to succeed, then mutate, and finally append the outcome.

Consider the awkward retry rather than the clean first attempt. The controller writes a pending intent, sends the mutation, and loses the response when its deadline expires; at that instant it does not know whether the provider accepted the request. Blindly starting over with a new change identifier destroys the audit chain, while marking the first attempt failed asserts something the controller did not observe. The reconciler should retain the original identifier, read current DNS state, and classify the outcome as applied, not applied, or still unknown before choosing another action. If the zone is already absent, an idempotent delete can converge without pretending a second human approved it. If the zone remains but its version changed, the controller should stop for review rather than apply an authorization made against older state. This is why the intent record needs status transitions and observed outcomes, not a single success boolean, and why an append-only history is more useful during an incident than a mutable row containing only the latest answer. The design costs storage and some on-call complexity. It buys a defensible account of what automation knew before each irreversible step.

Retries are policy decisions.

Here is a small policy core. It deliberately does not contain a DNS-provider client; the irreversible adapter belongs behind this decision and should receive only a validated request.

package guard

import (
    "context"
    "errors"
    "fmt"
    "strings"
    "time"

    "golang.org/x/net/idna"
)

type DeleteRequest struct {
    Domain      string
    Actor       string
    Reason      string
    ChangeID    string
    Destructive bool
}

type Intent struct {
    Domain    string
    Actor     string
    Reason    string
    ChangeID  string
    CreatedAt time.Time
}

type IntentWriter interface {
    AppendPending(context.Context, Intent) error
}

type Guard struct {
    Allowed map[string]struct{}
    Log     IntentWriter
    Now     func() time.Time
}

func canonicalDomain(raw string) (string, error) {
    name := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(raw)), ".")
    ascii, err := idna.Lookup.ToASCII(name)
    if err != nil {
        return "", fmt.Errorf("invalid domain: %w", err)
    }
    if ascii == "" || strings.Contains(ascii, "..") {
        return "", errors.New("invalid empty label")
    }
    return ascii, nil
}

func (g Guard) Authorize(ctx context.Context, req DeleteRequest) (string, error) {
    if !req.Destructive {
        return "", errors.New("destructive flag is required")
    }
    if strings.TrimSpace(req.ChangeID) == "" || strings.TrimSpace(req.Reason) == "" {
        return "", errors.New("change ID and reason are required")
    }

    domain, err := canonicalDomain(req.Domain)
    if err != nil {
        return "", err
    }
    if _, ok := g.Allowed[domain]; !ok {
        return "", fmt.Errorf("domain %q is not allowlisted", domain)
    }

    intent := Intent{
        Domain: domain, Actor: req.Actor, Reason: req.Reason,
        ChangeID: req.ChangeID, CreatedAt: g.Now().UTC(),
    }
    if err := g.Log.AppendPending(ctx, intent); err != nil {
        return "", fmt.Errorf("record pending intent: %w", err)
    }
    return domain, nil
}

The example rejects closed when the intent store is unavailable. That choice adds operational friction during an outage, but allowing deletion without the audit prerequisite creates a much larger recovery problem. Make the provider call idempotent at the adapter boundary as well: retries after an ambiguous timeout must converge on the requested final state, while each attempt and observed result remains linked to the same change identifier.

Authorization can also become stale between validation and mutation. Keep the interval short, attach the provider operation to the same immutable request, and use a provider-supported precondition or version token when one exists. If the interface offers no conditional mutation, serialize destructive work per domain and read the current state immediately before the call.

Instrument the evidence chain, not merely the API call

SPF, DKIM, and DMARC contribute different evidence. SPF authorizes sending hosts for a domain. DKIM binds a message to a cryptographic signature and a selector published in DNS. DMARC evaluates identifier alignment and publishes policy plus reporting instructions. RFC 7489 also makes an operational point that matters here: receivers apply DMARC to messages, while aggregate reports give domain owners feedback about observed authentication results.

The dashboard needs layers. Record the controller decision and reason, intent-log latency and failure, mutation outcome, authoritative DNS observations, recursive-resolution observations, message authentication results, and DMARC aggregate trends. A green delete request cannot make the panel green by itself. DNS publication and actual mail observations are separate stages with different failure modes.

Capacity planning matters even for a rare operation. A fleet-wide retirement job can generate reads against authoritative servers, writes to the intent store, and provider requests in a burst. Bound concurrency per zone, apply backoff to transient failures, and budget enough intent-store capacity that the safety path does not become the reason operators bypass the controller. The destructive rate should normally be tiny, which also makes a sudden rise a useful anomaly signal.

Keep that path boring.

For the gaming workload, use a canary message stream that exercises the same authenticated path as password resets without containing player data. Capture Authentication-Results at a mailbox under your control, correlate it with the deployed selectors and DMARC aggregate reports, and page on sustained loss of expected authentication rather than on a single recursive lookup. This closes the gap between configuration existence and deliverability evidence. It does not claim that authentication alone guarantees inbox placement; receiver policy can use other signals.

Which ownership model fits the on-call team?

The decision is less about feature count than failure ownership. A managed control plane can reduce adapter maintenance, while an internal controller can make policy and audit integration easier to inspect. Manual changes avoid controller code but scale poorly as repeatable, reviewable operations.

Approach	What the team owns	Main on-call exposure	Lock-in boundary	Best evidence to demand
Managed DNS automation	Policy configuration, credentials, and integration	External control-plane dependency plus local misuse	Provider API and resource model	Exportable intent history, conditional-change semantics, DNS observations
Internal controller over provider APIs	Policy, code, storage, adapters, and upgrades	Entire decision and execution path	Adapter interfaces and provider-specific preconditions	Tests for deny paths, append-only intent records, reconciliation metrics
Human-reviewed manual changes	Runbooks, access control, and reviewer availability	Operator error and slow recovery	Console or CLI workflow	Review record, before-and-after record sets, resolver checks

I would set an SLO for the guard itself around correct rejection and auditable completion, not raw request throughput. The deny path is part of the product. Test it with missing flags, Unicode and trailing-dot forms, adjacent suffixes, stale approvals, duplicate retries, and intent-store failure. Then exercise restoration from the stored pre-change record set; a backup that has never been restored is an assumption.

The threshold has an error budget too

A page on every failed DNS lookup will train the on-call engineer to mute the alarm. Recursive resolvers cache answers, observations can differ during a controlled change, and one probe is not delivery evidence. Require multiple independent observations over a defined window for the delivery page, while keeping policy violations such as an unallowlisted destructive request as immediate, non-customer-impacting security events.

The exact window cannot be universal. Derive it from the DNS TTLs you publish, the cadence of your probes and DMARC reports, and the maximum interruption the password-reset SLO permits. Review it after planned changes. A threshold longer than the user-facing error budget is useless; a threshold shorter than normal propagation behavior produces false positives and burns the attention needed for a real authentication loss.

That is the closing trade-off: early policy denials should be sensitive because they stop before mutation, while delivery paging should require corroborated evidence. Conflating the two either weakens the deletion guard or makes the on-call rotation absorb noise. Keep both signals, give them different urgency, and make the path from recorded intent to observed mail authentication queryable by one change identifier.

DEV Community