DEV Community

NikitaChristensen2691
NikitaChristensen2691

Posted on

Unattended DKIM Rotation Jobs Explained for Node.js TXT Verification and Alerts

#dkim #dns #node #sre

Unattended DKIM Rotation Jobs Explained for Node.js TXT Verification and Alerts

Short answer: Treat DKIM rotation as an idempotent reconciliation loop: record intent, publish one TXT value, verify what DNS exposes, and alert on drift before a logistics tenant sends mail.

The page that arrives too late

The page says a tenant's sending domain is failing authentication. The dashboard shows a healthy rotation timestamp. That mismatch is the incident: the database says selector s2026a is active, while recursive DNS still answers with yesterday's key, or answers nothing at all.

I start by reading the two records side by side. A rotation worker can succeed at writing its own state and still fail to make the published world agree. DNS caching, an incorrect zone, an overwritten TXT value, and a selector typo all create the same operational symptom. The first useful signal is therefore not “rotation ran”; it is “the expected TXT set is visible from the resolver path we use for checks.”

It drifted.

The useful investigation is a timeline, not a hunch. Suppose tenant north-417 was scheduled at 02:00 UTC. The worker wrote s2026a to its intent table, the authoritative zone accepted a TXT value at 02:01, and a recursive probe at 02:03 still returned s2025z. At 02:31, a second deployment replaced the record with a truncated string. If the alert only measures job completion, all three events look healthy. If it stores the desired hash, observed chunks, resolver identity, and timestamps, the on-call can see exactly where intent diverged from publication and whether the right response is to wait, republish, or stop retirement. That evidence also makes a postmortem concrete: the question becomes which transition lacked an observation, not which engineer happened to be awake.

The alert should fire on age and disagreement. For example, page when a selector has been in publishing for 30 minutes, or when three consecutive probes disagree. Those numbers are policy choices, not standards. Your mileage may vary: measure normal DNS propagation in each managed zone before choosing a window.

How should an unattended Node.js job rotate, publish TXT, verify a sending domain, and alert?

Keep the state machine boring: planned, publishing, verified, then retiring. Generate a new selector before touching the old one. Publish the new selector._domainkey.tenant.example TXT record, verify the exact value through DNS, and only then make the selector eligible for signing. Retire the old selector after the longest message and cache lifetime your mail path permits.

The worker must be idempotent. A retry for the same rotation ID should produce the same desired record and should not delete a selector that is still referenced by an in-flight message. Store a hash of the intended TXT value, the observed value, probe timestamps, and the actor that changed the record. That gives the on-call engineer a timeline instead of a guess.

Here is a deliberately small interface. The DNS client and notifier are adapters around your chosen authoritative DNS and alerting systems; their contracts are the part worth testing.

type DNS interface {
    PublishTXT(ctx context.Context, name, value string) error
    LookupTXT(ctx context.Context, name string) ([]string, error)
}

func reconcile(ctx context.Context, dns DNS, notify func(string), name, wanted string) error {
    if err := dns.PublishTXT(ctx, name, wanted); err != nil {
        notify("DKIM publish failed for " + name)
        return err
    }
    got, err := dns.LookupTXT(ctx, name)
    if err != nil || !contains(got, wanted) {
        notify("DKIM TXT drift for " + name)
        return fmt.Errorf("verification did not observe the intended TXT value")
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

The production version records an attempt before each side effect, uses bounded retries with jitter, and emits a metric for intent_observed_delta_seconds. Never turn a failed lookup into an empty desired value; that converts a transient observation problem into destructive drift.

What can make a verified selector drift later?

Verification is a snapshot. A later zone edit can replace the value, and a resolver can retain an older answer until its TTL expires. Query at least two resolver paths when the risk justifies it, but label the result clearly: authoritative answers establish publication, while recursive answers establish what senders may currently see.

TXT is a set of character strings, not a promise that every library returns one identical string. Normalize quoting and concatenate chunks according to the DNS library's documented behavior before comparing. Compare the DKIM key material, not incidental presentation.

Standards define the surrounding signals. DMARC alignment and reporting are described in RFC 7489, while DKIM itself uses a selector-specific DNS name. They do not define your rotation schedule, your alert threshold, or your rollback policy. Those belong in a runbook reviewed with the mail and DNS owners.

Choosing thresholds without creating alert fatigue

A page on every propagation delay trains people to ignore pages. A page only after a tenant's mail is rejected is useless. Start with a warning for an extended publishing state, a critical alert for repeated mismatch after the measured propagation budget, and a separate alert when the old selector approaches retirement without a verified replacement.

Signal Action Why
Intent and authoritative TXT disagree Open an incident Publication drift is actionable
Recursive answer is old but authoritative answer matches Warn and recheck Caches may still be converging
Replacement is verified and old selector is still in use Block retirement In-flight mail may need the old key

The trade-off is explicit: shorter windows reduce exposure but increase false positives; longer windows protect quiet operations but leave more time for drift. This workflow is not suitable when tenants control arbitrary DNS providers and cannot expose a verifiable change path. In that case, keep the state as pending, give the tenant a concrete TXT challenge, and stick with manual confirmation or a provider-specific integration.

After an incident, replay the exact desired record against a staging zone and test duplicate deliveries, worker restarts, and clock skew. The useful postmortem question is not “why did DNS fail?” It is “which observation did we trust, and why did it disagree with intent?”

References

Further reading

Top comments (0)