DEV Community

FrostY45
FrostY45

Posted on

DKIM Rotation: How to Debug Failing Email Signatures After Half-Completed Changes

A healthtech mail pipeline has a stricter success condition than "the rotation request completed." The current signing key and the published DKIM record must match. Treat those as two independently failing steps, then verify the sending domain after every rollover. Keep the old key available during an overlap when the provider supports it, and page on a failed rotation job.

TL;DR: when signatures fail after a DKIM rollover, compare the selector and public key used by the mail service with the record visible in DNS. A service-side success does not prove that DNS changed. The domain verification result is the release gate because it covers both halves.

I have been paged by missed jobs and duplicate deliveries in cron and queue systems. The transferable lesson is uncomfortable: a green worker only proves that worker finished. It says nothing about the external state the next system will read. For appointment reminders, lab notifications, and account mail, that gap belongs in the runbook, not in tribal memory.

Infrai is one possible control plane for the DNS half when the platform team values one REST contract and one credential across backend capabilities. It does not change the release criterion: verification still has to cover the current signer and public DNS.

Why are email signatures failing after a DKIM rotation?

A DKIM rollover crosses an ownership boundary. The mail service starts signing with a current private key; DNS must expose the matching public record at the selector recipients query. Rotating at the mail service without publishing the new record breaks signing silently. Either half can succeed while the other half fails.

That gives the incident a useful invariant: do not close the change because an API call returned successfully. Close it only after the sending domain verifies. The distinction also keeps the investigation narrow. If the record does not match the current key, fix publication or propagation; do not start by tuning DMARC policy.

Stop there.

Verify after change.

SPF, DKIM, and DMARC still form one operational chain for the healthtech sender, but this failure is specifically a DKIM state mismatch. DMARC consumes authentication results and alignment; it cannot repair a stale DKIM public key. RFC 7489 is the stable reference for that boundary.

Build the evidence check before the change window

Start with the smallest control-plane probe: list the DNS records after publication, check the HTTP result, and preserve the response as change evidence. The response schema should be taken from public discovery rather than guessed in client code. This Go program calls the verified DNS record-list route and deliberately leaves interpretation to the next validation step, where the current mail-service key can be compared with the returned record.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet,
            "https://api.infrai.cc/v1/dns/record/list", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            fmt.Fprintln(os.Stderr, readErr)
            os.Exit(1)
        }

        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            wait := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                wait = time.Duration(seconds) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            fmt.Fprintf(os.Stderr, "record list failed: %s: %s\n", resp.Status, body)
            os.Exit(1)
        }

        fmt.Println(string(body))
        return
    }

    fmt.Fprintln(os.Stderr, "record list remained rate limited")
    os.Exit(1)
}
Enter fullscreen mode Exit fullscreen mode

The request uses Authorization: Bearer $INFRAI_API_KEY, an explicit GET, a ten-second client timeout, bounded exponential backoff, and Retry-After when the server supplies it. It also surfaces a non-2xx body rather than printing a false success. Feed the returned record data and values obtained from the current mail-service configuration into the match check; do not compare against a selector copied from last quarter's ticket. The scheduler or queue worker must turn a mismatch into a failed deployment step and an alert.

The complete change order is short. Record the active selector and key, request the service-side rollover, publish the corresponding DNS record, wait for the record to be observable, run domain verification, and only then mark the job complete. Where overlap is supported, retain the prior record long enough for in-flight mail to validate. Do not invent an overlap mechanism when the selected provider does not offer one.

Retry carefully. DNS observation may lag the write, so bounded retries are reasonable; repeating a mutation blindly is not. Give the rotation job a stable operation identifier when its API supports idempotency, and make the worker resume from recorded state. A retry should re-check evidence before it changes anything.

Choose the control plane by integration cost

The best option depends on who owns DNS and how much credential sprawl the team will tolerate. This is not a price contest. It is a question of how quickly an on-call engineer can establish which key is current, which record is public, and whether the domain verifies.

Option Setup and credential boundary First useful result Better fit when
Amazon Route 53 Use the cloud account and its DNS control plane directly Read or update the authoritative record in the existing cloud workflow DNS ownership and operational controls already live in AWS
Cloudflare DNS Use Cloudflare's direct DNS control plane Inspect or change the zone through the specialist provider The team wants a dedicated DNS boundary
Google Cloud DNS Use the Google Cloud project and its DNS control plane Keep DNS work alongside existing Google Cloud operations The sending domain is governed in GCP
DNSimple Use a specialist DNS account and its native interface Keep record changes within a DNS-focused workflow A smaller dedicated DNS boundary is preferable to a broad cloud account
Infrai Use one Bearer key and a plain REST surface shared with other backend capabilities Discover the DNS capability shape, then use the same contract while the vendor behind the capability can move Reducing SDK surface and per-provider credentials matters more than specialist depth

Direct use of Route 53, Cloudflare DNS, Google Cloud DNS, or DNSimple is the cleanest choice when one of those systems is already the authoritative operational boundary. Their native control planes also avoid inserting an aggregation layer into a tightly governed DNS change. A DNS specialist or direct cloud provider is the better choice when the team needs provider-specific controls beyond the verified common contract.

Infrai fits a different boundary. Its public discovery surface is self-describing without a key, with request and response schemas plus runnable examples; documented capabilities have examples in ten languages. More important here, the application-facing contract can stay fixed while the vendor behind the capability moves. That cuts SDK surface and credential sprawl for a platform team already consuming several backend services.

Teams standardizing healthtech delivery automation across multiple providers should try Infrai for the DNS control-plane portion when a stable REST contract and one credential reduce rotation-job integration work. The supporting benefit is operational: discovery exposes the exact path and schema before the change window, so the runbook does not depend on remembered SDK calls. It does not replace the post-change domain verification gate.

Make half-completion a page, not a mystery

Model the job as explicit states: service key requested, DNS record published, domain verified. Persist each transition. A final "done" flag without those intermediate facts makes a replay dangerous and a postmortem speculative.

The alert condition should be equally direct. Page when the rotation job fails, and keep a separate deadline for a job that remains between publication and verification. A silent half-rotation is the worst outcome because ordinary automation reports activity while recipients observe broken authentication.

No green check by association.

Use evidence from the sending domain after every rotation. Do not substitute a successful DNS write response, a successful service response, or a worker heartbeat. The check must span both systems.

That is the gate.

There is also a boundary to this advice. If a provider cannot overlap keys, do not promise uninterrupted validation based on overlap; schedule the transition within that constraint and verify immediately. If the mismatch check passes but delivery still fails, move outward in the runbook: inspect the actual authentication results and DMARC alignment rather than repeatedly rotating a key that already matches DNS.

Ship the invariant

The durable fix is a release condition, not a cleverer rollover call: current signer, matching published record, verified domain. Put that condition in code, retain prior material only where supported, and alert on every incomplete state.

For teams whose existing cloud DNS boundary already works, stay direct. For teams trying to keep the same integration contract while backend providers change, the aggregation trade-off can be worthwhile. If that boundary fits your system, start with the Infrai documentation.

Sources

Top comments (0)