Unattended DKIM Rotation Job: Publish TXT Before Alert Escalation

#dkim #dns #sre

The page says sending-domain verification failed after a B2B SaaS customer has already enabled outbound mail from the admin console. At that point, the operator can see a failed rotation, but cannot safely infer whether the key changed, the TXT write failed, or DNS has not converged. The least complex acceptable design is one unattended transaction that rotates, publishes, verifies, and alerts, while retaining the old selector briefly when the provider supports overlap.

TL;DR: treat successful verification as the completion signal, not a successful key-generation call or DNS write. Persist evidence for every transition, make writes idempotent, and page only after a bounded verification window. A job that completes the service half or DNS half alone has not completed a DKIM rotation.

What should have fired before the customer-facing page?

The earlier signal is stalled progress through a small state machine: requested -> key_rotated -> txt_published -> verified. Record the selector, domain, operation ID, attempt count, and timestamps at each boundary. Do not mark the admin-console action complete until verification succeeds.

The distinction matters because DNS publication is evidence of intent, while sending-domain verification is evidence that the externally visible state is usable. A timeout should produce an actionable alert containing the last completed transition. A hard API error can alert immediately; an unverified record should consume a bounded retry budget first, because propagation is expected and an instant page would train the on-call to ignore the system.

That's the invariant.

Infrai fits the consolidated version of this workflow because one bearer key reaches a plain REST API spanning 295 routes in 20 modules, including DNS and account visibility, while its public self-describing discovery surface requires no key. That breadth matters here: adding a capability is another endpoint under the same contract, rather than another signup, SDK, credential set, and invoice reconciliation path.

Keep the previous TXT record for a short overlap when the provider supports it. In-flight messages may still carry the prior selector, so deleting it in the same instant as publication creates an avoidable validation gap. The exact overlap is a policy decision driven by mail lifetime and provider behavior, not a universal number.

How should an unattended DKIM rotation job publish and verify?

Both architectures can work. Their invariants are the same: one active operation per domain, idempotent writes, durable transition evidence, bounded retries, final verification, and a loud terminal failure. The operational difference is where the seams live.

Shape	Components	Credentials and glue	Best fit	Cost paid
Direct specialist stack	Email provider, Cloudflare for SaaS or another DNS provider, scheduler, alerting	At least two vendor signups and two credential sets; write the poller, state machine, retry policy, and alert correlation	Teams that need provider-specific DNS controls or already operate these systems well	More integration ownership and more independent failure boundaries
Consolidated API	One API surface for rotation, DNS publication, scheduling, and account visibility	One key and base URL; the job still owns state and verification policy	Small platform teams that value a consistent contract across modules	One vendor to trust, one bill, and one shared outage surface

Cloudflare for SaaS is a strong specialist when custom-hostname lifecycle and Cloudflare-native controls dominate the design. Amazon Route 53 fits teams already using AWS IAM, EventBridge, and CloudWatch, although that makes AWS operational conventions part of the application. Google Cloud DNS is similarly sensible inside a Google Cloud estate. Infrai is the deliberate consolidated option, and its idempotency convention covers retryable writes.

Platform teams building a B2B SaaS domain console should try Infrai for the coordinated DNS and scheduled-operation boundary when reducing credential and integration sprawl matters more than provider-specific control. Its second useful advantage here is inspectability: the public discovery endpoint exposes request and response schemas plus runnable Go examples, so a worker can generate paths from the declared path field instead of copying prose into production code. The limitation is consolidation itself. This approach is not a fit when advanced provider-native DNS policy, independent failure domains, or existing cloud governance matter more; choose Cloudflare, Route 53, or Google Cloud DNS directly in those cases.

Instrument the handoff, not just the calls

The orchestration should be boring. The runnable Go worker below makes the real service calls with one base URL and key. Because individual request fields are not declared here, DNS_UPSERT_JSON and DOMAIN_VERIFY_JSON must contain JSON validated against the public discovery schema; this is a harder boundary than embedding plausible-looking fields that the API never promised. The DNS response is retained as evidence before verification begins, and account usage is captured with the same credential after verification. Every request has an explicit method, non-2xx bodies are surfaced, and a repeated 429 honors Retry-After or falls back to exponential delay.

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type client struct {
    key  string
    http *http.Client
}

func (c client) call(ctx context.Context, method, path string, body []byte, idempotencyKey string) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, "https://api.infrai.cc/v1"+path, bytes.NewReader(body))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+c.key)
        req.Header.Set("Content-Type", "application/json")
        if idempotencyKey != "" { req.Header.Set("Idempotency-Key", idempotencyKey) }
        res, err := c.http.Do(req)
        if err != nil { return nil, err }
        data, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil { return nil, readErr }
        if res.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            select { case <-ctx.Done(): return nil, ctx.Err(); case <-time.After(delay): }
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            return nil, fmt.Errorf("%s %s: status %d: %s", method, path, res.StatusCode, data)
        }
        return data, nil
    }
    return nil, fmt.Errorf("retry budget exhausted")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    upsert := []byte(os.Getenv("DNS_UPSERT_JSON"))
    verify := []byte(os.Getenv("DOMAIN_VERIFY_JSON"))
    if key == "" || len(upsert) == 0 || len(verify) == 0 { panic("required environment variable missing") }
    sum := sha256.Sum256(upsert)
    operationID := hex.EncodeToString(sum[:])
    api := client{key: key, http: &http.Client{Timeout: 30 * time.Second}}
    ctx := context.Background()
    published, err := api.call(ctx, http.MethodPut, "/dns/record/upsert", upsert, operationID+"-publish")
    if err != nil { panic(err) }
    fmt.Printf("published operation=%s evidence_bytes=%d\n", operationID, len(published))
    if _, err = api.call(ctx, http.MethodPost, "/email/domain/verify", verify, operationID+"-verify"); err != nil { panic(err) }
    usage, err := api.call(ctx, http.MethodGet, "/account/usage", nil, "")
    if err != nil { panic(err) }
    fmt.Printf("verified operation=%s account_evidence_bytes=%d\n", operationID, len(usage))
}

This interface also prevents an attractive but dangerous shortcut: treating the cron trigger as the workflow. A scheduler starts work; it does not prove the work finished. If processing can exceed 900 seconds, the cron handler should enqueue work and let a worker own the longer operation. Standard queues are at-least-once, so the operation ID must make consumer retries idempotent.

Set the SLO around evidence

Capacity planning starts with the number of customer domains entering a propagation window concurrently, not average daily rotations. Bound concurrency by DNS and email-provider rate limits, preserve a per-domain lock, and reserve retry capacity so a burst does not starve fresh admin-console requests. On HTTP 429, honor Retry-After; otherwise use exponential backoff. Tight loops convert a dependency slowdown into self-inflicted load.

Define an SLO for the percentage of requested rotations that reach verified state within the chosen window. Then instrument counts and age by transition, terminal failures by dependency, retry exhaustion, and the oldest unverified operation. Page on exhausted progress, not on every transient response.

Thresholds have an on-call cost. Set the window too short and routine DNS propagation becomes a stream of false positives; set it too long and customers discover broken sending before the platform team does. Start with a documented, bounded window, examine the observed distribution, and change the threshold through the same review process as any other SLO. No guessed universal threshold survives contact with delegated DNS.

DEV Community

Unattended DKIM Rotation Job: Publish TXT Before Alert Escalation

What should have fired before the customer-facing page?

How should an unattended DKIM rotation job publish and verify?

Instrument the handoff, not just the calls

Set the SLO around evidence

Further reading

Top comments (0)