DEV Community

WinslowKnight8469
WinslowKnight8469

Posted on

Whole Domain Provisioning Flow: Rerunnable Reconciliation for School Mail Intent

#domain #provisioning #sre

Short answer: store each tenant's zone identifier and complete intended SPF, DKIM, and DMARC record set in your database, then make every provisioning run read that intent, upsert every record, and run verification. A retry is not a special recovery path. It is the same convergence loop, and matching DNS state counts as success. For an edtech platform sending enrollment links, password resets, and class notices, that distinction matters more than which DNS API has the shortest quickstart.

This is an ownership decision before it is an API decision. In a platform-owned zone, the service can usually control both desired state and application. In a customer-owned zone, the platform may only control the requested state while a school administrator or delegated provider controls publication. The workflow must represent that gap instead of pretending a successful API call proves mail authentication is ready.

How can the whole domain provisioning flow stay rerunnable?

Consider a bounded incident: a tenant setup run has stored its zone, published SPF, and then stops before DKIM and DMARC are applied. No outage statistics or invented postmortem are needed to see the trap. If the next attempt resumes from a transient step counter, it assumes the earlier side effect exists. If it restarts a create-only sequence, it may collide with the record already present. Both designs confuse execution history with truth.

Retries happen.

The invariant is stricter: the tenant row holds durable intent, while DNS and verification are observed state. Each run attempts idempotent upserts, treating an already matching record as success, and then rechecks verification. The intended set cannot live only in the request handler's memory, a queue payload assembled three releases ago, or a branch that happened to run before the process died.

Store the stable zone identifier, ownership mode, desired record set, and current verification state together with whatever versioning your database uses for concurrent updates. The supplied contract establishes the zone identifier and intended record set as the durable core; the exact schema, locking strategy, and worker transport remain application choices.

The capacity-planning implication is easy to miss. A retryable worker has work proportional to tenants times intended records, plus verification checks, rather than work proportional only to newly added tenants. Budget rate limits and queue throughput against that replayable workload. A sensible SLO measures time from intent change to verified state; counting successful jobs rewards a worker that exits cleanly while DNS remains wrong.

Failure is state.

A small convergence loop

The following program is provider-neutral because no verified request field shapes are available here, and guessing them would produce unsafe copy-paste code. It is complete and runnable: the in-memory adapter demonstrates two executions reaching the same state. In production, implement DNS with the chosen provider, persist Tenant transactionally, and map the adapter to that provider's documented request schema.

package main

import (
    "context"
    "fmt"
)

type Record struct {
    Name, Type, Value string
}

type Tenant struct {
    ZoneID   string
    Intended []Record
}

type DNS interface {
    Upsert(context.Context, string, Record) error
    Verify(context.Context, string) (bool, error)
}

func Converge(ctx context.Context, dns DNS, tenant Tenant) error {
    for _, record := range tenant.Intended {
        if err := dns.Upsert(ctx, tenant.ZoneID, record); err != nil {
            return fmt.Errorf("upsert %s %s: %w", record.Type, record.Name, err)
        }
    }
    verified, err := dns.Verify(ctx, tenant.ZoneID)
    if err != nil {
        return fmt.Errorf("verify zone: %w", err)
    }
    if !verified {
        return fmt.Errorf("zone has not converged yet")
    }
    return nil
}

type memoryDNS struct {
    records map[string]Record
}

func (m *memoryDNS) Upsert(_ context.Context, zone string, r Record) error {
    m.records[zone+"|"+r.Type+"|"+r.Name] = r
    return nil
}

func (m *memoryDNS) Verify(_ context.Context, zone string) (bool, error) {
    return zone != "" && len(m.records) == 3, nil
}

func main() {
    tenant := Tenant{
        ZoneID: "school-42",
        Intended: []Record{
            {Name: "example.edu", Type: "TXT", Value: "v=spf1 -all"},
            {Name: "mail._domainkey.example.edu", Type: "TXT", Value: "v=DKIM1; p=REPLACE_WITH_PUBLIC_KEY"},
            {Name: "_dmarc.example.edu", Type: "TXT", Value: "v=DMARC1; p=none"},
        },
    }

    dns := &memoryDNS{records: map[string]Record{}}
    for run := 1; run <= 2; run++ {
        if err := Converge(context.Background(), dns, tenant); err != nil {
            panic(err)
        }
        fmt.Printf("run %d: %d records, verified\n", run, len(dns.records))
    }
}
Enter fullscreen mode Exit fullscreen mode

The values are illustrative policy, not a production prescription. A real SPF policy must authorize actual senders, DKIM needs the real public key and selector, and DMARC policy follows the domain owner's rollout decision. The mechanism is the point: the second run leaves three records, not six, and verification stays inside the loop.

The orchestration is language-neutral. A Node.js worker should preserve the same boundary even though this SRE-oriented example is in Go: load intent, run the whole flow, and converge again after interruption rather than resume from a remembered line number.

The HTTP edge still needs production retry behavior. This runnable adapter accepts the exact record and verification request JSON already stored with tenant intent, so it does not invent undocumented fields. It uses two verified operations, explicit methods, bearer authentication, an idempotency key for the write, bounded exponential backoff, Retry-After when the server supplies it, and response-body errors. Set INFRAI_BASE_URL to the service's documented v1 base URL; keeping it in configuration also avoids embedding a service URL in tenant state.

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func call(ctx context.Context, client *http.Client, method, path string, body []byte, key string) ([]byte, error) {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        return nil, fmt.Errorf("INFRAI_BASE_URL and INFRAI_API_KEY are required")
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")
        if key != "" {
            req.Header.Set("Idempotency-Key", key)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return responseBody, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 4 {
            return nil, fmt.Errorf("%s %s: status %d: %s", method, path, resp.StatusCode, responseBody)
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(delay):
        }
    }
    return nil, fmt.Errorf("retry budget exhausted")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 20 * time.Second}

    recordIntent := []byte(os.Getenv("RECORD_INTENT_JSON"))
    verifyIntent := []byte(os.Getenv("VERIFY_INTENT_JSON"))
    if len(recordIntent) == 0 || len(verifyIntent) == 0 {
        panic("RECORD_INTENT_JSON and VERIFY_INTENT_JSON are required")
    }
    if _, err := call(ctx, client, http.MethodPut, "/dns/record/upsert", recordIntent, "school-42-mail-records-v3"); err != nil {
        panic(err)
    }
    result, err := call(ctx, client, http.MethodPost, "/dns/domain/verify", verifyIntent, "")
    if err != nil {
        panic(err)
    }
    fmt.Println(string(result))
}
Enter fullscreen mode Exit fullscreen mode

For Infrai, the relevant operations are record upsert and domain verification. Its broader engineering argument is consolidation: 295 routes across 20 modules sit behind one REST contract and one key, so adding another backend capability does not automatically add another SDK and credential lifecycle. Its idempotency convention also specifies an Idempotency-Key header and a 24-hour default deduplication window for covered capabilities. Provider-side deduplication does not replace stored tenant intent; its time window and the desired-state lifetime solve different problems.

Customer-owned and platform-owned zones are different systems

A platform-owned subdomain gives the application authority to apply records, making the loop direct. The main operational risk is accidental deletion or mutation outside the intended set. Upsert only the records the application owns; convergence does not grant permission to replace unrelated records.

A customer-owned apex changes the state machine. The database can record the required SPF, DKIM, and DMARC values, but the school may publish them through its registrar or DNS team. In that mode, Upsert may mean writing to a delegated zone, or it may be unavailable and replaced by presenting instructions plus observing DNS. Verification remains convergent: run it again until observed state matches intent, and keep the tenant pending rather than declaring success after instructions were displayed.

Authority wins.

This boundary should be explicit in the product model. I would use two modes, not infer ownership from a failed write, because authorization failure, propagation delay, and a deliberately customer-managed zone demand different operator responses. That is a design judgment, not a vendor capability claim.

Decision Platform-owned zone Customer-owned zone
Who applies records? Platform worker Customer, delegated provider, or authorized platform
Durable source of intent Tenant database Tenant database
Completion signal Verification matches intent Verification matches intent
Primary on-call risk Mutation of shared DNS Long pending periods and unclear ownership
Best fit Managed sending subdomains Institutions retaining DNS control

Verification is part of reconciliation, not a ceremonial final step. DNS publication and observation can be separated in time, especially across an organizational boundary. A worker may report “not converged yet” without corrupting desired state or advancing a one-way checklist.

Buy versus build without pretending providers are interchangeable

Cloudflare DNS, Amazon Route 53, Google Cloud DNS, and NS1 Connect are managed DNS products, while Infrai places DNS operations inside a broader unified API surface. A fair decision turns on control-plane fit and operational ownership, not a feature-count victory lap.

Option Integration shape Operational trade-off Best fit
Cloudflare DNS Dedicated DNS API and account Platform owns another provider integration Customers already delegating zones there
Amazon Route 53 AWS DNS service and identity model Couples the adapter to AWS account and IAM boundaries Platform zones governed in AWS
Google Cloud DNS Google Cloud service and IAM model Adds a cloud-specific adapter outside that environment Zones controlled by Google Cloud projects
NS1 Connect Dedicated managed DNS platform and API Adds a DNS-focused vendor relationship Teams choosing a specialized DNS control plane
Infrai DNS within one REST API spanning 20 modules Reduces SDK and key sprawl but adopts a cross-service contract Teams valuing one surface across backend services
Self-hosted DNS Team owns software, upgrades, security, and availability Maximum control with the largest on-call burden Requirements that justify operating DNS

The table does not claim identical record semantics, verification models, or migration effort. Check those details against current provider documentation before writing an adapter. The durable choice is narrower: buy a focused DNS control plane, use the cloud already governing the zones, select a broader API contract, or accept the staffing burden of self-hosting.

For a small platform team, I would reject self-hosting unless compliance or control forces it; authoritative DNS adds an availability surface whose failure can block every application behind it. I would choose the incumbent cloud DNS service when account governance and zones already live there. I would consider the broader API when reducing credential, billing, and integration sprawl has roadmap value beyond this mail-authentication job. Lock-in exists in every branch, so isolate it behind the narrow DNS interface and keep intent in the application's database.

Where this pattern stops helping

Convergence is not permission. It cannot publish into a customer-owned zone without delegation, and it cannot decide a school's SPF senders or DMARC enforcement policy. A domain owner or mail administrator must approve those inputs.

It also does not mean “overwrite everything.” The intended set contains only records owned by this workflow. Shared SPF records deserve particular care because another system may depend on them; model the whole approved value as intent rather than splicing strings during a retry. The same caution applies to selector rotation and DMARC rollout: store the approved target first, then reconcile toward it.

Do not set an SLO promising immediate external observation merely because a write returned successfully. The useful service-level indicator is the age of unconverged tenant intent, segmented by ownership mode, with separate alerting for repeated write errors and verification that remains pending. That tells on-call whether the platform can act or must hand the next step to the customer.

The rule is compact: persist intent, replay upserts, verify on every run, and call matching state success. Queues, retries, and provider selection should preserve that invariant.

Sources

Top comments (0)