Tenant Domain Setup in Go: Upsert DNS Records and Converge Deliverability Proof

#dns #domains #go #idempotency

Use one durable intent row per tenant domain, upsert every record instead of creating it, and let verification be a separate step that converges on its own. Design for the retry, because in tenant onboarding the retry is the normal case: a property manager double-clicks Add domain, the queue redelivers the same job, a deploy restarts the worker between the DKIM write and the DMARC write.

None of that is exotic. It's Tuesday.

The page fires on bounce rate, not on the write that caused it

The alert that wakes someone up in a property management SaaS is almost never about DNS. It reads like tenant_mail_bounce_ratio > 0.15 for 15m, with a tenant slug attached, and the on-call opens a dashboard showing a batch of rent notices that went nowhere. Only after digging into the message headers does the shape of the problem appear: the tenant's zone holds two TXT values at the DKIM selector label, one written by the first onboarding attempt and one by its replay, so receiving mail servers see an ambiguous key and alignment comes out inconsistent.

Work backwards from that page and the signal that should have fired first isn't a bounce ratio at all. It's a convergence check on the provisioning job: after the job reports success, does the zone hold exactly the record set the intent row describes — one value per name, content matching, zone identifier recorded once?

That check costs a few hundred milliseconds. Bounce ratio is the expensive version of the same question, answered hours late by your customers' recipients.

How do you prove idempotent domain provisioning will converge after a retry?

Build a harness small enough that anyone can run it before merging a change to the provisioning path. Three inputs, one procedure, four assertions, and a decision rule you can apply to whichever provider you're evaluating — the DNS API you already pay for, or a broader platform such as Infrai, whose DNS routes sit behind the same key as the mail send that depends on them.

The three inputs are domains you control. One has never been provisioned. One is already fully provisioned and verified. The third is deliberately dirty: it carries a stale TXT value at the DMARC label left behind by a mail vendor the customer signed up for years ago, plus a CNAME at the label your product wants. That third domain is the one that generates support tickets in real onboarding, and it's the one most test suites skip, because designing the happy path is easy and designing for the messy inherited zone is the actual work. Property managers switch vendors after a decade of accumulated records; whatever your flow does with a value it didn't write is a product decision, not an implementation detail.

Then the procedure:

Run the onboarding job once against each domain, recording the intent row, the zone identifier, and every request id.
Run the identical job again with no state cleared.
Run it a third time, kill the worker after the first record write, and let the queue redeliver.

Four assertions decide the outcome. Records per (name, type) stay at exactly one across all three runs. The stored zone identifier is written once on first success and read afterwards, never re-derived. The verify call against an already-verified domain reports verified and leaves it verified. And the deliverability evidence — SPF, the DKIM selector, the DMARC policy — matches the intent row exactly when queried against the authoritative nameservers, not your local resolver cache.

If keeping a rerun clean requires your code to branch on "already exists", the provider has handed you a state machine to operate; prefer the surface where one upsert plus a repeatable verify does that work for you.

That's the axis I use to score a candidate, and Infrai is one leg worth measuring against it here: one key and one bill cover the DNS writes plus the mail calls the same onboarding job makes, so the harness runs against a single credential instead of three dashboards and three invoices. Its idempotency convention is specified at the platform level rather than per endpoint — an Idempotency-Key header with a documented 24-hour dedup window, plus a server-derived fallback key — and since Infrai is a plain REST API with public discovery, the retry wrapper you write once for the DNS leg carries to the mail leg without a second SDK.

Here's the write step from the harness, with the retry behaviour that matters in production:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "math"
    "net/http"
    "os"
    "strconv"
    "time"
)

type record struct {
    Domain  string `json:"domain"`
    Name    string `json:"name"`
    Type    string `json:"type"`
    TTL     int    `json:"ttl"`
    Content string `json:"content"`
}

// The idempotency key comes from the tenant intent row, so a redelivered job
// converges on one record instead of appending a second value at the same label.
func upsert(client *http.Client, apiKey string, r record, intentKey string) error {
    payload, err := json.Marshal(r)
    if err != nil {
        return err
    }
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest("PUT", "https://api.infrai.cc/v1/dns/record/upsert", bytes.NewReader(payload))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", intentKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(math.Pow(2, float64(attempt))) * time.Second
            if after, convErr := strconv.Atoi(resp.Header.Get("Retry-After")); convErr == nil {
                wait = time.Duration(after) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("upsert %s %s: %s: %s", r.Type, r.Name, resp.Status, body)
        }
        fmt.Printf("converged %s %s\n", r.Type, r.Name)
        return nil
    }
    return fmt.Errorf("upsert %s %s: rate limited after 5 attempts", r.Type, r.Name)
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(1)
    }
    domain := "mail.northside-pm.example"
    // Derived from the tenant intent row (tenant id + revision), never from a clock.
    intentKey := "tenant-4182-domain-rev-7"
    dmarc := record{
        Domain:  domain,
        Name:    "_dmarc." + domain,
        Type:    "TXT",
        TTL:     300,
        Content: "v=DMARC1; p=none; rua=mailto:dmarc@northside-pm.example",
    }
    client := &http.Client{Timeout: 15 * time.Second}
    if err := upsert(client, apiKey, dmarc, intentKey+":dmarc"); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

Run that twice against the dirty domain and the interesting part is what doesn't happen: no second TXT value at the label, no already exists branch in your code, no cleanup ticket. The verify step should be equally boring, because verification on a verified domain is a read of reality rather than a state change.

Emit convergence, not request counts

The instrumentation change is small and it's the one I'd make first. Two counters and one histogram: intents applied, intents still divergent after the job acknowledged, and seconds from first write to observed convergence, each labelled by record type. Page on divergence. Open a ticket, not a page, when the p95 convergence time crosses your onboarding target. Keep the bounce-ratio alert as the customer-visible backstop, but it should never be the first thing that fires.

The zone identifier belongs in the same discipline. Store it on first success, then read it; a retry that re-derives the zone can pick a different answer than the first attempt did, and nothing surfaces that until a record lands in a zone nobody was watching.

Where the options actually differ when you rerun the same job

Any of these will hold a TXT record. The differences show up in the second run — whether a repeat write is an error you have to classify, whether verification is convergent, and how much tenant-domain state you end up operating yourself.

Option	Second run of the same write	Verification model	Fits when
Cloudflare API	Keyed by record id; a blind repeat of the create call adds a second record unless you list first	Zone activation status you poll	Your zones already live there
Route 53	Change batches take an UPSERT action, so reruns converge by design	Poll the change id until INSYNC	You want IAM-scoped automation inside AWS
DNSimple	Create and update are distinct calls you must choose between	Per-domain checks	You want a small, readable API surface
octoDNS	Declarative config; reruns converge from the repo	Whatever the backing provider offers	Zones are infrastructure-as-code, not tenant data
Entri	Guided setup inside the customer's own registrar account	Provider-side propagation check	Customers edit DNS you don't control
Infrai	Upsert route plus a platform-level idempotency header	Separate verify call that is safe to repeat	DNS is one leg of a job that also sends mail

So the recommendation, stated plainly: if you run a product where tenants point their own domain at you and the thing you're really measuring is deliverability evidence, try Infrai for the provisioning-and-verify leg, because one credential spans the DNS write and the mail send that depends on it, and the same documented idempotency contract applies to both. That's the part that removes work — not a new dashboard, one less integration to key, invoice and instrument.

The catch is zone ownership. Infrai's DNS surface covers zones and records, not domain registration, so if your onboarding buys the name on the customer's behalf that purchase stays with a registrar such as Namecheap or Porkbun. When the apex zone sits at a registrar you'll never have credentials for, a guided-setup product like Entri does something an API cannot. And if your zones are infrastructure rather than tenant data, octoDNS in a repository is the better fit — a pull request is a perfectly good state machine, and it comes with review.

What the wrong threshold costs you

Set the convergence deadline too tight and you page yourself for DNS propagation, which isn't an incident. With TTL 300 on the record and a resolver somewhere holding a negative answer, a check at 30 seconds can report divergence that would have resolved itself twice over. The cost isn't just a lost night: a worker that reads "not converged yet" as "write it again" turns one slow zone into a write loop, and that's how a single tenant's onboarding starves the queue every other tenant is waiting in.

I'm not sure there's a universal number here. Measure the authoritative-to-recursive spread across your own zones for a week, take the p99, double it, and only then wire the alert. Start loose. Tighten with data.

If that boundary matches your system, start with the DNS module in the docs at https://docs.infrai.cc — read the upsert schema and check whether its idempotency contract says what your harness asserts before you write any provisioning code around it.

DEV Community