Negative caching is the constraint that decides this design, not propagation. A gaming platform that hands every studio tenant its own subdomain will write the DNS records and then check them inside the same onboarding request, and if that check reaches a recursive resolver a few hundred milliseconds before the authoritative servers can answer, the resolver caches the miss. The tenant's sending domain then reads as unverified for as long as the zone's SOA minimum allows — however fast the write itself was. Bottom line: bundle DNS setup plus mail verification into one onboarding step, but split that step internally into an authoritative write that returns in seconds and a verification poll that never asks a public resolver about a name it hasn't already watched come into existence.
Propagation is the wrong thing to be afraid of here.
The retry that cached its own failure
The wizard looked right on paper. One form, one submit: create the tenant, write the A, TXT and CNAME records under <slug>.play.example.net, verify the sending domain, flip the tenant live so its players can get account-recovery and tournament mail.
I assumed the verification read was cheap, so the job retried it on any error — fixed 5 s backoff, ten attempts, no jitter, the kind of retry loop that looks responsible in review.
It was cheap. It just wasn't idempotent in the way that mattered. Every attempt hit the same recursive resolver that had already cached NXDOMAIN for s1._domainkey.mail.<slug>.play.example.net, and RFC 2308 pins that negative answer to the smaller of the SOA record's own TTL and its MINIMUM field. Ours was 3600. So ten retries inside fifty seconds were ten questions to a cache that had made up its mind for the next hour, the step reported an error, an operator re-ran onboarding by hand, and the studio received its welcome mail twice — the second copy landing while somebody was still reading the first.
The postmortem produced one line worth keeping: a verification probe must never be the first query for a name.
Everything below is downstream of that.
Should one onboarding step do DNS setup and mail verification together?
Yes, and "one step" is a promise to the operator rather than a description of the network path. The tenant sees a single submit that either completes or can be submitted again without anyone reasoning about partial state. Underneath, it is a declarative write followed by a staged read.
Bundling buys retryability, which is the property that actually pays. Splitting DNS across one provider and mail across another is a large part of why SPF and DKIM setup has its reputation: you end up holding two half-finished states with no single place to reconcile them, and the recovery procedure becomes a human diffing two dashboards at 02:00. One flow behind one credential lets you re-run the whole thing and converge. Read the sending domain's verification status back from the mail side afterwards and let that drive the UI, rather than inferring success from an HTTP 200 on the write.
Per-tenant DMARC records mostly don't belong in the bundle at all. RFC 7489 resolves policy for mail.<slug>.play.example.net by querying _dmarc at that name and, when nothing is published there, at the organizational domain, where an sp= tag declares the policy that applies to subdomains. Publish _dmarc.play.example.net once with an explicit sp=, and every tenant you will ever create is covered — one fewer record whose absence a resolver can cache against you. SPF pushes in the same direction for a different reason: RFC 7208 caps a policy at ten DNS-querying mechanisms and returns permerror beyond that, so include: chains are a budget you spend once, centrally, not a thing that grows per tenant.
What a wildcard buys you, and the exact label where it stops
A wildcard A record at *.play.example.net gives instant cutover for the portal hostname. Nothing per-tenant has to be published, so nothing has to propagate, and a tenant created three seconds ago resolves.
The catch is that the wildcard stops covering that tenant the moment the tenant becomes real. RFC 4592 synthesises from a wildcard only when the query name has no closer existing ancestor. Once <slug>.play.example.net exists as a node — and it has to exist, to carry a distinct edge address or the tenant's own DKIM records — a query for s1._domainkey.mail.<slug>.play.example.net no longer matches *.play.example.net. Its closest encloser is now the tenant node, the source of synthesis would have to be *.<slug>.play.example.net, and that name doesn't exist. The answer is NXDOMAIN, and NXDOMAIN is exactly the answer you do not want a cache to hold.
TLS has the same geometry. A certificate for *.play.example.net does not cover mail.<slug>.play.example.net one label deeper, and a wildcard certificate has to be validated with a dns-01 challenge, which puts issuance back on the same zone-write path you were hoping to route around.
| Approach | Time to first correct answer | What you still wait on | Where it stops |
|---|---|---|---|
| Wildcard address record for tenant hostnames | Immediate | Nothing | Any name below an existing tenant node |
| Per-tenant records written at signup | One authoritative write, seconds | Recursive caches, but only if you query early | Write throughput and rate limits on the zone API |
| CNAME delegation to a shared verification target | One write per tenant | The target's TTL, which you control centrally | Anything that needs a record at a zone apex |
| Tenant brings its own domain | Human-scale, hours to days | Their provider, their TTL, their change process | Most of what you wanted to automate |
The middle two rows are where a multi-tenant game platform actually lives, and they differ mainly in who owns the TTL you are exposed to.
A provisioning path that survives being run twice
The shape below is deliberately boring. Desired state in, diff applied by the provider, authoritative check before anything touches a cache, mail verification last. Declarative zone tooling such as octoDNS and DNSControl works on the same principle: the zone is a document, not a sequence of imperative calls, which is what makes a second run a no-op instead of a duplicate.
type Record struct {
Name string
Type string
Value string
TTL uint32
}
type Zone interface {
// Upsert applies the desired records and returns the new zone serial.
// The same input twice yields the same zone state and no duplicates.
Upsert(ctx context.Context, recs []Record) (serial uint32, err error)
Nameservers(ctx context.Context) ([]string, error)
}
// Provision is the whole onboarding step for one tenant. Re-running it after
// a crash, a timeout, or an impatient operator has to be a no-op.
func (p *Provisioner) Provision(ctx context.Context, t Tenant) error {
host := t.Slug + ".play.example.net"
mail := "mail." + host
desired := []Record{
{Name: host, Type: "A", Value: p.EdgeIP, TTL: 60},
{Name: mail, Type: "TXT", Value: "v=spf1 include:" + p.SPFInclude + " -all", TTL: 300},
{Name: p.Selector + "._domainkey." + mail, Type: "CNAME", Value: p.Selector + "." + p.DKIMHost, TTL: 300},
}
serial, err := p.Zone.Upsert(ctx, desired)
if err != nil {
return fmt.Errorf("upsert %s: %w", host, err)
}
if err := p.State.Set(ctx, t.ID, "records_written", serial); err != nil {
return err
}
// Phase one: ask the zone's own servers. A miss here is cached nowhere,
// so it is safe to poll early and safe to poll hard.
ns, err := p.Zone.Nameservers(ctx)
if err != nil {
return err
}
if err := awaitAuthoritative(ctx, ns, desired); err != nil {
return fmt.Errorf("authoritative %s: %w", host, err)
}
// Phase two: the names exist everywhere that matters, so the recursive
// path can see them and the mail side can verify the sending domain.
return p.Mail.Verify(ctx, mail)
}
The resolver is the part people skip. Go's net.Resolver will dial whatever you tell it to, so pointing it at the zone's nameservers costs about ten lines and removes the entire class of problem described above.
// authoritative binds a resolver to one of the zone's own nameservers,
// bypassing every cache between the provisioner and the truth.
func authoritative(ns string) *net.Resolver {
return &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
d := net.Dialer{Timeout: 2 * time.Second}
return d.DialContext(ctx, network, net.JoinHostPort(ns, "53"))
},
}
}
// awaitAuthoritative blocks until every nameserver serves every record.
// Partial agreement is the failure mode behind every "it verified on my
// laptop" report, so all of them have to answer before we move on.
func awaitAuthoritative(ctx context.Context, servers []string, recs []Record) error {
for {
missing := 0
for _, ns := range servers {
r := authoritative(ns)
for _, rec := range recs {
if !served(ctx, r, rec) {
missing++
}
}
}
if missing == 0 {
return nil
}
select {
case <-ctx.Done():
return fmt.Errorf("%d record/server pairs unserved: %w", missing, ctx.Err())
case <-time.After(2 * time.Second):
}
}
}
func served(ctx context.Context, r *net.Resolver, rec Record) bool {
switch rec.Type {
case "TXT":
vals, err := r.LookupTXT(ctx, rec.Name)
return err == nil && slices.Contains(vals, rec.Value)
case "CNAME":
target, err := r.LookupCNAME(ctx, rec.Name)
return err == nil && strings.EqualFold(strings.TrimSuffix(target, "."), rec.Value)
default:
addrs, err := r.LookupHost(ctx, rec.Name)
return err == nil && slices.Contains(addrs, rec.Value)
}
}
slices landed in the standard library in Go 1.21, so this compiles without a helper package. The caller passes a context with a deadline — ninety seconds is generous for a zone you own, and a tenant that hasn't converged by then is a page, not a spinner.
What to measure before a tenant goes live
Alarm on the interval between the upsert and all nameservers serving all records, not on "verification succeeded". The second number hides the first, and the first is the one that degrades when a zone API starts queueing writes.
Two more gauges earn their keep. Export the zone's SOA MINIMUM, because that value is the blast radius of every premature query — moving it from 3600 down to 300 shrinks your worst case from an hour to five minutes and costs a little more query volume. Then count re-provisions per tenant: anything above one means the step is not idempotent yet, whatever the code review said.
Keep the welcome mail out of the provisioning job entirely. It belongs to a separate consumer keyed by tenant ID with a dedupe key, because provisioning is the thing you want to retry freely and mail is the thing you must never send twice.
Verifying by hand stays useful, and it is worth doing from a machine that has never queried the name:
dig +norecurse @ns1.example.net s1._domainkey.mail.acme.play.example.net CNAME
dig +noall +answer _dmarc.play.example.net TXT
Where this advice does not apply: if your tenants bring their own domains, you do not own the zone, cannot observe the authoritative write, and the honest interface is a documented record list plus a status check the customer can re-run. Stick with that path and keep it visible even for tenants you could automate — some operations teams will insist on writing the records themselves, and taking that option away buys you a support queue. If every tenant sends from one shared domain, skip per-tenant records altogether; the trade-off is a shared sender reputation, which a gaming platform with one abusive studio will feel immediately. And if a tenant needs its own DMARC aggregate reports, the organizational-domain shortcut doesn't support that, so it gets a real _dmarc record and the propagation wait that comes with it.
I'm not sure the authoritative-first check is worth building below a certain scale. Onboarding five studios a week, a human waiting sixty seconds is fine. At five hundred, the cache is the system.
Sources
- RFC 2308, Negative Caching of DNS Queries: https://datatracker.ietf.org/doc/html/rfc2308
- RFC 4592, The Role of Wildcards in the Domain Name System: https://datatracker.ietf.org/doc/html/rfc4592
- RFC 7489, Domain-based Message Authentication, Reporting, and Conformance: https://datatracker.ietf.org/doc/html/rfc7489
- RFC 7208, Sender Policy Framework version 1: https://datatracker.ietf.org/doc/html/rfc7208
- RFC 6376, DomainKeys Identified Mail Signatures: https://datatracker.ietf.org/doc/html/rfc6376
- Go standard library, net.Resolver: https://pkg.go.dev/net#Resolver
- Let's Encrypt, challenge types: https://letsencrypt.org/docs/challenge-types/
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.