Treat a seller's custom domain as desired state, not as a form submission. The add call hands you a zone id, you store it on the tenant row in the same transaction that stores the hostname, and from then on every record write and every verification attempt is a reconciler replaying that stored intent. Use a scheduled worker for the verify step. The customer's HTTP request is the worst place in your system to sit and wait for a nameserver to agree with you.
That one rule is also what buys you a rollback path, because a row you can bump is a row you can bump backwards.
Marketplace onboarding is never one-shot. A seller points shop.northbay-supply.com at your storefront for a campaign, moves it back to their old host three weeks later, then asks you to move it again on a Friday afternoon. Each of those is a cutover, and each one needs to be reversible by a person who was not in the original thread.
Where the drift comes from
The interesting bug is not "the record was wrong". It's that your database says one thing and the published zone says another, and nothing in the system is looking at both.
Three sources cover almost all of it. The first is a write that never happened: the zone got created, the request died somewhere after that, and the tenant row shows an onboarding that looks complete because the zone id is sitting right there. The second is a human — someone edits the record by hand at the registrar during an incident, fixes the symptom, and never tells the platform that owns the intent. The third is caching: you cut over with a 3600-second TTL still on the record, and for the next hour a meaningful slice of resolvers happily serves the old target while your dashboard shows green.
Negative caching is the one that catches people. If a resolver asked for the hostname before you published anything, the NXDOMAIN answer is cached according to the SOA minimum (RFC 2308), not according to the TTL you're about to set. Publish first, then verify, then tell the seller — in that order.
Nobody pages you for a zone that was created. They page you when a storefront 404s at 09:00 on launch day and the only artifact is a Slack message saying "it worked yesterday".
How should I add a zone, write the record, and verify a custom domain without blocking onboarding?
Four steps, and the ordering matters more than the client library you use to implement them.
Add the domain and persist the returned zone id immediately, in the same write as the hostname. That identifier is the join key for every later record operation, and losing it means you can neither update nor clean up what you created. Write the record as a single upsert carrying zone id, type, name and content together — there is no partial write, so a client that sends a name today and content tomorrow is not a resumable workflow, it's two broken states. Verify from a scheduled job rather than inline, because propagation is not something a request handler can wait out. And key the whole sequence so that a seller mashing refresh produces one zone, one record and one verification attempt.
Put the intent write in the Node.js request handler and nothing else. The handler returns 202, the reconciler does the DNS work, and the UI polls a status field. Splitting it this way is what lets the same code path serve onboarding, cutover and rollback: all three are just a new revision of the intent row, and the reconciler doesn't care which one it is.
Idempotency keys should be derived from state you already have, not generated per attempt. Tenant id, hostname, revision. A retry after a timeout sends the same key and lands on the same result; a genuine cutover bumps the revision and gets a fresh key.
The write path in Go
This is the whole reconciler minus the database access — one upsert, one verification call, retries that honour Retry-After, and a key that makes both safe to repeat.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
// Intent is the tenant row. ZoneID comes back from the domain add call
// and never changes; Revision is bumped on every cutover and rollback.
type Intent struct {
TenantID string
Hostname string
ZoneID string
Target string
Revision int
}
type client struct {
base string
key string
http *http.Client
}
func newClient() *client {
return &client{
base: os.Getenv("DNS_API_BASE"), // API root for your provider
key: os.Getenv("INFRAI_API_KEY"), // ifr_...
http: &http.Client{Timeout: 15 * time.Second},
}
}
// do sends one request, retrying only on 429 and only with backoff.
// The Idempotency-Key means a retry re-reads the first result
// instead of applying a second change.
func (c *client) do(method, path, idem string, payload any) ([]byte, error) {
body, err := json.Marshal(payload)
if err != nil {
return nil, err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, c.base+path, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+c.key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idem)
resp, err := c.http.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(backoff(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s: http %d: %s", method, path, resp.StatusCode, data)
}
return data, nil
}
return nil, fmt.Errorf("%s %s: rate limited after 5 attempts", method, path)
}
func backoff(retryAfter string, attempt int) time.Duration {
if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
// Reconcile publishes the stored intent, then asks for verification.
// Safe to call every five minutes until the hostname reports verified.
func (c *client) Reconcile(in Intent) error {
idem := fmt.Sprintf("%s:%s:r%d", in.TenantID, in.Hostname, in.Revision)
if _, err := c.do(http.MethodPut, "/v1/dns/record/upsert", idem+":cname", map[string]any{
"zone_id": in.ZoneID,
"type": "CNAME",
"name": in.Hostname,
"content": in.Target,
"ttl": 300,
}); err != nil {
return err
}
out, err := c.do(http.MethodPost, "/v1/dns/domain/verify", idem+":verify", map[string]any{
"zone_id": in.ZoneID,
})
if err != nil {
return err
}
fmt.Printf("tenant=%s host=%s rev=%d verify=%s\n", in.TenantID, in.Hostname, in.Revision, out)
return nil
}
func main() {
in := Intent{
TenantID: "t_8841",
Hostname: "shop.northbay-supply.com",
ZoneID: os.Getenv("ZONE_ID"),
Target: "storefronts.example-marketplace.com",
Revision: 4,
}
if err := newClient().Reconcile(in); err != nil {
fmt.Fprintln(os.Stderr, "reconcile:", err)
os.Exit(1)
}
}
Two details worth defending. The TTL of 300 is not a default I inherited, it's a decision: short enough that a rollback takes minutes, long enough that a popular storefront isn't re-resolving constantly. And the reconciler returns the provider's error body verbatim, because a 4xx here almost always carries the actual reason — a hostname that belongs to another zone, a CNAME sitting on an apex, a record type the zone won't accept.
Choosing where the records actually live
The decision axis is not features, it's how quickly you can prove that published records match your intent, and how cheap it is to write both directions.
| Option | How you write records | Verification story | Where it gets awkward |
|---|---|---|---|
| Cloudflare for SaaS | REST per zone, plus custom-hostname objects | Built-in hostname validation and certificate issuance per tenant | Tied to their edge; the custom-hostname model is its own concept to learn |
| AWS Route 53 |
ChangeResourceRecordSets with UPSERT batches |
Change ids move to INSYNC, which is a real propagation signal |
Per-tenant zones multiply fast; IAM scoping for tenant isolation is work |
| DNSimple | REST zone-record endpoints, plus webhooks | Webhooks tell you when a record changed under you | Smaller surface if you also want scheduling and workers nearby |
| Entri / Approximated | Guided end-user flow or a hostname proxy | Handles the seller-side registrar dance for you | You hand over part of the cutover; less control of the rollback timing |
| Infrai | Plain HTTP calls, one key covering DNS and the scheduler | Verification endpoint plus a scheduled trigger on the same contract | No registrar transfers, no per-tenant certificate product |
Infrai is a reasonable pick when the verify loop is the annoying part rather than the DNS itself: records, the scheduled trigger and the worker sit behind one key with consistent conventions, so adding the five-minute reconciler is one more endpoint instead of one more integration to operate. The catch is scope. If per-tenant TLS for custom hostnames is the hard half of your product, stick with Cloudflare for SaaS or Approximated, which are built around exactly that; if your zones are already in Route 53 and your compliance story depends on it, don't move them for the sake of a tidier worker.
There's a fourth option people forget: keep the records in a declarative file and reconcile with octoDNS or DNSControl, or with external-dns if you're already on Kubernetes. Drift detection comes free, since the whole model is a diff against provider state. It fits badly with per-seller hostnames created at runtime, which is why marketplaces usually end up calling an API instead — but if your custom domains are counted in dozens rather than thousands, the config-file approach is less machinery for the same guarantee.
Verify, then keep verifying
The verification call tells you the provider's view. dig tells you the internet's view, and on cutover day only the second one matters.
dig +short NS northbay-supply.com
dig +short CNAME shop.northbay-supply.com @1.1.1.1
dig +short CNAME shop.northbay-supply.com @8.8.8.8
Query at least one authoritative server and two public resolvers, compare the answers to the Target field on the intent row, and record the result with a timestamp. That comparison is your drift check, and it should run on the same five-minute schedule as the reconciler — not just during onboarding, but for the life of the hostname. A record that was correct on Tuesday is only evidence about Tuesday.
Rollback then costs nothing dramatic. Write the previous target back into the intent row, bump the revision, and let the next run upsert it; the new revision produces a new idempotency key, so the change applies once and the retry semantics stay intact. Keeping the prior revisions around is what makes this work — if the table only holds current state, your rollback is a human retyping a hostname from memory at the worst possible moment.
Two fields and a revision bump. That's the whole reverse path.
One thing I'd flag as genuinely uncertain: the right interval for the drift check after a hostname is stable. Five minutes is cheap during a cutover and mostly noise a month later, and I'm not aware of a published number that settles it. Backing it off to hourly once a hostname has verified clean for a day seems defensible, but your traffic pattern and your paging tolerance should decide that, not a blog post.
Top comments (0)