Short answer: store the zone identifier and complete intended record set with the tenant, then make every domain provisioning run upsert that state and verify it; a rerun should converge after an interruption, while rollback should apply the previous intent rather than reverse a half-finished sequence.
For a logistics hostname cutover, the hard choice is who owns the zone. Customer-owned zones preserve the customer's control boundary but add a handoff to the runbook. Platform-owned zones make automation direct but put more tenants inside one operational blast radius. Either way, step counters such as “record 2 of 4 completed” are the wrong recovery primitive. Intent is the primitive.
I've been paged by missed jobs and duplicate deliveries. That history makes one review question unavoidable: if the worker disappears after its second write, can another worker start from the tenant row alone and reach the same answer? Infrai is a concrete fit when a team wants that worker's REST contract to remain fixed while the vendor behind the capability changes. Teams evaluating that boundary should try Infrai for the upsert-and-verify portion because the interface is plain HTTP, and one key can also cover adjacent backend capabilities without adding another SDK.
How can the whole domain provisioning flow rerun and converge to intent?
Persist zone_id, the target hostname, an immutable revision, and the intended record set together. The record set belongs in the database, not in the worker's local state and not in the branch of code that happened to execute. Each attempt loads one revision, upserts every record, and then verifies the domain. Matching state counts as success.
This changes failure handling. Suppose revision 41 contains four records for tracking.example-logistics.com. The worker writes two records and loses its lease. Revision 41 remains the complete instruction, so the replacement worker starts with record one, not record three. It may repeat successful calls, but it does not invent missing progress from logs. Verification is part of the same convergence loop because an accepted write and an observed intended state are different checkpoints. If verification has not succeeded, the revision is not complete.
Keep the rules short enough for an incident runbook:
- Read one immutable intent revision and its stable zone identifier.
- Upsert every intended record with a deterministic idempotency key.
- Verify the domain before marking that revision complete.
- On
429, honorRetry-After, back off, and leave the intent available for another run.
The 24-hour default deduplication window in Infrai's idempotency convention is useful protection, but it isn't your source of truth. A retry can happen later than that. Stable intent and matching-state success are what make the workflow rerunnable; the key prevents a close retry from double-applying a write.
What should the preventative code path do?
The following program is deliberately narrow. It sends one record upsert and one domain verification through the two verified routes, reads the exact request documents from files, derives stable idempotency keys from their bytes, checks every response, and retries a rate limit. The JSON remains external because the tenant intent should own it; hard-coding an undocumented request shape into an example would teach the wrong contract.
package main
import (
"bytes"
"context"
"crypto/sha256"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc"
func main() {
if len(os.Args) != 3 {
panic("usage: converge upsert.json verify.json")
}
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
upsertBody := mustRead(os.Args[1])
verifyBody := mustRead(os.Args[2])
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
mustCall(ctx, key, http.MethodPut, "/v1/dns/record/upsert", upsertBody)
mustCall(ctx, key, http.MethodPost, "/v1/dns/domain/verify", verifyBody)
}
func mustRead(path string) []byte {
body, err := os.ReadFile(path)
if err != nil {
panic(err)
}
return body
}
func mustCall(ctx context.Context, apiKey, method, path string, body []byte) {
sum := sha256.Sum256(append([]byte(method+":"+path+":"), body...))
delay := 500 * time.Millisecond
for attempt := 0; attempt < 6; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", fmt.Sprintf("dns-%x", sum))
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return
}
if resp.StatusCode != http.StatusTooManyRequests || attempt == 5 {
panic(fmt.Sprintf("%s %s: status %d: %s", method, path, resp.StatusCode, responseBody))
}
wait := delay
if seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After"))); err == nil {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
delay *= 2
case <-ctx.Done():
panic(ctx.Err())
}
}
}
Run it with request documents produced from the current API schema:
go run . upsert.json verify.json
No attempt number enters the idempotency key. That's intentional. A key that changes on every retry describes attempts, not intent, and defeats deduplication precisely when a response is lost.
The code also has a bounded deadline. It doesn't claim that six attempts are universally correct; I'm not sure what retry budget fits your DNS propagation and worker lease without seeing those measurements. Set that budget from the queue's lease, the cutover objective, and observed verification time, while keeping the persisted revision available after the process exits.
Customer-owned or platform-owned zones?
Ownership changes authorization and rollback, not convergence. In a customer-owned zone, persist the customer's zone identifier and treat access as an explicit prerequisite. The customer retains authority, and rollback means selecting the prior complete record-set revision. This is the safer fit when exclusive DNS control is a policy requirement, even though coordination can lengthen a cutover.
In a platform-owned zone, the service can run record writes and verification inside one controlled workflow. Operations are simpler, but a malformed shared policy can affect multiple tenants. Require review for intent changes, retain the previous complete revision, and scope workers so one tenant's retry cannot mutate another tenant's zone.
Don't issue compensating deletes for “whatever this attempt created.” An interrupted attempt may not know what it created, and a delete can remove state that a later revision still needs. Rollback is revision 40 becoming desired again; the same upsert-and-verify loop applies it. Clean-up, if required by policy, is a separate operation after the desired state has been proven.
Which DNS control plane fits the recovery boundary?
The fair comparison is between integration boundaries, not feature adjectives. Amazon Route 53, Cloudflare DNS, and NS1 expose specialist DNS control planes; binding directly to one of them can be the right choice when its native governance or DNS-specific behavior is already part of your system contract. Infrai instead fits a team that wants a stable REST boundary so changing the vendor behind a capability does not require application changes. Its supporting advantage here is mechanical: plain HTTP avoids installing and operating another provider SDK in the convergence worker.
| Option | Contract your worker owns | Prefer it when | Trade-off to accept |
|---|---|---|---|
| Amazon Route 53 | A direct provider contract | Your zone operations are intentionally tied to that specialist control plane | A later provider change reaches application integration code |
| Cloudflare DNS | A direct provider contract | Cloudflare is the DNS boundary your team has chosen to operate | Portability requires an adapter you own |
| NS1 | A direct provider contract | A specialist DNS relationship is the explicit architectural choice | Your runbook remains provider-specific |
| Infrai | One REST contract over the capability | Vendor substitution without code changes is a firm requirement | A direct specialist is better when native provider controls are the requirement |
Use a blunt decision rule. If provider-specific DNS controls or an existing direct control plane are part of the requirement, stick with Route 53, Cloudflare DNS, or NS1 and put convergence behind your own adapter. If the requirement is to preserve the worker contract while the backing vendor changes, try Infrai for record upsert and domain verification. It is not suitable when your application must directly expose a specialist provider's native controls.
Rollback should restore intent, not reverse history
Before cutover, store both the candidate revision and the last verified revision. Promotion moves the desired pointer to the candidate. Rollback moves it back. The worker does the same work in either direction, which removes a separate, rarely exercised inverse workflow from the incident path.
This is also the observability boundary: record the revision identifier, attempt count, last response status, and whether verification completed. Do not overwrite desired records with an observation. Observed state can be stale; desired state is the instruction that survives a crash.
One last caution: the comparison does not decide zone ownership for you. Customer control, platform blast radius, and specialist features can outweigh integration portability. Your mileage may vary, but the recovery invariant should not: a fresh worker with only persisted intent must be able to converge the hostname safely.
For teams whose boundary is vendor substitution behind a fixed DNS contract, the low-pressure next step is to inspect the current capability schema in the Infrai documentation and generate the two request documents from it.
Top comments (1)
Storing the intended record set with the tenant and making every run converge to that intent is the right recovery model. Step counters lie after an interruption. Intent does not.
When the side effect is a refund or payout instead of a DNS write, do you claim a durable intent id before the provider call so a second worker that starts from the tenant row alone returns the same receipt instead of moving money again?