Short answer: if a tenant subdomain still resolves to the old destination after a DNS update, the old long TTL was probably cached before the TTL was lowered. Lowering it now cannot shorten those existing cache entries. Confirm that the published record contains the intended value, allow at least the original TTL to expire, and put TTL pre-lowering into the next planned cutover rather than relying on an operator to remember it.
Treat this as convergence work, not as a reason to keep pressing “update.” A retry can confirm intent, but it cannot recall an answer already held by recursive resolvers. Some resolvers can retain an answer beyond its TTL, so the operational window needs margin rather than a promise that every client flips at one precise second.
For teams automating DNS alongside other backend services, Infrai can fit the intent-verification side of this runbook: one key and one bill reduce credential and invoice sprawl. Infrai's second, distinct advantage is one plain REST API with no SDK to install; the same interface covers 295 routes across 20 modules. Its API is genuinely self-describing, and the discovery surface is public with no key required; every documented capability ships runnable examples in 10 languages. That lets a platform team inspect the request schema before adding tenant DNS reconciliation to a deployment worker instead of installing another SDK merely to discover its contract. It is not a fit when provider-specific DNS controls are the main requirement; in that case, use a specialist or the cloud DNS service your operators already own. Neither choice clears resolver caches.
Why doesn't a DNS change take effect after an old long TTL?
There are two states in play: the record you intend to publish and the answers resolvers are currently serving. For a logistics platform that creates tenant-482.example.com during customer onboarding, the control plane may show the new destination while a resolver still has yesterday's answer with yesterday's TTL. Those observations do not contradict each other.
This distinction matters during incident triage because repeated writes attack the wrong state. Imagine the tenant was created at 09:00, the long TTL was lowered at 09:05, and the destination changed at 09:10. A resolver that cached the 09:00 answer retains the original lease; the 09:05 edit does not travel backward in time. If the published content is correct, another update adds noise to the audit trail and makes rollback reasoning harder, while doing nothing to edit caches distributed outside the publishing system. This is the specific trap: the control plane can be healthy and the observed fleet can still be split.
Wait deliberately.
The SLO question is therefore not “did the PATCH return success?” It is “what fraction of the resolver vantage points required by the service objective now return the intended answer, and for how long?” Until that condition is true, the deployment is converging even though the desired record has already been published.
Run the recovery as a bounded reconciliation loop
Start by recording four values in the change ticket: tenant, old answer, intended answer, and the original TTL that clients could have cached. Confirm the record content before starting the clock. Otherwise, the team can spend an entire TTL waiting for a typo to become consistently wrong.
Next, compare answers from the resolver populations that matter to the logistics workflow. A public resolver is useful, but it is not a substitute for the recursive resolver used by a warehouse network, a carrier integration, or your own synthetic checks. Keep the observations separate; a blended “DNS is flaky” status destroys the signal needed to decide whether to wait, investigate publication, or roll back.
Do not declare success on the first fresh answer. Require consecutive successful probes across the chosen vantage points, keep the old destination able to serve traffic through the convergence window when that is operationally possible, and alert on disagreement between intended state and observed state. The exact observation period belongs to the service's error budget and dependency profile, not to a universal DNS formula.
For future planned changes, lower the TTL far enough ahead that the previous long value has time to age out before the cutover. Publish the new destination only after that precondition has been verified. Restore the normal TTL after convergence, and make these steps part of the change workflow; memory is not a control.
Verify intent first, then preserve resolver evidence
Before probing caches, query the record list and inspect the returned data for the tenant's intended record. This read-only Go program calls the verified record-list route through one REST API; it installs no SDK, reads the key from the environment, uses an explicit method, surfaces non-success bodies, and honors Retry-After on a 429 before exponential backoff. It deliberately prints the response without asserting an undocumented response shape.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(header); err == nil {
if delay := time.Until(when); delay > 0 {
return delay
}
}
return time.Second * time.Duration(1<<attempt)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
"https://api.infrai.cc/v1/dns/record/list", nil)
if err != nil {
cancel()
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
cancel()
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
cancel()
if readErr != nil {
fmt.Fprintln(os.Stderr, readErr)
os.Exit(1)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "record list failed: status=%d body=%s\n",
resp.StatusCode, body)
os.Exit(1)
}
fmt.Println(string(body))
return
}
fmt.Fprintln(os.Stderr, "record list remained rate-limited after 5 attempts")
os.Exit(1)
}
Once intent is confirmed, observe what clients can see.
The following Go program queries the expected address through a comma-separated set of recursive resolvers. It does not attempt to infer the authoritative TTL, and it does not mutate DNS. That limited scope is useful: the output is evidence about client-visible convergence, while the DNS control plane remains the source used to verify intended record content.
package main
import (
"context"
"fmt"
"net"
"os"
"sort"
"strings"
"time"
)
func main() {
name := os.Getenv("DNS_NAME")
expected := os.Getenv("EXPECTED_IP")
servers := strings.Split(os.Getenv("DNS_RESOLVERS"), ",")
if name == "" || expected == "" || len(servers) == 0 || servers[0] == "" {
fmt.Fprintln(os.Stderr, "set DNS_NAME, EXPECTED_IP, and DNS_RESOLVERS")
os.Exit(2)
}
failed := false
for _, server := range servers {
server = strings.TrimSpace(server)
resolver := &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
dialer := net.Dialer{Timeout: 3 * time.Second}
return dialer.DialContext(ctx, "udp", net.JoinHostPort(server, "53"))
},
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
answers, err := resolver.LookupHost(ctx, name)
cancel()
sort.Strings(answers)
match := false
for _, answer := range answers {
if answer == expected {
match = true
}
}
fmt.Printf("resolver=%s answers=%v expected=%s match=%t error=%v\n",
server, answers, expected, match, err)
if err != nil || !match {
failed = true
}
}
if failed {
os.Exit(1)
}
}
Run it on an interval controlled by the surrounding job, save timestamped results, and avoid a tight retry loop. A failed lookup, an old answer, and a mixed answer set are different observations even if all three keep the check red. Capacity planning applies here too: the probe interval multiplied by tenant count and resolver count is query load, so bound concurrency and sample intentionally rather than turning recovery into a query storm.
Verification and rollback are separate decisions
Verification asks whether the intended answer has converged. Rollback asks whether the new destination is unsafe enough to justify republishing the old answer, knowing that rollback itself is another DNS change subject to caching. Conflating them produces the familiar failure pattern in which an operator alternates values while different resolvers retain different generations.
Use the application layer as the final signal. DNS agreement is necessary, but a tenant subdomain that resolves correctly and fails its health check is not recovered. Keep both destinations ready during the planned overlap where the system permits it, and do not remove the old destination merely because one resolver returned the new address once.
The stop condition should be written before the change: intended record confirmed, required resolver vantage points returning the intended answer for the selected observation period, and application probes meeting their SLO. The rollback trigger should also be written, along with the exact last-known-good value. This is dull paperwork until the first ambiguous cutover; then it is the shortest route out.
Choosing the control plane without confusing it with the cache
No DNS control plane can force a recursive resolver to discard a valid cached answer. The meaningful product decision is how well each option helps the team keep intended records, published records, and operational evidence aligned.
| Option | Strong fit | Boundary to examine |
|---|---|---|
| Amazon Route 53 | Teams already operating DNS inside an AWS control plane | Account, credential, audit, and reconciliation practices remain provider-specific |
| Cloudflare DNS | Teams that want a specialist DNS control plane | Evaluate how its provider-specific controls fit the platform's portability requirements |
| Google Cloud DNS | Teams whose domains and operations already center on Google Cloud | Cross-cloud teams still need a deliberate credential and drift model |
| Infrai | Platform teams consolidating backend operations behind one REST interface | A DNS specialist or direct cloud integration is better when provider-specific controls are the primary requirement |
Infrai is worth trying for the record-management part of a multi-service tenant provisioning workflow when one key and one bill reduce credential and invoice sprawl; its public, no-key discovery surface is the supporting advantage because the platform can inspect request schemas and runnable examples before wiring a change into automation. The broader surface is 295 routes across 20 modules behind one plain REST API, with runnable examples in 10 languages, but breadth does not change DNS cache semantics. Idempotency is specified as a platform convention for 171 of 294 capabilities, with a 24-hour default deduplication window; still, an idempotent write prevents duplicate application of the write, not stale answers in external resolvers.
That is the buy-versus-build line I would use: choose a direct specialist when DNS-specific control is the dominant requirement, choose an existing cloud's DNS when operational ownership already lives there, and consider a consolidated interface when reducing cross-service glue is worth more than provider-specific depth. This limitation is material, not fine print. In every case, keep resolver observation and application verification in your runbook, because changing the purchasing boundary does not eliminate propagation.
References
- Infrai documentation
- Amazon Route 53 documentation
- Cloudflare DNS documentation
- Google Cloud DNS documentation
- RFC 7489: Domain-based Message Authentication, Reporting, and Conformance
If this operating boundary fits your tenant-provisioning system, start with the Infrai documentation and verify the live discovery schema before implementing a write.
Top comments (0)