A newly moved e-commerce tenant resolving to the old edge is a routing problem first and a cache problem second. For short DNS TTLs everywhere versus pre-change lowering, the least complex policy is to keep a normal TTL for stable tenant subdomains, lower it in a pre-change step before a scheduled target change, and restore it after verification. That gives a platform a stated cutover window without making every ordinary lookup depend on a permanently tiny cache lifetime.
TL;DR: short DNS TTLs everywhere buy future flexibility, but they do not shorten an answer already cached with the old TTL. Pre-change lowering works only when the lower value has been published early enough for old positive answers to age out. Check the authoritative zone before blaming recursive resolvers, and set the alert window from the actual cutover plan.
The page arrives with a tenant hostname going to the old edge after a routing move. A responder sees a correct application deployment, a successful publish job, and an unhappy shopper. Those signals are easy to confuse. The question is whether the zone has the intended answer, or whether a resolver is still allowed to use its previous one.
Should short DNS TTLs run everywhere before a pre-change cutover?
No universal TTL is correct. A low TTL reduces the maximum age of newly cached answers, which can make an unscheduled correction less painful. It also means recursive resolvers must come back to authoritative infrastructure more often. That is a real operational trade-off, especially for a busy tenant namespace, rather than a setting that makes every DNS change immediate.
Pre-change lowering separates routine operation from migration agility. Publish the lower TTL, wait at least the former TTL before switching the target, then leave the lower value in place only as long as the rollback window requires. A resolver that cached the old target one second before the lower TTL was published may legally retain that old answer for the old TTL. Changing a number after the fact cannot reach into that cache.
The old TTL is the timer that matters.
RFC 2181 section 8 defines the valid DNS TTL range as 0 through 2^31-1 seconds. The protocol range is not a deployment recommendation. The useful value is the one that fits a service's tolerated stale-routing period, the time needed to prepare a rollback, and the query load its authoritative path can sustain.
There is a separate trap for first-time tenant names. RFC 2308 permits negative caching, using a value derived from the SOA record. Under RFC 2308 section 5, that negative TTL is the smaller of the SOA TTL and the SOA MINIMUM field. If a shopper or crawler asked for northwind.example.com before provisioning completed, some resolvers can retain the negative response. Publishing the record later does not force those resolvers to ask again immediately. Create and verify a new tenant record before exposing the hostname; lowering the record TTL cannot repair a cached absence.
The tempting assumption is that a low positive-record TTL covers this case. It does not.
Work backward from the tenant-routing page
Treat a tenant-domain update as a small deployment with distinct desired, published, and observed states. A queue acknowledgement shows that a worker accepted work. A change response shows that some authority accepted a request. Neither proves that the current authoritative answer is the target associated with this tenant revision.
Start the runbook with a direct authoritative lookup. If it differs from desired state, repair the publication path or roll it back, then stop speculating about resolver behavior. If it matches, record the old TTL and change timestamp; those define the latest time an earlier positive answer might still be valid. Recursive checks belong after that distinction, because they describe user experience rather than the source of truth.
One durable change identifier matters more than a fast retry loop. A retry should reconcile the same requested tenant mapping, not create another competing update. This is where duplicate queue deliveries turn a routine DNS move into an ambiguous incident: two workers can each report success while the final answer reflects only one desired state. The record used for reconciliation needs the tenant, owner name, record type, requested target, and revision. Without that small piece of state, an on-call responder has to infer which write was intended from job logs, retries, and timestamps, then decide whether a late worker is safely repeatable or about to undo the cutover. A revision that is older than the stored desired revision should be rejected or reconciled before it can overwrite the newer mapping; otherwise a delayed duplicate can turn a completed migration back into a stale route.
Make the data model carry the record type as well. A CNAME has different coexistence rules from address records, and DNS name syntax has its own failure modes. A target string that looks plausible in a deployment record is not enough evidence that the resulting RRset is valid.
Add the signal that should have fired earlier
The earlier signal is drift duration: the age of a desired tenant revision whose authoritative answer still disagrees. A raw lookup error is too broad, and a successful publisher response is too weak. The alert should name the tenant, owner name, intended target, record type, change identifier, and revision time so the on-call can compare facts without reconstructing the job history.
Keep pending and drift separate. A mismatch immediately after an expected change is pending; a mismatch that survives the documented window is drifted. This preserves a useful page while the change is still allowed to propagate through caches.
package dnsstate
import "time"
type State string
const (
Pending State = "pending"
Converged State = "converged"
Drifted State = "drifted"
)
func Classify(matches bool, changedAt, now time.Time, window time.Duration) State {
if matches {
return Converged
}
if now.Sub(changedAt) <= window {
return Pending
}
return Drifted
}
The function is intentionally small. It does not decide what the window should be, because that policy must include the pre-change wait, the planned cutover, and the service's tolerated stale-routing interval. Using the same window for every record type and every tenant class can be convenient, but it trades away a meaningful distinction between a planned migration and an unexpected correction.
Choose a threshold that operators can trust
The threshold has a false-positive cost. If it is shorter than the published TTL transition and normal delivery time, a routine cutover pages someone who can do nothing useful yet. That trains the team to wait out real drift. If it is much longer than the business's stale-routing tolerance, the alert is accurate but arrives after the customer impact it was meant to limit.
Use the change plan as the source of the initial threshold, then review pages against the authoritative readback and recorded transition times. The policy should be explicit about the trade-off:
| Situation | DNS posture | Verification before closing the event |
|---|---|---|
| Stable tenant route | Normal service TTL | Desired record matches the authoritative answer |
| Scheduled target move | Lower TTL before the switch, then restore it | The prior TTL had time to expire before cutover |
| New tenant hostname | Publish before announcement | Positive answer exists and negative-cache exposure is considered |
| Emergency correction | Publish the intended state and read it back | Authoritative state is correct before measuring recursive results |
Permanent short TTLs fit systems where the added authoritative-query load and lower cache efficiency are acceptable in exchange for faster unplanned changes. Pre-change lowering fits scheduled moves where the team can wait through the old TTL. Neither policy compensates for a wrong target, a missing certificate, or a route that has not been deployed.
The operational goal is not a clever DNS number. It is a cutover with a known cache horizon, an idempotent change record, and a page that identifies drift instead of reporting vague lookup trouble.
Top comments (0)