DEV Community

SullivanReed1247
SullivanReed1247

Posted on

DNS TTL in Healthtech: Why Changes Aren't Immediate Across Resolver Caches

Treat a DNS edit as a staged rollout, not an immediate mutation. TL;DR: TTL limits how long a caching resolver may reuse an answer after it receives that answer; it does not schedule one global propagation event, flush caches already holding an older answer, or prove that every authoritative server is serving the intended record.

That distinction matters in a healthtech admin console. A domain used for appointment reminders or patient notifications can look correct in the control plane while recursive resolvers still return the previous value. The console should therefore report intent, authoritative publication, and observed recursive answers as separate states. “Saved” is not “visible everywhere.”

What does TTL really control when DNS changes aren't immediate?

The word propagation hides several clocks. First, the record change has to reach the authoritative servers for the zone. Then recursive resolvers that already cached the old answer can continue using it until that cached answer expires. Resolvers that did not cache it may ask an authoritative server sooner, so two users can observe different answers at the same moment without either resolver being broken.

Caches disagree.

TTL is carried in DNS resource records and expressed in seconds; RFC 1035 defines its field as a 32-bit unsigned integer. A recursive resolver decrements the remaining lifetime of its cached copy. It can fetch again after expiry, but expiry is permission to refresh, not a promise that all resolvers will refresh at the same instant. Consider two clinic networks before a notification-domain cutover. Resolver A cached the old answer near the start of its lifetime, Resolver B cached the same answer just before the administrator saved the replacement, and Resolver C had no cached answer. C can see the replacement first, A later, and B last. The record has one configured TTL, yet the three cache entries have different remaining lifetimes because their fetch times differ. An admin console that shows only the newest value erases this timing information and invites an operator to “fix” a change that is already behaving correctly.

There is another edge: absence is cacheable too. RFC 2308 defines negative caching for responses such as a name error. Creating a record does not necessarily make a recently cached negative answer disappear. This is a nasty fit for domain onboarding because the first verification attempt can seed a negative cache shortly before the administrator publishes the record.

Short TTLs narrow some cache windows, but they do not repair a wrong delegation, inconsistent authoritative servers, or an accidentally published record. They also increase query traffic toward authoritative infrastructure. No magic here.

Decision record: model three states, not one

The decision is to store the requested record set as intent, verify the authoritative result independently, and sample recursive resolution as observation. The admin console may move a domain through pending publication, published, and observed, but it must retain the evidence behind each label: queried name, record type, returned values, responding server class, observation time, and TTL remaining when available.

Three invariants keep the workflow honest:

  1. A successful write changes intent; it does not certify public visibility.
  2. Authoritative agreement is required before the system calls a record published.
  3. A recursive mismatch before the prior cache window closes is expected state, not an automatic rollback signal.

The failure boundary is equally important. The DNS controller owns record intent and publication checks. It cannot evict arbitrary recursive caches. Notification delivery should not depend on a single verification lookup, especially when email authentication records are involved: DMARC policy discovery is DNS-based, and receivers evaluate what their own DNS path returns.

Model What the console shows Benefit Failure mode
Save equals live One success state Minimal UI and storage Hides authoritative lag and cached answers
Fixed waiting timer “Ready” after a configured delay Predictable workflow Confuses elapsed time with observed DNS state
Intent plus observations Requested, authoritative, and recursive states Explains drift and supports evidence-based retries Requires polling, timestamps, and careful status wording

The third model costs more engineering effort, but it matches the boundary the system actually has. For a healthtech console, that traceability is worth the extra state because operators need to distinguish “the requested policy is wrong” from “the previous policy is still cached.”

This design has real limitations. Recursive sampling adds storage, query load, and operational complexity, while still never proving what every resolver on the Internet currently returns. That trade-off makes it unsuitable for a small private zone whose clients all use one controlled resolver; direct cache invalidation and a simpler status model fit that boundary better. For public notification domains, the evidence is still useful as long as the console labels it as sampled observation rather than global certainty.

Put the critical path behind evidence

The verifier below illustrates the state transition without tying it to a DNS provider. Its resolver and authoritative reader are injected interfaces; production implementations need bounded timeouts, retry backoff, and durable observations. Values are normalized before comparison because presentation differences should not create false drift.

from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum
from typing import Callable, FrozenSet


class PublicationState(str, Enum):
    PENDING_PUBLICATION = "pending_publication"
    PUBLISHED = "published"
    OBSERVED = "observed"


@dataclass(frozen=True)
class RecordIntent:
    name: str
    record_type: str
    values: FrozenSet[str]


@dataclass(frozen=True)
class CheckResult:
    state: PublicationState
    checked_at: datetime
    authoritative_values: FrozenSet[str]
    recursive_values: FrozenSet[str]


def normalize(values: set[str]) -> FrozenSet[str]:
    return frozenset(value.strip().rstrip(".").lower() for value in values)


def check_publication(
    intent: RecordIntent,
    read_authoritative: Callable[[str, str], set[str]],
    read_recursive: Callable[[str, str], set[str]],
) -> CheckResult:
    expected = normalize(set(intent.values))
    authoritative = normalize(
        read_authoritative(intent.name, intent.record_type)
    )
    recursive = normalize(read_recursive(intent.name, intent.record_type))

    if authoritative != expected:
        state = PublicationState.PENDING_PUBLICATION
    elif recursive != expected:
        state = PublicationState.PUBLISHED
    else:
        state = PublicationState.OBSERVED

    return CheckResult(
        state=state,
        checked_at=datetime.now(timezone.utc),
        authoritative_values=authoritative,
        recursive_values=recursive,
    )
Enter fullscreen mode Exit fullscreen mode

One recursive vantage point is not global proof. Sample a small, documented set of independent recursive paths and display each observation rather than compressing disagreement into a green check. Do not hammer resolvers until the TTL reaches zero; schedule the next useful check from evidence, add jitter, and cap retries. This protects the verification path from becoming its own rate-limit problem.

Wait deliberately.

For changes with a known cutover, lower the TTL before the cutover far enough in advance for older, longer-lived cache entries to expire. After the transition is stable, restore the normal TTL according to the zone's availability and query-load requirements. This technique reduces a cache window. It still cannot compensate for changing the wrong zone or leaving authoritative servers inconsistent.

Failure handling for notification domains

Email authentication makes vague DNS status especially risky. DMARC records are TXT records under _dmarc, and RFC 7489 describes policy discovery through DNS. A malformed or stale policy is therefore more than a cosmetic console discrepancy. Keep the exact record type and owner name in the audit trail, validate syntax before publication, and avoid declaring success from a generic “domain verified” flag.

Rollbacks deserve the same treatment as forward changes. A rollback is another DNS publication with another cache window; it cannot recall the new value from resolvers that already cached it. The operational response should preserve both intended versions, their activation times, and observations. Otherwise an operator sees alternating answers and cannot tell expected cache overlap from a second writer changing the zone.

Alert on durable contradictions, not every temporary mismatch. Useful signals include disagreement among authoritative servers, intent that never appears authoritatively, and recursive observations that remain stale beyond the previously published TTL plus a bounded checking margin. The margin is an operational policy, not a DNS guarantee, so expose it in the status explanation.

Compliance changes the logging choice. DNS record values are public, but admin identities, patient-program labels, internal tenant IDs, and notification metadata may not be. Log the minimum evidence needed to explain publication and access it under the same controls as the rest of the administrative audit trail.

Rejected option and where it still fits

We rejected a fixed “wait one TTL, then mark live” timer. It assumes the current TTL is the lifetime of every old cached answer, overlooks negative caching, and says nothing about authoritative agreement. It also makes a delayed publication look healthy merely because time passed.

The timer remains valid as a user-interface hint: “check again after this time” can reduce pointless polling when the prior TTL is known. It should never be the source of truth. A low-risk internal hostname with one controlled resolver may also use a simpler timer because the cache population and invalidation policy are under one team's control; a public notification domain is a different boundary.

The practical rule is compact: record intent, prove authority, observe recursion, and attach time to every claim. TTL bounds reuse of a particular cached answer. Evidence determines whether the intended DNS state is actually being served.

References

Top comments (0)