DNS and a service registry solve different parts of internal naming. For a logistics admin console, use DNS for stable identity and a registry for rapidly changing instances; put a bounded cache and an explicit health signal between callers and either system. The trade-off is operational evidence: DNS gives you a durable name, while a registry can show which instance is alive right now.
Short answer: keep a DNS name as the human-facing contract, and add registry lookup only for services whose instances change often enough that TTL-based freshness cannot meet the stale-resolution budget.
A useful rule is simple: if a hostname should survive a deploy, make it DNS-backed; if an endpoint can appear or disappear several times during a deploy, discover it through a registry and still expose a stable service name to humans. Stale resolution is not a single bug. It is the sum of DNS TTL, client caching, registry leases, and rollout timing.
Evidence beats intuition.
The incident lesson: a name can outlive its endpoint
In a bounded production exercise for a logistics console, I traced a failed label lookup to two caches disagreeing. The DNS record still pointed at the old gateway because its TTL was 300 seconds, while the registry had already removed the old instance. A worker retried the same request against the stale address and created duplicate work. The useful signal was not another retry; it was the age of the resolution and the deployment revision attached to it.
The timeline mattered. At 09:10 the rollout controller marked revision 42 ready. At 09:11 the registry lease for revision 41 expired. A resolver in the admin subnet still had the DNS answer from 09:08, so requests kept reaching the old gateway until that cache entry aged out. The retry policy saw connection success followed by an application-level rejection and replayed a label operation that was not idempotent. Once the request ID, resolved address, lease expiry, and revision were logged together, the on-call could distinguish propagation delay from a missing healthy target instead of guessing.
That distinction changes the runbook. Record the resolved address, resolver timestamp, registry lease expiry, and request ID on every admin operation. When a cutover is delayed, operators can tell whether propagation is pending or the registry has no healthy target. I am not sure one universal TTL exists: your mileage will vary with resolver behavior, client libraries, and how much downtime a rollback can tolerate.
The first fix is boring: stop writes, inspect evidence, and replay only operations with an idempotency key. Do not “solve” stale data by adding more retries.
How should internal endpoints handle DNS, service registry, deploy frequency, and stale resolution?
Start with ownership. A DNS zone is a contract for names, delegation, and records. A service registry is a short-lived inventory of instances and metadata. Mixing those roles creates confusing recovery steps: deleting an instance from the registry cannot instantly invalidate a recursive resolver cache, and lowering a DNS TTL does not force an already-running process to resolve again.
For low-frequency deploys, a DNS name with a measured TTL and health-checked load balancer is often easier to audit. For high-frequency deploys, register instances with a lease, require health renewal, and make clients refresh on lease expiry rather than on every request. Keep the public-looking name stable in configuration so a rollback changes routing state, not application code.
The control loop should be observable. Emit metrics for resolution age, registry lease age, failed lookups, and the percentage of calls using an address from the previous deploy. Alert on stale age relative to your cutover budget, not on an arbitrary count. A deploy frequency number by itself is not a decision rule; it matters only beside the maximum stale window and the rollback target. For each metric, define the event that changes the operator's next action: a rising resolution age points to cache propagation, an expired lease points to registry renewal, and a high previous-revision percentage points to an incomplete cutover. Keep those signals on the same dashboard as request IDs and deployment revisions, because a graph without the corresponding name and revision still leaves the on-call reconstructing the incident from logs.
Here is the comparison I put in a design record before choosing a path:
| Approach | Access method (SDK/REST) | Onboarding cost | Best fit | Main limitation |
|---|---|---|---|---|
| Authoritative DNS | Resolver APIs or ordinary DNS clients; no SDK required | Low when zones and delegation already exist | Stable service names, external consumers, infrequent address changes | TTL and client caches can preserve stale answers; health metadata is indirect |
| Service registry | Registry SDK or HTTP API, usually behind a client library | Medium: catalog, leases, health checks, and ACLs must be operated | Frequent deploys, instance churn, and health-aware routing | Requires a clear partition and outage behavior; consumers must understand leases |
| Hybrid boundary | A small internal REST or Go interface hides DNS and registry sources | Medium to high: two control planes and reconciliation | Human-stable names with fast instance turnover | More moving parts and evidence to correlate during rollback |
The table is a starting point, not a scorecard. The right row depends on who consumes the name. An external resolver cannot query a private registry, while an internal worker that needs revision metadata gains little from a DNS-only answer.
A small resolver boundary prevents retry storms
The application should not know whether a target came from DNS, a registry, or a test double. This Go boundary keeps freshness and idempotency decisions in one place:
package discovery
import (
"context"
"fmt"
"time"
)
type Target struct {
Address string
Revision string
Expires time.Time
}
type Resolver interface {
Resolve(context.Context, string) (Target, error)
}
func ResolveFor(ctx context.Context, r Resolver, name string, now time.Time) (Target, error) {
target, err := r.Resolve(ctx, name)
if err != nil {
return Target{}, fmt.Errorf("resolve %s: %w", name, err)
}
if !target.Expires.After(now) {
return Target{}, fmt.Errorf("resolution for %s is stale", name)
}
return target, nil
}
The caller should attach a request ID and deployment revision, then retry only operations that are idempotent. A stale target is a control-plane signal; treating it as a generic network error can multiply traffic while the rollout is already under pressure. The resolver boundary also gives tests a deterministic clock, so a case at the exact expiry instant is covered rather than left to a live DNS cache.
Keep cache policy outside the resolver interface. A DNS adapter can honor the authoritative TTL, while a registry adapter can cap lease age and discard an expired health result. Both adapters should return the same evidence fields to logs and traces. This is where a little discipline pays off: the admin console can show “resolved 47 seconds ago, revision 42” instead of a vague “lookup failed.”
Comparing operating models without vendor folklore
Managed DNS and self-operated DNS differ in delegation and audit workflows, but neither removes the need to reason about TTL and cache behavior. Registry implementations differ in health checks, lease semantics, and partition handling. Treat those as engineering trade-offs, not a ranking exercise.
A managed DNS API can reduce zone-maintenance work while leaving client caches outside your control. A registry can return fresher membership while requiring disciplined deregistration and a clear behavior during partitions. In a regulated logistics environment, the audit trail for who changed a record may matter as much as lookup latency.
The catch is that a registry is not suitable when the consumer is an external resolver or when you need a name to remain valid during a registry outage. Stick with DNS when the contract is long-lived and human-operated; add a registry when instance churn and health metadata are first-class requirements. A hybrid is also a poor fit for a small team that cannot operate two control planes or rehearse their failure modes.
Decision record for the admin console
Write down four numbers before choosing: expected deploys per day, acceptable stale-resolution window, rollback completion target, and the evidence an operator must see before declaring a cutover complete. Test those numbers with a rehearsal that includes recursive-cache delay, an expired lease, and a partial rollout. Capture the resolver timestamp and revision in the test output so the rehearsal checks evidence, not just request success.
For mail-related domains, publish and monitor SPF, DKIM, and DMARC records as separate deliverability controls. DMARC reporting is specified in RFC 7489. Internal endpoint discovery does not replace those records, and a successful internal lookup is not evidence that customer mail will be delivered.
A good design leaves one boring path for recovery: freeze writes, inspect resolution age and health evidence, route back to the last known revision, and replay only idempotent admin actions. That path matters more than whether the first lookup came from DNS or a registry.
Top comments (0)