Service Discovery in Distributed Systems

#frontend #webdev #react

Service Discovery in Distributed Systems

Service discovery is the mechanism that lets services find one another dynamically instead of hardcoding IP addresses or hostnames, and it is one of the core building blocks of a healthy microservices architecture. A good design balances low-latency routing, fast failure detection, and safe updates as instances scale up, down, or move.

Why discovery matters

In a static system, clients can call fixed endpoints. In a distributed system, service instances are ephemeral, so addresses change frequently and clients need a reliable way to locate healthy targets. Without discovery, teams end up embedding configuration everywhere, which makes deployments brittle and operationally expensive.

Discovery also protects availability. When an instance fails, the registry or load balancer should stop sending traffic to it quickly, rather than letting clients discover the failure one request at a time. That is why health checks and timely catalog updates are part of the architecture, not an optional extra.

Core building blocks

A practical service discovery system usually has four pieces: service instances, a registry, health checking, and a client or proxy that uses the registry. The registry stores the current address and health state of each instance, and health checks keep that catalog accurate. Clients can either query the registry directly or delegate the lookup to an intermediary such as a load balancer or service mesh proxy.

Two common patterns are client-side discovery and server-side discovery. In client-side discovery, the application fetches and caches instance lists, then selects a target itself. In server-side discovery, the client calls a load balancer or gateway, and the proxy chooses a healthy backend instance on its behalf.

Choosing an architecture

Client-side discovery gives you direct control and very low routing overhead, but it pushes complexity into every service that calls others. That can be fine in a small ecosystem with a shared client library, but it becomes painful when many languages and teams are involved.

Server-side discovery is easier for application developers because the proxy handles lookup and routing centrally. The trade-off is that the proxy tier becomes a critical dependency, so you must design it for high availability and scale. A service mesh extends this idea by moving discovery and traffic policy into sidecar proxies, which adds flexibility but also operational complexity.

Pattern	Good for	Main drawback
Client-side discovery	Small-to-medium systems with shared libraries	Every client must implement cache, retry, and selection logic
Server-side discovery	Heterogeneous services and simpler clients	Proxy layer becomes a key dependency
Service mesh discovery	Large platforms needing traffic policy, retries, and observability	Added control-plane and sidecar overhead

A reference design

A robust discovery stack starts with a highly available registry that replicates catalog state between nodes. Consul is a common example: it supports DNS and prepared queries for lookups, and it uses Raft to replicate catalog information across agents. Your services register themselves on startup, renew their lease or heartbeat, and are removed from routing when health checks fail.

A typical request flow looks like this:

Service A starts and registers its address and port with the registry.
The registry runs health checks and stores only healthy instances.
Service B resolves Service A by name through DNS, API, or a proxy.
The client or proxy selects one healthy instance and sends traffic.
If an instance fails, health checks remove it from future routing quickly.

That flow keeps service-to-service communication decoupled from the exact machine layout, which is the main value of discovery.

Failure modes

The hardest problems are not lookup itself but stale data and inconsistent state. If the registry lags behind reality, traffic can be routed to dead instances, so health checks and update propagation need to be fast enough for your failure model. If clients cache too aggressively, they may keep using instance lists that are no longer valid.

Another common issue is registry overload. If every client polls too frequently, discovery becomes a scaling bottleneck, so caching and backoff matter. You also need to think about split-brain behavior and regional failures, because a registry that cannot replicate state safely can make two parts of the system disagree about what is alive.

Building it in code

Here is a simple client-side discovery sketch in Python that caches instances and refreshes them periodically:

import time
import random
import requests

class ServiceDiscovery:
    def __init__(self, registry_url, refresh_ttl=30):
        self.registry_url = registry_url
        self.refresh_ttl = refresh_ttl
        self.cache = []
        self.cache_at = 0

    def refresh(self, service_name):
        r = requests.get(f"{self.registry_url}/services/{service_name}")
        r.raise_for_status()
        self.cache = r.json()["instances"]
        self.cache_at = time.time()

    def get_instance(self, service_name):
        if time.time() - self.cache_at > self.refresh_ttl or not self.cache:
            self.refresh(service_name)
        healthy = [i for i in self.cache if i.get("healthy", True)]
        if not healthy:
            raise RuntimeError("No healthy instances")
        return random.choice(healthy)

def call_user_service():
    sd = ServiceDiscovery("http://registry.internal", refresh_ttl=15)
    instance = sd.get_instance("user-service")
    return requests.get(f"http://{instance['host']}:{instance['port']}/profile")

print(call_user_service().status_code)

For server-side discovery, the application code becomes simpler because it targets a proxy address instead of handling instance selection itself:

import requests

def create_order():
    return requests.post("http://gateway.internal/order", json={
        "user_id": "u_123",
        "sku": "sku_456"
    })

print(create_order().status_code)

Operational advice

Treat service discovery as a production subsystem, not a configuration convenience. Monitor registration success rate, stale-instance rate, health-check latency, and lookup failures so you can spot breakage before it spreads. Use short enough TTLs to react to failures quickly, but not so short that you flood the registry with unnecessary reads.

For small teams, start with a managed or well-known system such as Consul, and keep your client logic thin. For larger platforms, standardize on one discovery approach per environment so teams do not mix incompatible assumptions about caching, retries, and routing.

Rizwan Saleem | https://rizwansaleem.co