I built a leaderless service-discovery system in Go. Here's what broke along the way

Manzi Niyongira Osee — Tue, 27 Oct 2026 21:29:18 +0000

For the past few weeks I've been building Atlas, a distributed service-discovery system in Go. It answers one question every microservice setup eventually asks:

Where is the payments service right now, and which copies of it are actually alive?

Most tools answer that with a central database or a leader node (Consul's servers, etcd, ZooKeeper). I wanted to see how far I could get without any leader at all.

🔗 GitHub: https://github.com/manziosee/atlas
🟢 Live demo (read-only): open the dashboard

How it works

Every node runs the same ~14 MB binary. Nodes find each other with SWIM gossip, the same family of protocols behind HashiCorp's memberlist:

Each node pings a random peer every second.
No answer? It asks up to three other nodes to try (ping-req).
Still nothing? The peer is marked suspect, and the rumour spreads by gossip.
If the peer is actually alive, it refutes by bumping its incarnation number. Otherwise it's declared dead within seconds.

On top of that sits a service registry. Every node stores registrations in SQLite and replicates them to every other node over the same gossip. So any node can answer "where is payments?" from its own memory, with no network hop to a central store.

curl -H "Authorization: Bearer $TOKEN" \
  "https://atlas-edge.fly.dev/v1/lookup/payments?tag=primary"

Three bugs that taught me the most

1. The cluster that never healed.
My chaos test cut a node off the network, waited for the others to declare it dead, then plugged it back in. It stayed dead. Forever. Each side had marked the other dead, gossip only goes to live members, and nodes only contacted their seeds at startup. Nothing ever crossed the healed link. The fix was small (keep pulling state from a random seed every 30 seconds), but finding it meant really understanding the protocol.

2. Health checks that flapped.
One slow response marked a service unhealthy; the next one marked it healthy again. Now a service needs 3 consecutive failures to go down and 2 passes to come back, and checks can be HTTP, TCP or gRPC-health.

3. "Only the owner can update this."
A registration belongs to the node that accepted it. That's simple and consistent, but clients behind a load balancer kept hitting the wrong node and getting 409. Now every node gossips its API address and quietly forwards the write to the owner, using the caller's own credential.

What's in it today

Membership with failure detection, suspicion, refutation and partition recovery
Replicated registry with TTL heartbeats, idempotent registration, If-Match updates and draining for zero-downtime deploys
Routing that prefers healthy instances in your region, with consistent hashing
HTTP + typed gRPC APIs, DNS (A/AAAA/SRV), and a live event stream you can resume
Client-side load balancing in Go: http://payments.atlas/... and grpc.NewClient("atlas:///payments") just work
Go and Python clients, a CLI, and an embedded dashboard
Encrypted gossip (AES-256-GCM), scoped and expiring tokens, signed releases, CI with race tests, chaos tests and a 1,000-node simulation

transport := atlas.NewTransport(client, nil, atlas.InstanceOptions{Region: "eu"})
httpClient := &http.Client{Transport: transport}
resp, err := httpClient.Get("http://payments.atlas/v1/charges") // round-robin across live instances

Honest positioning

Atlas isn't trying to replace Consul. Consul, Netflix's Eureka and Fly.io's Corrosion inspired a lot of it. The goal was to build these ideas from first principles and make the result small, secure by default and easy to run on an edge platform. Building it taught me more about distributed systems than anything I've read.

Want to help?

It's open source (Apache 2.0), and there's real work for any level:

🟢 Good first issues: runnable examples, docs, a Node.js/TypeScript client
🟡 Medium: a cluster map with live latency in the dashboard, TLS rotation tests
🔴 Deep end: chaos tests with packet loss, a multi-region demo, per-node cryptographic identity

👉 https://github.com/manziosee/atlas/issues

Feedback on the design is just as welcome as code. If something looks wrong to you, it probably is, and I'd love to know.