Our region router used to live in a PHP array. config/regions.php mapped a two-letter country code to an upstream video index shard, a CDN hostname, and a set of category weights. It worked fine when there was one origin box. It stopped working the day we had four, spread across three continents, and a Cloudflare Worker in front deciding which one to hit.
The failure mode was mundane and expensive. Someone (me) shifted BR from the us-east shard to a new sa-east shard, deployed via FTP to three of the four hosts, and forgot the fourth. For nine hours, roughly a quarter of Brazilian traffic on DailyWatch hit an origin that had no Portuguese-language trending rows cached, served a cold SQLite FTS5 query on every request, and got a 4-second TTFB. Nobody paged us because nothing was down. The graphs just sagged.
The fix was not "deploy more carefully." The fix was to stop treating routing config as code. Routing config changes on a different clock than code does — code ships weekly, region routing changes when a shard gets hot at 3am. Those need different substrates. This is what pushing region routing into etcd looked like, including the parts that were annoying.
Why etcd and Not Redis, Consul, or a Database Table
I evaluated four options and the choice came down to one property: watch semantics with a resumable revision.
-
A
regionstable in SQLite/MySQL. Requires polling. Polling at 5s across N origins is fine, but there's no way to know you missed an update, and there's no atomic multi-key write. Rejected. - Redis pub/sub. Fire-and-forget. If your subscriber is reconnecting during the publish, the message is gone and you're silently stale — exactly the failure we were trying to eliminate. Redis Streams fixes this but then you're building a changelog by hand.
- Consul KV. Genuinely fine. Blocking queries give you long-poll watches with an index. I'd have picked it if we were already running Consul for service discovery. We weren't, and Consul's agent-per-node model is more operational surface than a config store needs.
-
etcd. Watches are streamed over gRPC and carry a global monotonic
revision. If your watcher dies, you reconnect withstart_revision = last_seen + 1and etcd replays everything you missed — or tells you explicitly that history was compacted away, at which point you know to do a full resync. That explicit "you are now stale" signal is the whole ballgame.
etcd also gives you transactions (compare-and-swap across multiple keys) and leases, both of which turned out to matter more than I expected.
The cost: etcd is a Raft cluster. Three nodes minimum for any real availability, five if you want to survive two failures. It is latency-sensitive to disk fsync. Do not put it on the same burstable-IO volume as your database. This is a real operational commitment, not a docker run.
Key Layout Is the Actual Design Work
etcd is a flat sorted byte-keyspace with range queries. There are no directories, only prefix scans. So the key layout is your schema, and getting it wrong means either fetching too much or making N round trips.
What we landed on:
/dw/routing/v1/region/AU -> {"shard":"ap-southeast","cdn":"cdn-ap.","weight":1.0,"tier":"a"}
/dw/routing/v1/region/BR -> {"shard":"sa-east",...}
/dw/routing/v1/region/_default -> {"shard":"us-east",...}
/dw/routing/v1/shard/us-east -> {"origin":"10.4.1.20","healthy":true,"max_rps":800}
/dw/routing/v1/shard/ap-southeast -> {...}
/dw/routing/v1/epoch -> "5121"
/dw/routing/v1/lock/rebalance -> (lease-held, ephemeral)
Four rules that came out of doing this badly first:
-
Version the prefix (
v1). When the value schema changes incompatibly, you writev2alongside, migrate readers, then deletev1. Trying to do in-place schema evolution on a live config store is how you get a split-brain fleet. - Split region → shard from shard → origin. Originally I embedded the origin IP in each region value. Moving one shard's origin meant rewriting 40 keys in a transaction. Now it's one key.
-
Keep values small. etcd's default
--max-request-bytesis 1.5 MiB and every value is replicated through Raft and held in memory. Region config is hundreds of bytes. If you're tempted to put a blob in etcd, put it in object storage and put the URL in etcd. - One key for the epoch. A monotonically increasing counter bumped on every routing change. Cheap way for anything — a health check, a log line, a debug header — to answer "which config generation is this box on?" without diffing the whole map.
The Watcher Sidecar
PHP-FPM under LiteSpeed is a process-per-request model. There is no long-lived process to hold a gRPC watch stream, and no shared memory you'd want to write from a request handler. So the watcher is a separate long-running daemon that maintains a local snapshot file, and PHP reads the snapshot. The daemon is the only thing that talks to etcd.
Go, because the official etcd client is Go and the watch reconnection logic is already correct there:
package main
import (
"context"
"encoding/json"
"log"
"os"
"path/filepath"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
)
const prefix = "/dw/routing/v1/"
type snapshot struct {
Revision int64 `json:"revision"`
Written int64 `json:"written_at"`
Keys map[string]string `json:"keys"`
}
func main() {
out := os.Getenv("ROUTING_SNAPSHOT")
if out == "" {
out = "/var/lib/dailywatch/routing.json"
}
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"10.4.0.11:2379", "10.4.0.12:2379", "10.4.0.13:2379"},
DialTimeout: 5 * time.Second,
DialKeepAliveTime: 30 * time.Second,
DialKeepAliveTimeout: 5 * time.Second,
})
if err != nil {
log.Fatalf("etcd dial: %v", err)
}
defer cli.Close()
state := snapshot{Keys: map[string]string{}}
for {
rev, err := resync(cli, &state)
if err != nil {
log.Printf("resync failed, retrying: %v", err)
time.Sleep(2 * time.Second)
continue
}
if err := write(out, state); err != nil {
log.Printf("snapshot write: %v", err)
}
log.Printf("synced %d keys at revision %d", len(state.Keys), rev)
if err := follow(cli, &state, out); err != nil {
// Compaction or stream death: fall back to a full resync.
log.Printf("watch ended: %v", err)
}
}
}
func resync(cli *clientv3.Client, s *snapshot) (int64, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
resp, err := cli.Get(ctx, prefix, clientv3.WithPrefix())
if err != nil {
return 0, err
}
s.Keys = make(map[string]string, len(resp.Kvs))
for _, kv := range resp.Kvs {
s.Keys[string(kv.Key)] = string(kv.Value)
}
s.Revision = resp.Header.Revision
return s.Revision, nil
}
func follow(cli *clientv3.Client, s *snapshot, out string) error {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
ch := cli.Watch(ctx, prefix,
clientv3.WithPrefix(),
clientv3.WithRev(s.Revision+1),
clientv3.WithPrevKV(),
)
for wr := range ch {
if err := wr.Err(); err != nil {
return err // includes ErrCompacted -> caller does a full resync
}
for _, ev := range wr.Events {
k := string(ev.Kv.Key)
switch ev.Type {
case clientv3.EventTypePut:
s.Keys[k] = string(ev.Kv.Value)
case clientv3.EventTypeDelete:
delete(s.Keys, k)
}
}
s.Revision = wr.Header.Revision
if err := write(out, *s); err != nil {
log.Printf("snapshot write: %v", err)
}
}
return nil
}
// write is atomic: temp file in the same directory, then rename.
func write(path string, s snapshot) error {
s.Written = time.Now().Unix()
buf, err := json.Marshal(s)
if err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(path), ".routing-*")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
if _, err := tmp.Write(buf); err != nil {
tmp.Close()
return err
}
if err := tmp.Sync(); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := os.Chmod(tmp.Name(), 0o644); err != nil {
return err
}
return os.Rename(tmp.Name(), path)
}
Three details that are not decoration:
-
WithRev(s.Revision+1). Without this you start watching from now and silently lose every event between yourGetand yourWatch. That window is small and it will bite you exactly once, in production, at a bad time. -
wr.Err()returningErrCompacted. etcd garbage-collects old revisions. If your watcher was down longer than the compaction window, history is gone. The only correct response is a full resync — which is whyfollowreturns the error up to the retry loop rather than trying to be clever. -
Write temp file,
fsync, thenrename.rename(2)within a filesystem is atomic. PHP-FPM workers reading concurrently will see either the whole old file or the whole new file, never a truncated one. Writing in place would eventually hand some request a half-written JSON document.
Reading the Snapshot from PHP Without Melting the Disk
The PHP side never talks to etcd. It reads a local file, and it does so through APCu with a stat-based freshness check, so the common path is a memory lookup plus one stat().
<?php
declare(strict_types=1);
final class RegionRouter
{
private const SNAPSHOT = '/var/lib/dailywatch/routing.json';
private const CACHE_KEY = 'routing:v1';
private const STALE_AFTER = 900; // seconds
/** @var array{revision:int,written_at:int,keys:array<string,string>}|null */
private static ?array $memo = null;
public function route(string $country): RouteDecision
{
$snap = $this->snapshot();
$cc = strtoupper(substr($country, 0, 2));
$raw = $snap['keys']['/dw/routing/v1/region/' . $cc]
?? $snap['keys']['/dw/routing/v1/region/_default']
?? null;
if ($raw === null) {
return RouteDecision::fallback('no-default-key');
}
$region = json_decode($raw, true, 8, JSON_THROW_ON_ERROR);
$shardRaw = $snap['keys']['/dw/routing/v1/shard/' . $region['shard']] ?? null;
if ($shardRaw === null) {
// Dangling reference: region points at a shard that no longer exists.
return RouteDecision::fallback('dangling-shard:' . $region['shard']);
}
$shard = json_decode($shardRaw, true, 8, JSON_THROW_ON_ERROR);
if (($shard['healthy'] ?? false) !== true) {
return RouteDecision::fallback('shard-unhealthy:' . $region['shard']);
}
return new RouteDecision(
origin: $shard['origin'],
cdnHost: $region['cdn'],
weight: (float) ($region['weight'] ?? 1.0),
revision: $snap['revision'],
);
}
/** @return array{revision:int,written_at:int,keys:array<string,string>} */
private function snapshot(): array
{
clearstatcache(true, self::SNAPSHOT);
$mtime = @filemtime(self::SNAPSHOT);
if ($mtime === false) {
return self::$memo ?? ['revision' => 0, 'written_at' => 0, 'keys' => []];
}
if (self::$memo !== null && self::$memo['written_at'] >= $mtime) {
return self::$memo;
}
$cached = apcu_fetch(self::CACHE_KEY, $ok);
if ($ok && is_array($cached) && $cached['written_at'] >= $mtime) {
return self::$memo = $cached;
}
$body = @file_get_contents(self::SNAPSHOT);
if ($body === false) {
return self::$memo ?? ['revision' => 0, 'written_at' => 0, 'keys' => []];
}
/** @var array{revision:int,written_at:int,keys:array<string,string>} $decoded */
$decoded = json_decode($body, true, 16, JSON_THROW_ON_ERROR);
if (time() - $decoded['written_at'] > self::STALE_AFTER) {
error_log(sprintf(
'routing snapshot stale by %ds (rev %d)',
time() - $decoded['written_at'],
$decoded['revision']
));
}
apcu_store(self::CACHE_KEY, $decoded, 60);
return self::$memo = $decoded;
}
}
final class RouteDecision
{
public function __construct(
public readonly string $origin,
public readonly string $cdnHost,
public readonly float $weight,
public readonly int $revision,
public readonly ?string $degradedReason = null,
) {}
public static function fallback(string $reason): self
{
return new self('127.0.0.1', '', 1.0, 0, $reason);
}
}
The written_at >= $mtime comparison rather than == matters: the daemon can write twice within the same second, and mtime granularity on some filesystems will not distinguish them. Using >= means a stale memo can survive at most until the next second boundary, which for routing config is fine.
The part I'd emphasize to anyone building this: the failure path is the product. RouteDecision::fallback() gets hit when the snapshot is missing, when a region points at a deleted shard, when a shard is marked unhealthy. Each carries a distinct reason string, which we emit as a X-DW-Route-Degraded header on internal requests and count in a Prometheus gauge. When something is wrong you want to know which wrongness, not that routing "failed."
Writing Config Safely with Transactions and Leases
Reads are the easy half. Writes are where you can break the whole fleet in one command, so the control-plane tooling does three things: validates before writing, writes atomically, and holds a lock so two operators can't rebalance simultaneously.
#!/usr/bin/env python3
"""Rebalance regions between shards. Validates, locks, writes atomically."""
import json
import sys
import etcd3 # pip install etcd3-py-client
PREFIX = "/dw/routing/v1/"
LOCK_KEY = PREFIX + "lock/rebalance"
EPOCH_KEY = PREFIX + "epoch"
def load_all(client):
out = {}
for value, meta in client.get_prefix(PREFIX):
out[meta.key.decode()] = value.decode()
return out
def validate(keys):
"""Every region must point at a shard that exists. A default must exist."""
errors = []
shards = {
k.rsplit("/", 1)[1]
for k in keys
if k.startswith(PREFIX + "shard/")
}
if PREFIX + "region/_default" not in keys:
errors.append("missing region/_default")
for key, raw in keys.items():
if not key.startswith(PREFIX + "region/"):
continue
try:
region = json.loads(raw)
except json.JSONDecodeError as exc:
errors.append(f"{key}: invalid json: {exc}")
continue
target = region.get("shard")
if target not in shards:
errors.append(f"{key}: points at unknown shard {target!r}")
if not 0.0 < float(region.get("weight", 1.0)) <= 10.0:
errors.append(f"{key}: weight out of range")
return errors
def rebalance(client, moves):
"""moves: {'BR': 'sa-east', 'CL': 'sa-east'}"""
current = load_all(client)
proposed = dict(current)
for cc, shard in moves.items():
key = f"{PREFIX}region/{cc}"
if key not in proposed:
raise SystemExit(f"unknown region {cc}")
region = json.loads(proposed[key])
region["shard"] = shard
proposed[key] = json.dumps(region, separators=(",", ":"), sort_keys=True)
errors = validate(proposed)
if errors:
for err in errors:
print(f"INVALID: {err}", file=sys.stderr)
raise SystemExit(1)
epoch = int(current.get(EPOCH_KEY, "0"))
# Lock so two operators cannot interleave rebalances.
lease = client.lease(30)
got = client.transaction(
compare=[client.transactions.version(LOCK_KEY) == 0],
success=[client.transactions.put(LOCK_KEY, b"held", lease=lease)],
failure=[],
)
if not got[0]:
raise SystemExit("rebalance already in progress")
try:
changed = [f"{PREFIX}region/{cc}" for cc in moves]
ok, _ = client.transaction(
# Refuse if anyone modified these keys since we read them.
compare=[
client.transactions.value(EPOCH_KEY) == str(epoch).encode()
],
success=[
client.transactions.put(k, proposed[k].encode()) for k in changed
] + [
client.transactions.put(EPOCH_KEY, str(epoch + 1).encode())
],
failure=[],
)
if not ok:
raise SystemExit("epoch changed under us; re-run")
print(f"moved {len(moves)} regions, epoch {epoch} -> {epoch + 1}")
finally:
lease.revoke() # releases the lock even if we blew up above
if __name__ == "__main__":
rebalance(etcd3.client(host="10.4.0.11", port=2379), {"BR": "sa-east"})
What each mechanism buys:
-
The
epochcompare-and-swap turns the write into optimistic concurrency control. Read epoch, compute changes, write only if epoch is unchanged. Two concurrent rebalances: one wins, one gets told to retry against fresh state. No lost updates. - The lease-backed lock is belt-and-braces on top. Leases expire — if the operator's laptop closes mid-run, the lock evaporates after 30 seconds rather than wedging the system forever. This is the thing you cannot build correctly with a plain key.
-
Validation runs against the proposed full state, not the diff. A move is invalid if it creates a dangling reference, and you can only see that by checking the whole map. This one check would have caught the original
BRincident before it shipped.
Operational Things That Actually Bit Us
Compaction defaults will surprise you. etcd retains all revisions until compacted. Run with --auto-compaction-retention=1h (or --auto-compaction-mode=revision --auto-compaction-retention=10000). Without it, the backend DB grows until it hits --quota-backend-bytes (2 GiB default) and the cluster goes read-only with mvcc: database space exceeded. Recovering means compact, then etcdctl defrag, then etcdctl alarm disarm. Do this in a staging cluster once so your hands know the sequence.
Defrag is blocking, per member. etcdctl defrag locks the member while it runs. Do it one member at a time with --endpoints pointed at a single node, never --cluster.
Watch out for the Cloudflare layer disagreeing with etcd. We push a subset of routing state to a Cloudflare Worker KV namespace so edge decisions don't require an origin round trip, and KV is eventually consistent with up to a ~60s propagation delay. For the first month, etcd and edge KV would briefly disagree after a rebalance and requests would ping-pong. The fix was to make the origin authoritative and idempotent: if the edge routes a request to the wrong origin, that origin serves it correctly anyway rather than redirecting. Never let two config planes argue in the request path.
Give the watcher a dead-man's switch. The daemon writes written_at on every snapshot flush, including no-op flushes on a 60-second timer. A separate check alerts if written_at falls more than five minutes behind wall clock. Silent staleness is the exact failure we started with, and a watcher that dies quietly reproduces it perfectly.
Don't put secrets in etcd because it's convenient. Values are encrypted in transit with TLS but at rest they're plaintext in the bbolt file unless you've set up encryption providers. Routing config is not secret. API keys are. Keep them separate.
Client TLS is not optional. etcd with no auth on a private network is one misconfigured security group away from anyone rewriting your routing table. Use --client-cert-auth with per-service certs and RBAC roles scoped to the prefix each service actually needs — the watcher gets read on /dw/routing/v1/, the rebalance tool gets write.
What Changed
Routing changes went from a 20-minute FTP deploy across four hosts, with a real chance of partial application, to a single transaction that every origin picks up in under a second. The epoch counter appears in our health endpoint, so "are all four boxes on the same config?" is one curl per host instead of a diff of four files.
The honest accounting: we added a three-node Raft cluster to operate, a Go daemon per origin, and a class of failure (stale snapshot) that didn't exist before. For a two-server setup this is clearly not worth it — a config file in git and a deploy script is fine, and I'd say so. It became worth it at the point where the number of places config had to land exceeded the number of places a human could reliably check.
If you're heading this way, the two things I'd insist on: version your key prefix from day one, and build the degraded-path reason codes before you build the happy path. The happy path works on the first try. The degraded path is what you'll be reading at 3am.
Top comments (0)