Using etcd to Coordinate Video Region Routers Across a GDPR Fleet

At 03:12 on a Tuesday, viewers in Warsaw started getting served from our Frankfurt edge while Frankfurt itself was being drained for a kernel upgrade. Latency doubled, our SQLite-backed analytics showed a spike in abandoned playbacks, and the on-call (me) spent twenty minutes discovering that three of our region routers had a different idea of which edge owned Poland. One node had the old config, two had the new one, and the drain script had only told half the fleet. That mismatch is the whole problem this article is about.

A "region router" at ViralVidVault is the small service that decides, for a given viewer, which video-serving edge should handle their session. It is not the CDN. It sits in front of the CDN and encodes our policy: which country maps to which edge pool, which pools are draining, what the GDPR data-residency rules force for a given viewer, and what weight each pool currently gets. That policy changes constantly — an edge goes into maintenance, a viral clip overloads a pool, a legal rule shifts where EU traffic may terminate. When policy lives in a config file baked into each node, every change is a race, and races at the routing layer are outages.

We moved that policy into etcd. This post is the concrete design: how we model the keyspace, the Go control-plane watcher that turns key events into an in-memory routing table, how our PHP 8.4 front door reads the same config without holding a gRPC connection, and how we bootstrap and validate everything from Python. I will also be honest about the failure modes we hit, because "just use etcd" is the kind of advice that sounds finished and isn't.

What a region router actually decides

Before any distributed-systems machinery, it helps to pin down what the router computes. For each incoming request it needs, in order:

• The viewer's country, derived from the Cloudflare CF-IPCountry header (we run Cloudflare Workers at the very front).
• The data-residency class for that country. EU/EEA viewers must terminate on EU edges only — this is not a performance choice, it is a GDPR constraint we are contractually bound to.
• The set of candidate edge pools that satisfy residency and are currently active (not draining or down).
• A weight per pool, so we can shift load gradually instead of flipping 100% of a country at once.

The output is a single edge hostname the Worker or PHP layer redirects/proxies to. The decision itself is cheap. The hard part is that the inputs — pool health, weights, residency overrides — must be identical across every node within a second or two of a change, and must survive a node restart without a stale snapshot leaking through.

Why etcd and not a database column

We already run SQLite in WAL mode on every node for analytics and content metadata, so the obvious lazy answer is "add a routing_config table." We tried that first. It fails for a specific reason: SQLite is per-node. To share it you need a replication story, and once you're replicating a mutable config table across nodes with read-your-writes semantics and change notifications, you have rebuilt a worse etcd.

etcd gives us four things that matter here:

• A linearizable key-value store backed by Raft, so a write is either committed to a quorum or it isn't. There is no "half the fleet" state.
• Watches: a client can stream every change to a key prefix and know it has seen them in order, with a revision number to detect gaps.
• Leases: a key can be tied to a lease that must be renewed, so a pool that stops heartbeating automatically disappears from config. This is how draining becomes down without a human.
• Compare-and-swap transactions, so the control plane can change a weight only if the current value is what it expected — no lost updates when two operators touch the fleet at once.

We are not storing much data. The entire routing config is a few kilobytes. etcd is not a database here; it is a coordination primitive with exactly the consistency guarantee the routing layer needs.

Modeling routing config as an etcd keyspace

Keyspace design is where most etcd projects quietly go wrong. The temptation is to store one big JSON blob under /routing/config. Don't. A blob means every change rewrites everything, every watcher re-parses everything, and two operators editing different pools collide on the same key. Model the config as many small keys so writes are granular and watches are cheap.

Our layout:

• /vvv/pools/<pool_id> → JSON: { "region": "eu-central", "host": "edge-fra-1.vvv.internal", "status": "active", "residency": "eu" }
• /vvv/route/<country> → JSON: { "pools": [ {"pool": "eu-central", "weight": 70}, {"pool": "eu-west", "weight": 30} ] }
• /vvv/health/<pool_id> → lease-bound key written by each pool's heartbeat; absence means unhealthy.

A pool going into maintenance is one small write to one pools/<id> key. A weight shift for Poland is one write to route/pl. Nothing else is touched, and watchers only wake for the keys that changed. Country codes are lowercased ISO-3166 alpha-2 so CF-IPCountry maps directly after a strtolower.

A Go control-plane watcher

The control plane is a small Go service running on every router node. It watches the /vvv/ prefix, folds events into an in-memory routing table, and swaps the table atomically so request handlers never see a half-updated state. Go's official go.etcd.io/etcd/client/v3 makes the watch loop straightforward; the discipline is in the atomic swap and in re-syncing from the current revision on reconnect.

package main

import (
    "context"
    "encoding/json"
    "log"
    "sync/atomic"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
)

type Pool struct {
    Region    string `json:"region"`
    Host      string `json:"host"`
    Status    string `json:"status"`
    Residency string `json:"residency"`
}

type Table struct {
    Pools map[string]Pool            // pool_id -> Pool
    Route map[string][]WeightedPool  // country -> weighted pools
}

type WeightedPool struct {
    Pool   string `json:"pool"`
    Weight int    `json:"weight"`
}

// current holds a *Table; readers load it lock-free.
var current atomic.Pointer[Table]

func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"http://127.0.0.1:2379"},
        DialTimeout: 5 * time.Second,
    })
    if err != nil {
        log.Fatalf("etcd dial: %v", err)
    }
    defer cli.Close()

    for {
        if err := syncAndWatch(cli); err != nil {
            log.Printf("watch loop ended: %v; retrying in 2s", err)
            time.Sleep(2 * time.Second)
        }
    }
}

func syncAndWatch(cli *clientv3.Client) error {
    ctx := context.Background()

    // 1. Full snapshot at a known revision.
    resp, err := cli.Get(ctx, "/vvv/", clientv3.WithPrefix())
    if err != nil {
        return err
    }
    tbl := &Table{Pools: map[string]Pool{}, Route: map[string][]WeightedPool{}}
    for _, kv := range resp.Kvs {
        apply(tbl, string(kv.Key), kv.Value)
    }
    current.Store(tbl)
    log.Printf("synced %d keys at rev %d", len(resp.Kvs), resp.Header.Revision)

    // 2. Watch from the revision AFTER the snapshot — no gap, no dup.
    watchCh := cli.Watch(ctx, "/vvv/", clientv3.WithPrefix(),
        clientv3.WithRev(resp.Header.Revision+1))

    for wr := range watchCh {
        if wr.Err() != nil {
            return wr.Err() // triggers full resync
        }
        // Copy-on-write: clone, apply, atomic swap.
        old := current.Load()
        next := cloneTable(old)
        for _, ev := range wr.Events {
            apply(next, string(ev.Kv.Key), ev.Kv.Value)
        }
        current.Store(next)
    }
    return nil
}

func apply(t *Table, key string, val []byte) {
    switch {
    case len(val) == 0: // deletion
        return
    case hasPrefix(key, "/vvv/pools/"):
        var p Pool
        if json.Unmarshal(val, &p) == nil {
            t.Pools[key[len("/vvv/pools/"):]] = p
        }
    case hasPrefix(key, "/vvv/route/"):
        var r struct{ Pools []WeightedPool `json:"pools"` }
        if json.Unmarshal(val, &r) == nil {
            t.Route[key[len("/vvv/route/"):]] = r.Pools
        }
    }
}

Two details earn their keep. First, we watch from resp.Header.Revision+1, so the snapshot and the watch stream join without a gap or a duplicated event — the classic etcd correctness bug is watching from revision 0 and missing changes that happened during the Get. Second, atomic.Pointer[Table] means a request handler that grabbed the old table mid-swap keeps a fully consistent old snapshot rather than reading a torn map. There are no locks on the read path.

The control plane exposes the resolved decision over a Unix socket on localhost, which is what the PHP front door and the local health checks query. Keeping the etcd client in one Go process per node — rather than in every PHP worker — is deliberate, and it's the next section.

Reading config from PHP without a persistent gRPC connection

Our request front door is PHP 8.4 behind LiteSpeed. PHP's process model is hostile to long-lived streaming connections: each request is a fresh worker, so having every PHP worker open an etcd watch would mean hundreds of watches per node and a thundering herd on every reconnect. So PHP never talks to etcd directly. It reads the resolved table from the local Go control plane over the Unix socket, and it keeps a short-lived local cache in SQLite (WAL mode) so that if the control plane blips, the front door serves the last-known-good config instead of failing.

<?php
declare(strict_types=1);

final class RegionRouter
{
    private const SOCKET = '/run/vvv/control.sock';
    private const CACHE_TTL = 5; // seconds

    public function __construct(private \PDO $sqlite) {}

    public function resolveEdge(string $country): string
    {
        $country = strtolower($country);
        $table = $this->loadTable();
        $route = $table['route'][$country] ?? $table['route']['default'];

        // Filter to active pools that satisfy residency, keep weights.
        $candidates = [];
        foreach ($route as $wp) {
            $pool = $table['pools'][$wp['pool']] ?? null;
            if ($pool !== null && $pool['status'] === 'active') {
                $candidates[] = ['host' => $pool['host'], 'weight' => $wp['weight']];
            }
        }
        if ($candidates === []) {
            throw new \RuntimeException("no active edge for {$country}");
        }
        return $this->weightedPick($candidates);
    }

    private function loadTable(): array
    {
        // Fast path: local SQLite cache still fresh?
        $row = $this->sqlite->query(
            'SELECT payload, ts FROM route_cache WHERE id = 1'
        )->fetch(\PDO::FETCH_ASSOC);

        if ($row !== false && (time() - (int)$row['ts']) < self::CACHE_TTL) {
            return json_decode($row['payload'], true, flags: JSON_THROW_ON_ERROR);
        }

        // Slow path: ask the local Go control plane over the Unix socket.
        try {
            $payload = $this->fetchFromControlPlane();
            $stmt = $this->sqlite->prepare(
                'INSERT INTO route_cache(id, payload, ts) VALUES(1, :p, :t)
                 ON CONFLICT(id) DO UPDATE SET payload = :p, ts = :t'
            );
            $stmt->execute([':p' => $payload, ':t' => time()]);
            return json_decode($payload, true, flags: JSON_THROW_ON_ERROR);
        } catch (\Throwable $e) {
            // Control plane down: serve last-known-good, even if stale.
            if ($row !== false) {
                error_log('route: serving stale config: ' . $e->getMessage());
                return json_decode($row['payload'], true, flags: JSON_THROW_ON_ERROR);
            }
            throw $e;
        }
    }

    private function fetchFromControlPlane(): string
    {
        $fp = @stream_socket_client('unix://' . self::SOCKET, $errno, $errstr, 0.2);
        if ($fp === false) {
            throw new \RuntimeException("control plane unreachable: {$errstr}");
        }
        stream_set_timeout($fp, 0, 200_000); // 200ms
        fwrite($fp, "GET /table\n");
        $body = stream_get_contents($fp);
        fclose($fp);
        return $body;
    }

    private function weightedPick(array $candidates): string
    {
        $total = array_sum(array_column($candidates, 'weight'));
        $r = random_int(1, max(1, $total));
        foreach ($candidates as $c) {
            $r -= $c['weight'];
            if ($r <= 0) {
                return $c['host'];
            }
        }
        return $candidates[0]['host'];
    }
}

The layering here is the point. etcd is the source of truth. The Go control plane is the only etcd client on the node and holds the live watch. PHP reads a resolved table and caches it in SQLite WAL for 5 seconds. WAL mode matters: readers don't block the single writer, so a burst of concurrent requests all reading route_cache never contends. And the catch block is the difference between "the control plane restarted" being invisible and being an outage — stale-but-consistent config beats no config every time at the routing layer.

Bootstrapping and validation in Python

Operators change config through a Python CLI, never by poking etcd by hand. The CLI validates the whole config as a graph before writing — the invariant that must never break is that every country routes only to pools whose residency is legal for it. A single fat-fingered put that sends EU traffic to a US pool is a GDPR incident, not a latency blip, so validation runs client-side and the write uses a transaction.

import json
import sys
import etcd3  # python-etcd3

EU_COUNTRIES = {"de", "fr", "pl", "nl", "es", "it", "se", "at", "be", "ie"}

def load_pools(client):
    pools = {}
    for value, meta in client.get_prefix("/vvv/pools/"):
        pool_id = meta.key.decode().rsplit("/", 1)[-1]
        pools[pool_id] = json.loads(value)
    return pools

def validate_route(country, route, pools):
    errors = []
    weight_total = sum(wp["weight"] for wp in route["pools"])
    if weight_total != 100:
        errors.append(f"{country}: weights sum to {weight_total}, expected 100")
    for wp in route["pools"]:
        pool = pools.get(wp["pool"])
        if pool is None:
            errors.append(f"{country}: unknown pool {wp['pool']}")
            continue
        # The GDPR invariant: EU viewers only ever hit eu-residency pools.
        if country in EU_COUNTRIES and pool["residency"] != "eu":
            errors.append(
                f"{country}: RESIDENCY VIOLATION -> {wp['pool']} "
                f"is residency={pool['residency']}"
            )
    return errors

def put_route(client, country, route):
    pools = load_pools(client)
    errors = validate_route(country, route, pools)
    if errors:
        print("REFUSING WRITE:", *errors, sep="\n  ")
        sys.exit(1)

    key = f"/vvv/route/{country}"
    payload = json.dumps(route, separators=(",", ":"))
    # Compare-and-swap: only write if we read what we expected, or if absent.
    existing, _ = client.get(key)
    ok = client.transaction(
        compare=[client.transactions.value(key) == (existing or b"")],
        success=[client.transactions.put(key, payload)],
        failure=[],
    )
    if not ok:
        print(f"{country}: config changed under us, re-run")
        sys.exit(2)
    print(f"{country}: routing updated -> {payload}")

if __name__ == "__main__":
    c = etcd3.client(host="127.0.0.1", port=2379)
    new_route = {"pools": [{"pool": "eu-central", "weight": 60},
                           {"pool": "eu-west", "weight": 40}]}
    put_route(c, "pl", new_route)

The transaction with a compare on the current value is what prevented my 03:12 incident from ever recurring. If two operators drain the same country simultaneously, one write wins and the other is told to re-read and retry, instead of silently clobbering. Validation being client-side is a deliberate trade-off: etcd has no schema, so the guarantee is only as good as the tool everyone uses — which is why the raw etcdctl put command is locked down in production and the Python CLI is the only sanctioned path.

Handling the GDPR edge cases

Data residency is not a nice-to-have for a European product; it is the reason the routing layer exists at all. A few things we learned encoding it:

• Residency is a property of the pool, not the route. We store residency on each pools/<id> key. That way a route can never accidentally grant residency it doesn't have — the pool decides, and validation enforces the join.
• Default routes must be residency-safe. Our /vvv/route/default key points only at EU pools. An unknown or spoofed country falls back to EU-only, never to a US edge. Failing closed is the correct default when the legal downside is asymmetric.
• Health leases respect residency too. When an EU pool's lease expires and it drops out, the router does not silently spill EU traffic to a non-EU pool to keep availability up. It returns fewer candidates, and if the candidate set is empty we serve an error rather than violate residency. Availability never overrides the legal constraint.
• The Cloudflare Worker is the residency floor. The Worker reads CF-IPCountry and refuses to even reach the origin for EU traffic if no EU edge is advertised, so a bug in the PHP layer cannot leak EU sessions to the wrong region.

Failure modes we actually hit

Running this for a while surfaced problems the tutorials don't mention:

• Watch gaps on reconnect. Early on we watched from revision 0 after a disconnect and re-processed the entire history, which was harmless but wasteful — until a compaction meant revision 0 was gone and the watch failed outright. The fix is the snapshot-then-watch-from-next-revision pattern in the Go code above.
• etcd compaction. etcd keeps every revision until you compact. Left alone, the DB grows until it hits the space quota and goes read-only, which for us meant "config is frozen." We run auto-compaction on a 1-hour retention and alert on etcd_mvcc_db_total_size_in_bytes.
• Clock-independent leases, human-dependent renewals. A pool's health key is lease-bound, but the renewal runs in the pool's own agent. When that agent hung (not the pool), the key expired and the pool was pulled from rotation while perfectly healthy. We now renew from a supervised sidecar and treat lease loss as a warning to investigate, not an instant hard removal, with a short grace window.
• PHP stale-cache stampede. When the 5-second cache expired under high concurrency, dozens of workers hit the control-plane socket at once. We added a tiny jitter to the TTL per worker and let the SQLite write act as a soft lock, which flattened the spike.

Conclusion

The honest summary is that etcd did not make our routing smart — our routing logic is a weighted pick over a filtered candidate set, which is boring on purpose. What etcd made routing is consistent: every node on the fleet agrees on which edge owns which country within a second of a change, a config write either commits to a quorum or fails loudly, and a drained pool disappears on its own when its lease lapses. The layering — etcd as source of truth, one Go watcher per node, PHP reading a resolved table from a local socket with a WAL-mode fallback cache, and a Python CLI that refuses residency-violating writes — keeps each part doing the one thing it's good at.

If you're building anything where nodes must agree on mutable policy and disagreement is an outage, don't reach for a shared database column and don't invent your own gossip. Model the config as small keys, watch a prefix, fail closed on the constraints that carry legal weight, and always keep a last-known-good copy close to the request path. The 03:12 page is a good teacher, but etcd is a cheaper one.