Coordinating Video Region Routers With etcd for Live Config Changes

The 3 a.m. region flap that started this

At ViralVidVault our region routers decide two things for every incoming viewer: which video-metadata pool to query (Frankfurt, Amsterdam, or a fallback in Paris) and which Cloudflare edge behavior to apply. The routing map is keyed by country, because a Polish viewer and a Portuguese viewer do not have the same nearest healthy pool, and because our GDPR posture means EU traffic must stay pinned to EU processing regions.

For a long time that map lived in a PHP config array baked into the deploy. It worked until the Frankfurt metadata pool started returning 90th-percentile latencies north of 2 seconds one night. To move DE, AT, and CH traffic off Frankfurt, I had to edit a file, redeploy across every LiteSpeed origin, and pray the origins converged before the pool fully tipped over. That is a 4-8 minute cycle, and it is not atomic: for a window of a minute or two, some origins route DE to Frankfurt and some to Amsterdam. When you are measuring viral-spike traffic in requests per second, an inconsistent region map is its own outage.

The fix was to stop treating routing config as code and start treating it as coordinated state. This is exactly what etcd exists for: a strongly consistent, watchable key-value store where a single PUT propagates to every watcher in well under a second. This post is how we wired PHP 8.4 origins, a Go router sidecar, and Cloudflare Workers to one etcd source of truth, and the mistakes I made getting there.

Why etcd and not just a database row

We already run SQLite in WAL mode on every origin, so the obvious lazy answer is "put the region map in a table and poll it." I tried that first. Two problems killed it.

The first is that SQLite is per-origin. Each LiteSpeed box has its own file. To make a config change I would have to write to N databases and hope they all succeeded, which is precisely the non-atomic mess I was trying to escape. The second is polling latency versus load. To get sub-second reaction you poll every 500ms on every worker process, which is a lot of wasted queries for config that changes a few times a week but must change fast when it does.

etcd solves both. It gives you:

• One authoritative copy with a linearizable read guarantee, so there is no "which origin has the truth" question.
• Watch streams instead of polling. You open a long-lived watch on a key prefix and etcd pushes revisions to you the moment they commit.
• Leases, which let a pool register itself with a TTL. If the pool process dies or its health-checker stops renewing, the key evaporates and every watcher sees it disappear. That is automatic failover for free.
• A revision number on every key, which is a monotonic logical clock. You can detect stale reads and reject an update that raced against a newer one.

The thing etcd is not is a general datastore. It is designed for small, high-value coordination data. Our entire region map is under 4 KB. That fits etcd's model perfectly and would be an abuse of it if we tried to shove video metadata in there.

Modeling the region map as etcd keys

I keep the layout boringly flat. Everything lives under /vvv/routing/. Country routing decisions are one key per ISO country code, and pool health is one key per pool registered with a lease. Here is the shape, expressed as etcdctl commands so it is copy-pasteable:

# Static routing intent: which pool each country prefers, plus a fallback chain
etcdctl put /vvv/routing/country/DE '{"primary":"fra-1","fallbacks":["ams-1","par-1"]}'
etcdctl put /vvv/routing/country/PL '{"primary":"ams-1","fallbacks":["fra-1","par-1"]}'
etcdctl put /vvv/routing/country/_default '{"primary":"ams-1","fallbacks":["par-1"]}'

# Live pool health, written by each pool's health-checker under a 15s lease
LEASE=$(etcdctl lease grant 15 | awk '{print $2}')
etcdctl put --lease=$LEASE /vvv/routing/pool/fra-1 '{"status":"healthy","p90_ms":180,"region":"eu-central"}'

# Read everything back for a given prefix
etcdctl get --prefix /vvv/routing/

The separation matters. country/* keys are intent — human-edited, durable, no lease. pool/* keys are liveness — machine-written, short lease, self-healing. A router combines them: it reads the country's preferred primary, then checks whether that pool's liveness key still exists and is healthy. If not, it walks the fallback chain. No human has to edit the country map during an incident; the pool simply stops renewing its lease and traffic drains itself.

The Go router sidecar that watches for changes

The hot-path routing decision runs in a small Go sidecar next to each origin, because Go's etcd client library handles watches, reconnects, and lease semantics far more cleanly than anything I wanted to write in PHP. The sidecar keeps an in-memory copy of the whole /vvv/routing/ prefix, refreshed by a watch, and answers a Unix socket query in microseconds.

package main

import (
    "context"
    "encoding/json"
    "log"
    "sync"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
)

type Router struct {
    mu    sync.RWMutex
    store map[string]string // full key -> raw JSON value
}

func (r *Router) apply(key, val string, deleted bool) {
    r.mu.Lock()
    defer r.mu.Unlock()
    if deleted {
        delete(r.store, key)
        return
    }
    r.store[key] = val
}

// PickPool resolves a country to a live pool, walking the fallback chain.
func (r *Router) PickPool(cc string) string {
    r.mu.RLock()
    defer r.mu.RUnlock()

    raw, ok := r.store["/vvv/routing/country/"+cc]
    if !ok {
        raw = r.store["/vvv/routing/country/_default"]
    }
    var rule struct {
        Primary   string   `json:"primary"`
        Fallbacks []string `json:"fallbacks"`
    }
    json.Unmarshal([]byte(raw), &rule)

    for _, pool := range append([]string{rule.Primary}, rule.Fallbacks...) {
        if pv, ok := r.store["/vvv/routing/pool/"+pool]; ok {
            var h struct{ Status string `json:"status"` }
            json.Unmarshal([]byte(pv), &h)
            if h.Status == "healthy" {
                return pool
            }
        }
    }
    return "par-1" // last-resort hardcoded floor
}

func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"127.0.0.1:2379"},
        DialTimeout: 5 * time.Second,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()

    r := &Router{store: map[string]string{}}

    // Snapshot current state, then watch from that exact revision onward.
    resp, err := cli.Get(context.Background(), "/vvv/routing/", clientv3.WithPrefix())
    if err != nil {
        log.Fatal(err)
    }
    for _, kv := range resp.Kvs {
        r.apply(string(kv.Key), string(kv.Value), false)
    }

    watchCh := cli.Watch(context.Background(), "/vvv/routing/",
        clientv3.WithPrefix(), clientv3.WithRev(resp.Header.Revision+1))

    log.Printf("router ready, DE -> %s", r.PickPool("DE"))
    for wr := range watchCh {
        for _, ev := range wr.Events {
            r.apply(string(ev.Kv.Key), string(ev.Kv.Value), ev.Type.String() == "DELETE")
        }
        log.Printf("config updated, DE -> %s", r.PickPool("DE"))
    }
}

The subtlety worth calling out is WithRev(resp.Header.Revision+1). You take the snapshot's revision from the Get response and start the watch exactly one revision later. That closes the race where a change lands between your initial read and your watch registration. Miss that and you can silently drop an update on startup, which is the kind of bug that only bites during the incident you built this for.

Feeding PHP 8.4 without hammering etcd on every request

The PHP origins do the actual page rendering, and they need the region decision too. I did not want every PHP worker opening a gRPC connection to etcd — that is the wrong tool for a stateless request-per-process runtime, and it adds a network hop to the hot path. Instead I let a single Python daemon per origin mirror etcd into the local SQLite WAL database that PHP already reads. PHP never talks to etcd; it reads a local, memory-mapped SQLite file that is always at most one revision behind.

#!/usr/bin/env python3
"""Mirror etcd /vvv/routing/ into local SQLite (WAL) for PHP to read."""
import json
import sqlite3
import etcd3  # pip install etcd3

DB = "/var/lib/vvv/routing.sqlite"
PREFIX = "/vvv/routing/"


def ensure_schema(conn):
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute(
        "CREATE TABLE IF NOT EXISTS routing (key TEXT PRIMARY KEY, val TEXT, rev INTEGER)"
    )
    conn.commit()


def upsert(conn, key, val, rev):
    conn.execute(
        "INSERT INTO routing(key, val, rev) VALUES(?,?,?) "
        "ON CONFLICT(key) DO UPDATE SET val=excluded.val, rev=excluded.rev",
        (key, val, rev),
    )
    conn.commit()


def main():
    conn = sqlite3.connect(DB)
    ensure_schema(conn)
    client = etcd3.client(host="127.0.0.1", port=2379)

    # Initial sync so PHP has a complete map immediately.
    for value, meta in client.get_prefix(PREFIX):
        upsert(conn, meta.key.decode(), value.decode(), meta.mod_revision)

    # Stream changes forever; each event is one committed revision.
    events, _ = client.watch_prefix(PREFIX)
    for ev in events:
        key = ev.key.decode()
        if isinstance(ev, etcd3.events.DeleteEvent):
            conn.execute("DELETE FROM routing WHERE key=?", (key,))
            conn.commit()
        else:
            upsert(conn, key, ev.value.decode(), ev.mod_revision)


if __name__ == "__main__":
    main()

Because SQLite WAL allows concurrent readers while the writer commits, PHP workers never block on the mirror daemon and never see a half-written row. The PHP side is then trivial and, importantly, has zero external dependencies on the request path:

<?php
declare(strict_types=1);

final class RegionRouter
{
    private const DB_PATH = '/var/lib/vvv/routing.sqlite';
    private const FLOOR_POOL = 'par-1';

    public function __construct(private readonly \PDO $db) {}

    public static function open(): self
    {
        $pdo = new \PDO('sqlite:' . self::DB_PATH, options: [
            \PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION,
        ]);
        // Read-only-ish: never let a page request write the mirror.
        $pdo->exec('PRAGMA query_only = TRUE');
        return new self($pdo);
    }

    public function pickPool(string $countryCode): string
    {
        $rule = $this->ruleFor($countryCode) ?? $this->ruleFor('_default');
        if ($rule === null) {
            return self::FLOOR_POOL;
        }

        foreach ([$rule['primary'], ...$rule['fallbacks']] as $pool) {
            if ($this->poolHealthy($pool)) {
                return $pool;
            }
        }
        return self::FLOOR_POOL;
    }

    private function ruleFor(string $cc): ?array
    {
        $val = $this->rawValue('/vvv/routing/country/' . $cc);
        return $val === null ? null : json_decode($val, true, flags: JSON_THROW_ON_ERROR);
    }

    private function poolHealthy(string $pool): bool
    {
        $val = $this->rawValue('/vvv/routing/pool/' . $pool);
        if ($val === null) {
            return false; // lease expired -> key gone -> pool unhealthy
        }
        $h = json_decode($val, true, flags: JSON_THROW_ON_ERROR);
        return ($h['status'] ?? '') === 'healthy';
    }

    private function rawValue(string $key): ?string
    {
        $stmt = $this->db->prepare('SELECT val FROM routing WHERE key = ?');
        $stmt->execute([$key]);
        $val = $stmt->fetchColumn();
        return $val === false ? null : (string) $val;
    }
}

// Usage in a LiteSpeed request:
$router = RegionRouter::open();
$pool = $router->pickPool($_SERVER['HTTP_CF_IPCOUNTRY'] ?? 'XX');
header('X-VVV-Pool: ' . $pool);

Notice the PHP logic is a faithful mirror of the Go logic: same fallback walk, same "missing key means unhealthy" rule, same hardcoded floor. Keeping the two resolvers behaviorally identical is the single most important discipline here. If Go and PHP disagree about what "healthy" means, you get the split-brain routing you were trying to eliminate, just moved one layer down.

Pushing the same config to Cloudflare Workers

Cloudflare Workers sit in front of everything and cannot watch etcd — they are ephemeral and globally distributed. So I flip the direction: a small pusher reads etcd and writes the compiled country map into Workers KV whenever a revision changes. The Worker reads KV, which is eventually consistent but propagates globally within a few seconds, which is fine for the edge tier because it is only doing coarse routing, not health-sensitive failover.

export default {
  async fetch(request, env) {
    const country = request.cf?.country ?? 'XX';
    const raw = await env.ROUTING.get(`country/${country}`)
      ?? await env.ROUTING.get('country/_default');

    const rule = JSON.parse(raw);
    const url = new URL(request.url);
    // Coarse edge routing: pin EU countries to EU origins only.
    url.hostname = `${rule.primary}.origin.viralvidvault.com`;

    const resp = await fetch(url, request);
    const out = new Response(resp.body, resp);
    out.headers.set('X-VVV-Edge-Pool', rule.primary);
    return out;
  },
};

The important architectural line is that etcd remains the single writer of truth, and every other surface — SQLite mirror, Workers KV — is a read-only projection of it. Nothing writes routing intent except a human running etcdctl put (or our admin panel doing the same via the API), and nothing writes pool liveness except the pools themselves via leases. No surface ever writes back up the chain.

Leases, health, and automatic failover

The lease TTL is a tuning knob, and I got it wrong twice. Too long (60s) and a dead pool keeps receiving traffic for a full minute. Too short (5s) and a brief GC pause or network blip on a healthy pool drops its key, causing needless traffic thrash. We settled on a 15-second lease renewed every 5 seconds, so we tolerate two missed renewals before a pool is declared gone. That gives roughly 10-15 seconds worst-case drain, which for us beats both extremes.

A few things I would tell my past self:

• Renew, do not re-grant. Use KeepAlive on the lease, not a loop that grants a fresh lease each cycle. Re-granting churns revisions and, if it ever fails mid-cycle, briefly deletes your key.
• Watch compaction will bite you. etcd compacts old revisions. If your watcher disconnects for longer than the compaction window and tries to resume from a compacted revision, it errors. Handle that by re-snapshotting from current, not by retrying the old revision.
• Bound your fallback chain. A cyclic or missing fallback should terminate at a hardcoded floor pool, never loop. Both my Go and PHP resolvers end at par-1 unconditionally.

The GDPR line about what does not go in etcd

Because we are a European product, one rule is absolute: etcd holds routing policy, never viewer data. The keys describe countries and pools, not people. The only per-request input is the coarse CF-IPCOUNTRY header, which we use transiently to pick a pool and never persist against a user identity. This keeps the coordination layer entirely outside the scope of personal-data processing, which is exactly where you want your always-on, replicated, backed-up infrastructure store to sit. If it never ingests personal data, it never becomes a subject-access-request problem.

Conclusion

Moving the region map out of deploy-time PHP and into etcd turned a 4-8 minute, non-atomic redeploy into a sub-second, globally consistent PUT. The pattern that made it maintainable was strict layering: etcd is the only writer of truth, and the Go sidecar, the SQLite WAL mirror for PHP 8.4, and Cloudflare Workers KV are all read-only projections that resolve routing with identical fallback logic. Leases turned pool health into self-healing state, so incidents drain themselves without a human editing config under pressure. If you run video, ad, or content routing across several origins and an edge tier, this is a small amount of infrastructure that removes an entire class of split-brain outage — and if you keep personal data out of it, it stays comfortably outside your GDPR blast radius while doing so.