Coordinating Video Region Routers With etcd and Live Watch Reloads

At DailyWatch we route every video request through a small fleet of edge PHP nodes that decide, per visitor, which regional catalog and which upstream origin to serve. A viewer in Frankfurt should hit our EU catalog shard; a viewer in São Paulo should hit the LATAM shard; and when an origin gets saturated we need to drain traffic away from it within seconds, not on the next deploy. For a long time that mapping lived in a regions.php array baked into every node and shipped through the same FTP-based release that pushes the rest of the site. That worked until it didn't: draining a hot origin meant editing a file, redeploying to every node, and clearing the DailyWatch LiteSpeed cache before the change actually took hold. Minutes of lag during an incident is unacceptable when a single origin is throwing 502s.

The fix was to pull region routing config out of the deploy artifact entirely and put it in etcd, with each edge node watching for changes and reloading in-process. This post walks through the concrete design: the key layout, an atomic writer, a Go control agent that watches and pushes to a local socket, and the PHP router that reads a warm local snapshot on every request. Everything here is runnable and reflects the tradeoffs we actually hit.

Why etcd and not just a database column

We already run SQLite with FTS5 for the catalog and search on every node, so the obvious question was: why not a routing table? The answer is the notification model. SQLite gives you great local reads but no fan-out. To learn that a value changed, a node has to poll, and polling at the frequency we'd need (sub-second during an incident) hammers disk and still leaves a window. etcd is built around exactly the primitive we want: a strongly consistent key-value store with a watch API that streams changes to every subscriber the moment a write is committed via Raft. You write once to the cluster and every edge node knows within a round-trip.

The properties that matter for routing config specifically:

• Linearizable reads and writes — when we mark an origin as drained, every node sees a consistent view, no split-brain where half the fleet still sends traffic there.
• Watches with revisions — every change carries a monotonic revision number, so a node that reconnects after a blip can resume from where it left off instead of missing edits.
• Leases — an origin can register itself with a TTL. If the origin process dies and stops renewing, its key expires and it disappears from routing automatically.
• Small values, high read locality — routing config is kilobytes, read on every request. etcd is the source of truth, but each node keeps a warm local copy so request-path reads never touch the network.

That last point is the whole architecture in one sentence: etcd is authoritative, but the request path only ever reads a local snapshot. A viewer's request must never block on a Raft round-trip.

Key layout

Flat keyspaces get unmanageable fast, so we use a prefix hierarchy that mirrors how the router thinks. Everything lives under /dw/routing/:

• /dw/routing/region/<geo> — maps a Cloudflare CF-IPCountry code (or a continent bucket) to a catalog shard id. Value is a small JSON object.
• /dw/routing/origin/<id> — one key per upstream origin, written with a lease so dead origins vanish.
• /dw/routing/policy/global — feature-flag-ish knobs: default shard, failover order, whether to honor client hints.

A region value looks like this:

{
  "shard": "eu-1",
  "origins": ["fra-a", "fra-b", "ams-a"],
  "weight": {"fra-a": 50, "fra-b": 30, "ams-a": 20},
  "updated": "2026-07-30T09:12:00Z"
}

Storing JSON blobs rather than one etcd key per field is deliberate. A region's routing decision needs to change atomically — you never want a node to observe the new origin list but the old weights. One key, one value, one revision.

Writing config atomically from the control plane

The control plane is a thin CLI our on-call engineers run (and that our automated drain logic calls). The critical requirement: a multi-key change — say, draining fra-a while bumping fra-b's weight across three regions — must land as a single transaction so no node ever sees a half-applied state. etcd gives us this with a Txn. Here's the writer in Python using etcd3:

import json
import etcd3

client = etcd3.client(host="10.0.0.10", port=2379)

def drain_origin(origin_id: str, regions: dict[str, dict]) -> None:
    """Remove origin_id from the given regions and rebalance, atomically.

    `regions` maps region key -> the region's current parsed JSON value.
    """
    success_ops = []
    compares = []

    for key, value in regions.items():
        if origin_id not in value.get("origins", []):
            continue

        # Guard: only apply if the value hasn't changed under us.
        _, meta = client.get(key)
        compares.append(
            client.transactions.mod(key) == meta.mod_revision
        )

        value["origins"] = [o for o in value["origins"] if o != origin_id]
        weights = {o: w for o, w in value.get("weight", {}).items()
                   if o != origin_id}
        # Renormalize remaining weights to sum to 100.
        total = sum(weights.values()) or 1
        value["weight"] = {o: round(w * 100 / total) for o, w in weights.items()}
        value["updated"] = "2026-07-30T09:12:00Z"  # inject real UTC in prod

        success_ops.append(
            client.transactions.put(key, json.dumps(value, separators=(",", ":")))
        )

    if not success_ops:
        return

    ok, _ = client.transaction(compare=compares, success=success_ops, failure=[])
    if not ok:
        raise RuntimeError("config changed concurrently; retry the drain")

The compare clauses turn this into an optimistic-concurrency write: every region key must still be at the mod_revision we read, or the whole transaction fails and we retry. This is what stops two concurrent drains from clobbering each other. The renormalization keeps weights summing to 100 so the router's selection logic stays simple — it can treat weights as a probability distribution without re-summing on every request.

One detail worth calling out: we write compact JSON (separators=(",", ":")). Routing values are read constantly and the difference between pretty-printed and compact adds up in etcd's storage and in the watch payload size across a fleet.

The watch-and-push agent on each edge node

Here's the part that earns its keep. PHP under LiteSpeed is process-per-request; there's no long-lived PHP process to hold an etcd watch open. So each edge node runs a tiny Go sidecar — the control agent — that holds exactly one watch on the /dw/routing/ prefix, maintains an in-memory snapshot, and writes that snapshot to a local file plus notifies via a Unix socket. PHP then reads the file. The Go agent is the only thing that ever talks to etcd from the edge.

package main

import (
    "context"
    "encoding/json"
    "log"
    "os"
    "path/filepath"
    "sync"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
)

const (
    prefix   = "/dw/routing/"
    snapshot = "/dev/shm/dw-routing.json"
)

type Agent struct {
    mu    sync.RWMutex
    state map[string]json.RawMessage
}

func (a *Agent) writeSnapshot() error {
    a.mu.RLock()
    data, err := json.Marshal(a.state)
    a.mu.RUnlock()
    if err != nil {
        return err
    }
    // Write-then-rename so PHP never reads a half-written file.
    tmp := snapshot + ".tmp"
    if err := os.WriteFile(tmp, data, 0o644); err != nil {
        return err
    }
    return os.Rename(tmp, snapshot)
}

func (a *Agent) Run(ctx context.Context, cli *clientv3.Client) error {
    // 1. Seed the full current state, capturing the revision.
    resp, err := cli.Get(ctx, prefix, clientv3.WithPrefix())
    if err != nil {
        return err
    }
    a.state = make(map[string]json.RawMessage, len(resp.Kvs))
    for _, kv := range resp.Kvs {
        a.state[string(kv.Key)] = json.RawMessage(kv.Value)
    }
    if err := a.writeSnapshot(); err != nil {
        return err
    }

    // 2. Watch from the revision right after the snapshot — no gap, no dupes.
    wch := cli.Watch(ctx, prefix,
        clientv3.WithPrefix(),
        clientv3.WithRev(resp.Header.Revision+1))

    for wresp := range wch {
        if err := wresp.Err(); err != nil {
            return err // caller reconnects with backoff
        }
        a.mu.Lock()
        for _, ev := range wresp.Events {
            key := string(ev.Kv.Key)
            if ev.Type == clientv3.EventTypeDelete {
                delete(a.state, key)
            } else {
                a.state[key] = json.RawMessage(ev.Kv.Value)
            }
        }
        a.mu.Unlock()
        if err := a.writeSnapshot(); err != nil {
            log.Printf("snapshot write failed: %v", err)
        }
    }
    return ctx.Err()
}

func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"10.0.0.10:2379", "10.0.0.11:2379", "10.0.0.12:2379"},
        DialTimeout: 5 * time.Second,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()

    agent := &Agent{}
    ctx := context.Background()
    for {
        if err := agent.Run(ctx, cli); err != nil {
            log.Printf("watch loop ended: %v; reconnecting", err)
            time.Sleep(2 * time.Second) // backoff before re-seed
        }
    }
}

Three things make this correct rather than merely working:

• Snapshot revision → watch revision continuity. We seed from Get, remember resp.Header.Revision, and start the watch at revision+1. That closes the gap where a write could land between the seed and the watch. No missed edits, no double-applied edits.
• Write-then-rename. os.Rename is atomic on the same filesystem, so PHP either reads the old complete file or the new complete file — never a torn read. We put the snapshot on /dev/shm (tmpfs) so reads are pure memory, no disk I/O on the request path.
• Reconnect re-seeds. When the watch channel errors (network blip, etcd leader election), the loop restarts and re-seeds the full state. etcd's compaction means you can't always resume an old revision after a long disconnect, so a clean re-seed is the safe default.

Reading config in the PHP request path

Now the router. On every request, PHP reads the tmpfs snapshot, picks a shard from the visitor's country (which Cloudflare hands us in CF-IPCountry), and selects a weighted origin. Because the file is on tmpfs and PHP 8.4's json_decode is fast, this adds microseconds, not milliseconds. We also stat the file mtime and cache the decoded structure in APCu so repeated requests in the same worker don't re-decode.

<?php
declare(strict_types=1);

final class RegionRouter
{
    private const SNAPSHOT = '/dev/shm/dw-routing.json';

    /** @var array<string, mixed> */
    private array $config;

    public function __construct()
    {
        $mtime = @filemtime(self::SNAPSHOT) ?: 0;
        $cacheKey = 'dw:routing:' . $mtime;

        $cached = apcu_fetch($cacheKey, $hit);
        if ($hit) {
            $this->config = $cached;
            return;
        }

        $raw = @file_get_contents(self::SNAPSHOT);
        $this->config = $raw !== false
            ? (json_decode($raw, true, 32, JSON_THROW_ON_ERROR) ?: [])
            : [];

        // mtime in the key means stale entries expire naturally on change.
        apcu_store($cacheKey, $this->config, 300);
    }

    public function resolve(string $country): array
    {
        $regionKey = "/dw/routing/region/{$country}";
        $policy = $this->decode('/dw/routing/policy/global');
        $region = $this->decode($regionKey)
            ?? $this->decode('/dw/routing/region/' . ($policy['default'] ?? 'US'));

        if ($region === null) {
            // Fail open to a hardcoded safe default rather than 500.
            return ['shard' => 'us-1', 'origin' => 'iad-a'];
        }

        return [
            'shard'  => $region['shard'],
            'origin' => $this->pickWeighted($region['origins'], $region['weight'], $country),
        ];
    }

    /** @return array<string, mixed>|null */
    private function decode(string $key): ?array
    {
        return isset($this->config[$key])
            ? json_decode($this->config[$key], true, 16, JSON_THROW_ON_ERROR)
            : null;
    }

    /**
     * Deterministic weighted pick: same visitor sticks to the same origin
     * as long as weights are unchanged (sticky routing without a session).
     */
    private function pickWeighted(array $origins, array $weights, string $seed): string
    {
        $total = array_sum($weights) ?: 1;
        $point = (crc32($seed . implode($origins)) % $total);
        foreach ($origins as $origin) {
            $point -= ($weights[$origin] ?? 0);
            if ($point < 0) {
                return $origin;
            }
        }
        return $origins[0] ?? 'iad-a';
    }
}

// Usage in the front controller:
$router  = new RegionRouter();
$country = $_SERVER['HTTP_CF_IPCOUNTRY'] ?? 'US';
$route   = $router->resolve($country);
header('X-DW-Shard: ' . $route['shard']);
// ... proxy or serve from $route['origin'] ...

Note the decode() layer: the snapshot stores each etcd value as its raw JSON string, exactly as etcd holds it, so the agent never has to understand the schema. PHP decodes lazily, only the keys it touches for this request. The weighted pick is seeded by country plus the origin list, which gives us sticky routing — the same visitor lands on the same origin across requests as long as the config is stable, which keeps LiteSpeed and Cloudflare edge caches warm per-origin. When we drain an origin, the origin list changes, the seed changes, and traffic reshuffles cleanly.

The filemtime-keyed APCu entry is a small trick that matters: when the agent renames a new snapshot into place, the mtime changes, the cache key changes, and every PHP worker picks up the new config on its next request with zero explicit invalidation. No cache-clear step, no coordination.

Failure modes and how we handle them

A distributed config system is only as good as its behavior when things break. The ones we actually planned for:

• etcd unreachable from a node. The Go agent keeps serving the last snapshot it wrote. PHP keeps reading it. Routing is stale but functional. Config that can't update is far better than requests that 500. This is the single most important property: the request path has no runtime dependency on etcd being up.
• Snapshot missing or corrupt. resolve() fails open to a hardcoded us-1/iad-a default. A visitor always gets a video, even if it's not the geographically optimal origin.
• etcd compaction after a long disconnect. The agent can't resume an old revision, so on reconnect it re-seeds the full prefix and continues. Idempotent by construction.
• Origin crashes. Because origins register their own /dw/routing/origin/<id> key under a lease, a crashed origin's key expires within the TTL and the control plane's rebalance logic (or a simple watcher) removes it from region lists. We keep the lease TTL at 10 seconds — long enough to survive a GC pause, short enough to drain a dead box fast.
• Thundering herd on re-seed. If the whole etcd cluster restarts, every agent re-seeds at once. The Get with prefix is cheap (kilobytes), but we still jitter the reconnect backoff so a 500-node fleet doesn't stampede a freshly-elected leader.

One thing we explicitly did not do: watch etcd directly from PHP or open a connection per request. Under LiteSpeed's process-per-request model that would mean a new TCP connection and watch setup on every hit — catastrophic. The single-agent-per-node pattern is what makes etcd viable behind a PHP front end at all.

What changed operationally

Before, draining a hot origin was a code edit, an FTP deploy to every node, and a cache flush — call it three to five minutes of elevated error rate while the change propagated. Now it's one transactional write to etcd, and the entire fleet reflects it in well under a second because every node is already holding a watch. The control-plane CLI is scriptable, so our automated health checks can drain a misbehaving origin without a human in the loop, and roll it back the same way once it recovers.

The design comes down to a clean split of responsibilities: etcd is the strongly-consistent source of truth, a single Go agent per node turns pushes into a local tmpfs snapshot, and PHP reads that warm snapshot on every request with a hardcoded fail-open default. No request ever blocks on the network for config, no deploy is needed to change routing, and a total etcd outage degrades to stale-but-working rather than down. If you're running edge routing in front of a PHP/LiteSpeed stack and you're still baking config into your deploy artifact, moving it into etcd with a watch sidecar is one of the higher-leverage changes you can make.