Coordinating Video Region Routers with etcd for Live Config Changes

The night a single JSON file took down three regions

At 02:00 UTC one of our region routers started sending every European viewer to a US-East origin. The cause was embarrassingly boring: we deploy configuration to the streaming-discovery fleet over FTP, and a half-written regions.json was picked up mid-upload by the PHP 8.4 router before the transfer finished. For a platform whose entire value is fast discovery, thirty seconds of cross-continent latency is the difference between a play and a bounce. That incident is why we moved routing configuration out of flat files and into etcd, and this post is a field report from running it as the source of truth for the eight region routers behind TrendVidStream, a global multi-region streaming-discovery service.

The short version: FTP is still fine for shipping templates and static assets, but it is a terrible transport for configuration that many processes read concurrently and that has to change atomically. etcd fixed the atomicity, gave us a real change-notification channel, and — almost as a side effect — became our failover signal.

What a region router actually decides

Before etcd it was tempting to think of routing config as "just a file." It is not. Each of our eight region routers makes several decisions per request, and every one of them is driven by config that changes on its own cadence:

• Origin selection — which regional origin serves the discovery API for a viewer geolocated to that region.
• FTS5 shard — which SQLite FTS5 index the search endpoint queries, since we shard the catalog by continent.
• Cron cadence — how often the multi-region cron re-scrapes and re-ranks trending titles for that region.
• Failover weights — the ordered list of fallback origins used when the primary origin is unhealthy.
• Feature flags — whether a region gets the new autoplay-preview row or the control experience.

Those five knobs move independently. The failover list changes when a box dies, which can be seconds. The cron cadence changes when we tune scraping load, which is weekly. Trying to keep all of that coherent across eight routers with a file that gets overwritten wholesale is how you end up debugging at 02:00.

Why flat files and FTP stopped scaling

Three specific failures pushed us off files:

• Non-atomic reads. FTP has no notion of an atomic swap. A reader can open the file between the truncate and the final byte. Even rename tricks only help on the same filesystem — useless when the transport is FTP from a build host.
• No change notification. A file gives you nothing better than polling. Our routers were stat-ing regions.json every few seconds, which is both wasteful and slow to react.
• No consensus. With a file per host, two hosts can disagree indefinitely and nobody notices. There is no single revision number you can point at and say "everyone should be on this or newer."

etcd answers all three: writes are atomic and linearizable, Watch streams changes with the revision that produced them, and the Raft-backed store gives every reader a monotonic revision to reason about.

The etcd data model we settled on

We keep the layout boring and prefix-scannable. Routing lives under one prefix, health under another:

• /tvs/routing/<region> — the JSON blob a router needs for that region.
• /tvs/health/<region>/<addr> — a lease-backed key that exists only while an origin is alive.

Regions are the eight we serve: us-east, us-west, eu-west, eu-central, ap-south, ap-southeast, sa-east, af-south. Everything a router needs for a region is in a single value, so a read is one key and an update is one atomic write. No partial states, no cross-key transactions on the hot path.

Publishing config with a Python cron job

Our multi-region cron already regenerates ranking weights on a schedule. It now ends by pushing routing config into etcd instead of writing a file for FTP. The important detail is the compare-before-write: we only put when the value actually changed, so unchanged regions do not churn the revision number and do not wake up every watcher for nothing.

import json
import etcd3

# Region routing config publisher for TrendVidStream.
# Runs on the multi-region cron host after weights are regenerated.

REGIONS = ["us-east", "us-west", "eu-west", "eu-central",
           "ap-south", "ap-southeast", "sa-east", "af-south"]

def publish(client, region, config):
    key = f"/tvs/routing/{region}"
    payload = json.dumps(config, separators=(",", ":"), sort_keys=True)
    # Skip the write when nothing changed: no revision bump, no watcher wakeups.
    current = client.get(key)[0]
    if current is not None and current.decode() == payload:
        return False
    client.put(key, payload)
    return True

def main():
    client = etcd3.client(host="10.0.0.10", port=2379)
    for region in REGIONS:
        cfg = {
            "origin": f"origin-{region}.tvs.internal",
            "fts5_shard": region.split("-")[0],
            "cron_minutes": 15 if region.startswith("eu") else 30,
            "failover": ["us-west"] if region == "us-east" else ["us-east"],
            "enabled": True,
        }
        changed = publish(client, region, cfg)
        print(f"{region}: {'updated' if changed else 'unchanged'}")

if __name__ == "__main__":
    main()

That is the entire write path. Notice there is no locking and no coordination between cron runs — if two runs race, the compare-and-put makes the loser a no-op, and the worst case is one extra revision. etcd's linearizable writes do the hard part.

Watching for changes in the Go router

The routers themselves are Go. The pattern that matters here is the two-step load: do a snapshot read first, then start the watch from the snapshot's revision plus one. If you start the watch before reading, or from revision 0 on a compacted store, you either miss events or replay history you do not want. Anchoring the watch to the snapshot revision guarantees you see every change exactly once, in order, with no gap.

package main

import (
    "context"
    "encoding/json"
    "log"
    "sync"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
)

type RouteConfig struct {
    Origin    string   `json:"origin"`
    FTS5Shard string   `json:"fts5_shard"`
    CronMin   int      `json:"cron_minutes"`
    Failover  []string `json:"failover"`
    Enabled   bool     `json:"enabled"`
}

type Router struct {
    mu     sync.RWMutex
    routes map[string]RouteConfig
}

func (r *Router) set(region string, cfg RouteConfig) {
    r.mu.Lock()
    defer r.mu.Unlock()
    r.routes[region] = cfg
}

func (r *Router) Origin(region string) string {
    r.mu.RLock()
    defer r.mu.RUnlock()
    if c, ok := r.routes[region]; ok && c.Enabled {
        return c.Origin
    }
    return "origin-us-east.tvs.internal" // safe default
}

func main() {
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"10.0.0.10:2379", "10.0.0.11:2379", "10.0.0.12:2379"},
        DialTimeout: 5 * time.Second,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer cli.Close()

    router := &Router{routes: map[string]RouteConfig{}}

    // 1. Load the current snapshot with a linearizable range read.
    resp, err := cli.Get(context.Background(), "/tvs/routing/", clientv3.WithPrefix())
    if err != nil {
        log.Fatal(err)
    }
    for _, kv := range resp.Kvs {
        var cfg RouteConfig
        if err := json.Unmarshal(kv.Value, &cfg); err == nil {
            router.set(regionFromKey(string(kv.Key)), cfg)
        }
    }

    // 2. Watch from snapshot revision + 1 so no event is missed or replayed.
    wch := cli.Watch(context.Background(), "/tvs/routing/",
        clientv3.WithPrefix(), clientv3.WithRev(resp.Header.Revision+1))

    for wr := range wch {
        for _, ev := range wr.Events {
            region := regionFromKey(string(ev.Kv.Key))
            var cfg RouteConfig
            if err := json.Unmarshal(ev.Kv.Value, &cfg); err != nil {
                log.Printf("bad config for %s: %v", region, err)
                continue // keep the last good value, never apply garbage
            }
            router.set(region, cfg)
            log.Printf("route updated: %s -> %s (rev %d)",
                region, cfg.Origin, ev.Kv.ModRevision)
        }
    }
}

func regionFromKey(key string) string {
    for i := len(key) - 1; i >= 0; i-- {
        if key[i] == '/' {
            return key[i+1:]
        }
    }
    return key
}

The router holds the whole routing table in memory behind an RWMutex, so request-path reads never touch the network. etcd is consulted once at boot and thereafter only pushes deltas. When a bad blob arrives — say a publisher bug — the unmarshal fails, we log it, and the router keeps serving the last known-good value. That fail-safe behavior is exactly what the FTP setup lacked.

Reading etcd from PHP without a gRPC extension

Not every edge is Go. A large part of our discovery site is still PHP 8.4 running behind LiteSpeed, and I did not want to compile a gRPC extension onto shared-ish hosting. etcd's v3 API is also exposed as a gRPC-gateway JSON endpoint, which means plain curl. Keys and values come back base64-encoded, and a prefix scan is expressed as a range with range_end set to the key with its last byte incremented.

<?php
declare(strict_types=1);

// PHP 8.4 discovery edge. Reads etcd over its gRPC-gateway JSON API,
// so no gRPC extension is required on the LiteSpeed hosts.

final class EtcdRoutes
{
    private const CACHE = '/dev/shm/tvs_routes.json';
    private const TTL   = 5; // seconds; a floor between range reads

    public function __construct(private string $endpoint = 'http://10.0.0.10:2379') {}

    /** @return array<string,mixed> */
    public function region(string $region): array
    {
        $all = $this->snapshot();
        return $all[$region] ?? ['origin' => 'origin-us-east.tvs.internal', 'enabled' => true];
    }

    /** @return array<string,array<string,mixed>> */
    private function snapshot(): array
    {
        if (is_file(self::CACHE) && (time() - filemtime(self::CACHE)) < self::TTL) {
            return json_decode((string) file_get_contents(self::CACHE), true) ?: [];
        }

        // Range request: key .. range_end covers the whole prefix.
        $key = base64_encode('/tvs/routing/');
        $end = base64_encode('/tvs/routing0'); // '0' is the byte after '/'
        $body = json_encode(['key' => $key, 'range_end' => $end]);

        $ch = curl_init($this->endpoint . '/v3/kv/range');
        curl_setopt_array($ch, [
            CURLOPT_POST           => true,
            CURLOPT_POSTFIELDS     => $body,
            CURLOPT_HTTPHEADER     => ['Content-Type: application/json'],
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 2,
        ]);
        $raw = curl_exec($ch);
        if ($raw === false) {
            // etcd unreachable: serve the last good snapshot, never fail closed.
            return is_file(self::CACHE)
                ? (json_decode((string) file_get_contents(self::CACHE), true) ?: [])
                : [];
        }

        $routes = [];
        foreach (json_decode($raw, true)['kvs'] ?? [] as $kv) {
            $region = basename(base64_decode($kv['key']));
            $routes[$region] = json_decode(base64_decode($kv['value']), true);
        }
        file_put_contents(self::CACHE, json_encode($routes), LOCK_EX);
        return $routes;
    }
}

$routes = new EtcdRoutes();
$origin = $routes->region('eu-west')['origin'];
header('X-TVS-Origin: ' . $origin);

The PHP side caches the snapshot in /dev/shm for a few seconds so a burst of requests does not hammer etcd, and it degrades to the last good snapshot if etcd is unreachable. It is not watch-based — PHP-FPM workers are short-lived, so a tiny TTL plus shared-memory cache is the pragmatic choice. The Go routers get instant pushes; the PHP edges lag by up to five seconds, which is completely acceptable for discovery pages.

Health and failover with leases

The part that surprised me most was how naturally failover fell out of the same store. Each origin registers itself under a lease and keeps it alive. If a box crashes, the keepalive stops, the lease expires, and the key vanishes on its own. The publisher watches /tvs/health/ and rewrites the failover weights when an origin disappears — no separate health-check service, no cron reconciliation.

// Each origin registers under a lease; if the box dies the key expires
// and the publisher drops it from the failover weights.
func registerOrigin(cli *clientv3.Client, region, addr string) error {
    ctx := context.Background()
    lease, err := cli.Grant(ctx, 10) // 10s TTL
    if err != nil {
        return err
    }
    key := "/tvs/health/" + region + "/" + addr
    if _, err := cli.Put(ctx, key, "up", clientv3.WithLease(lease.ID)); err != nil {
        return err
    }
    ch, err := cli.KeepAlive(ctx, lease.ID)
    if err != nil {
        return err
    }
    go func() {
        for range ch {
            // drain keepalive acks; loop ends when the lease is lost
        }
        log.Printf("lease for %s lost — origin will be evicted", key)
    }()
    return nil
}

A 10-second TTL means we detect a dead origin within about ten seconds and re-point traffic on the next publisher pass. Tighten the TTL and you detect faster but risk flapping on a GC pause; loosen it and you carry a dead origin longer. Ten seconds has been a good balance for us.

Guarding against the split-brain we were afraid of

The original fear was: what if the config store itself disagrees with itself and we route half of Europe wrong? etcd's Raft consensus makes that specific failure impossible. A write is acknowledged only after a quorum of the cluster accepts it, and linearizable reads see the latest committed value. If the cluster loses quorum — say two of three nodes are gone — etcd stops accepting writes rather than serving stale or divergent data. For config, a store that refuses to change is far safer than one that changes inconsistently: our routers just keep running their last good in-memory table.

The revision number is the other quiet win. Every value carries the revision that produced it. When we debug, we can ask "what revision is each router on?" and get a straight answer. With files spread over FTP there was never a single number to compare.

What FTP still does, and what it does not

We did not rip out FTP — that would be throwing away a tool that works for what it is good at. Templates, CSS, and the PHP application code still ship over FTP automation, because those are versioned artifacts that change on a deploy cadence and are read once per process start. What moved to etcd is only the live, concurrent, atomically-changing configuration: routing tables, cron cadence, feature flags, and health. The rule of thumb we ended up with: if two processes read it at the same time and it can change between reads, it belongs in etcd; if it is a deploy artifact, FTP is fine.

Lessons after six months

• Anchor watches to a snapshot revision. The single most common etcd bug is a gap between the initial read and the watch. Read first, watch from revision + 1.
• Keep values self-contained. One key per region, whole config in the value. Avoid cross-key transactions on the request path.
• Compare before writing. Skipping no-op writes keeps the revision history and the watch stream quiet, which makes real changes easy to see.
• Always have a safe default in code. Every reader falls back to us-east and a cached snapshot. etcd being down should degrade latency, never availability.
• Leases are underrated. Health and failover for free, with no extra service to run.

Conclusion

Moving region-router configuration into etcd turned a fragile, file-shaped process into a boringly reliable one. Writes are atomic, changes propagate to the Go routers in milliseconds and to the PHP edges within a few seconds, and origin failover falls out of the same lease mechanism we use for health. FTP still ships our code and templates, but nothing that changes under live traffic touches a flat file anymore. If you run a multi-region service and you are still overwriting a config file in place, the 02:00 incident is waiting for you too — etcd is a small dependency to add for never debugging that one again.