One of our Singapore edge nodes started returning HTTP 200 for the manifest endpoint while quietly serving stale HLS segments for eleven minutes. Cloudflare's health check saw a 200 and kept routing traffic. Our uptime monitor in Frankfurt saw a 200 and stayed green. Meanwhile viewers in Jakarta and Manila were staring at a spinning buffer because the origin shield in sin had wedged its segment cache. A single-region probe told us nothing was wrong. That incident is why I built a multi-region health-check aggregator that treats "is the CDN up" as a distributed question with a distributed answer.
I run the backend for TopVideoHub, an Asia-Pacific video aggregator serving trending clips across CJK and Southeast Asian markets. Our audience is spread across a dozen countries with wildly different last-mile conditions, and our CDN footprint has to match. A health check that only probes one PoP is worse than useless — it's actively misleading, because it gives you the confidence to page nobody while a quarter of your users can't play video. This post walks through the actual architecture: how the probes run, how results get aggregated with a quorum model instead of a naive average, how I store and query the time series in SQLite, and the Go worker that does the concurrent regional probing.
Why a 200 Is Not a Health Signal for Video
Text sites can mostly get away with checking that / returns 200. Video delivery cannot, for three reasons that bit us repeatedly:
-
The manifest lies about the segments. An HLS
.m3u8playlist is just text. It can return 200 and validate perfectly while the.tsor.m4ssegments it references 404 or time out at the edge. You have to fetch a segment to know the pipe is real. - Freshness matters more than availability. For live and near-live content, a segment that's 40 seconds stale is a failure even though every request succeeds. A pure up/down check has no vocabulary for "technically serving, but serving the past."
- Geography is the whole problem. A probe from the same datacenter as the origin will almost never reproduce what a mobile user in Ho Chi Minh City experiences going through three transit networks and a congested peering point.
So my health signal is a tuple, not a boolean: (reachable, manifest_valid, segment_fetchable, segment_age_seconds, ttfb_ms). Each probe returns that tuple, and the aggregator's job is to turn many regional tuples into one honest verdict.
The Probe: What One Check Actually Does
Each probe targets a specific (region, edge_hostname) pair. It does a small chain of dependent requests, because the failure modes are ordered — there's no point checking segment freshness if the manifest didn't parse. Here's the core probe logic in Python, which is what I prototyped with before moving the hot path to Go:
import re
import time
import httpx
SEGMENT_RE = re.compile(r'^(?!#)(.+\.(?:ts|m4s))\s*$', re.MULTILINE)
def probe_edge(base_url: str, manifest_path: str, timeout: float = 6.0) -> dict:
"""Probe one CDN edge. Returns a health tuple as a dict."""
result = {
"reachable": False,
"manifest_valid": False,
"segment_fetchable": False,
"segment_age_s": None,
"ttfb_ms": None,
"error": None,
}
manifest_url = base_url.rstrip("/") + manifest_path
try:
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
t0 = time.perf_counter()
m = client.get(manifest_url, headers={"Cache-Control": "no-cache"})
result["ttfb_ms"] = round((time.perf_counter() - t0) * 1000, 1)
result["reachable"] = True
if m.status_code != 200 or "#EXTM3U" not in m.text[:64]:
result["error"] = f"manifest status {m.status_code}"
return result
result["manifest_valid"] = True
segments = SEGMENT_RE.findall(m.text)
if not segments:
result["error"] = "no segments in manifest"
return result
# Fetch the newest segment (last in the playlist) and check freshness.
seg_url = _resolve(manifest_url, segments[-1])
s = client.get(seg_url, headers={"Range": "bytes=0-1023"})
if s.status_code not in (200, 206):
result["error"] = f"segment status {s.status_code}"
return result
result["segment_fetchable"] = True
last_mod = s.headers.get("Last-Modified") or s.headers.get("Date")
if last_mod:
age = time.time() - _http_date_to_epoch(last_mod)
result["segment_age_s"] = round(max(age, 0), 1)
except httpx.TimeoutException:
result["error"] = "timeout"
except httpx.HTTPError as e:
result["error"] = f"http error: {e.__class__.__name__}"
return result
Two details earn their keep here. First, the Range: bytes=0-1023 header — I don't need the whole segment to prove the edge can serve it, just the first kilobyte. On a busy schedule fetching full segments from every region would waste bandwidth and skew latency numbers. Second, the freshness check leans on Last-Modified, falling back to Date. Not every CDN sets Last-Modified on segments, and when it's missing I record segment_age_s as null and let the aggregator treat freshness as "unknown" rather than "failed" — a distinction that matters, as we'll see.
Aggregation: Quorum, Not Average
The naive move is to average the regions: five green, one red, 83% healthy, ship it. That's exactly the logic that let our Singapore incident stay silent, because from a global average one bad PoP disappears into the noise. The right question is not "what fraction of probes passed" but "is any region my users care about currently broken, and do enough independent probes agree."
So I use a weighted quorum with two ideas baked in:
- Weights by audience. A failing edge in Tokyo or Seoul matters far more to us than one in São Paulo, because that's where our traffic lives. Each region carries an audience weight, and the verdict is computed against weighted thresholds.
- Quorum against flapping. A single probe failing once is almost always a transient network hiccup between the prober and the edge, not a real outage. I require agreement — at least N of the last M probes for a region must fail before that region is declared down. This kills the pager-at-3am false positives.
Here's the aggregation in PHP 8.4, which is where the rest of our stack lives (LiteSpeed in front, SQLite underneath):
<?php
declare(strict_types=1);
enum EdgeVerdict: string
{
case Healthy = 'healthy';
case Degraded = 'degraded';
case Down = 'down';
}
final readonly class RegionHealth
{
public function __construct(
public string $region,
public float $audienceWeight, // 0.0 - 1.0, sums to ~1 across regions
public int $recentFailures, // failing probes in the window
public int $windowSize, // total probes in the window
public ?float $worstSegmentAge, // seconds, null if all unknown
) {}
public function isDown(int $quorum): bool
{
// Require a quorum of failures, not a single blip.
return $this->recentFailures >= $quorum;
}
public function isStale(float $maxAgeSeconds): bool
{
return $this->worstSegmentAge !== null
&& $this->worstSegmentAge > $maxAgeSeconds;
}
}
final class HealthAggregator
{
public function __construct(
private int $failureQuorum = 3, // 3 of last N probes
private float $maxSegmentAge = 30.0, // seconds before "stale"
private float $downWeightBudget = 0.15, // >15% weighted-down = global down
) {}
/** @param list<RegionHealth> $regions */
public function aggregate(array $regions): array
{
$downWeight = 0.0;
$staleWeight = 0.0;
$perRegion = [];
foreach ($regions as $r) {
$verdict = EdgeVerdict::Healthy;
if ($r->isDown($this->failureQuorum)) {
$verdict = EdgeVerdict::Down;
$downWeight += $r->audienceWeight;
} elseif ($r->isStale($this->maxSegmentAge)) {
$verdict = EdgeVerdict::Degraded;
$staleWeight += $r->audienceWeight;
}
$perRegion[$r->region] = $verdict->value;
}
$global = match (true) {
$downWeight >= $this->downWeightBudget => EdgeVerdict::Down,
$downWeight > 0 || $staleWeight > 0.10 => EdgeVerdict::Degraded,
default => EdgeVerdict::Healthy,
};
return [
'global' => $global->value,
'down_weight' => round($downWeight, 3),
'stale_weight' => round($staleWeight, 3),
'regions' => $perRegion,
];
}
}
The downWeightBudget is the honest lever. Set it to 0.15 and you're saying: if regions carrying more than 15% of our audience are down, that's a global incident, page someone. A single small-audience PoP failing shows up as degraded — visible on the dashboard, logged, but not waking anyone. This mapping from technical state to human urgency is the entire point of the aggregator, and getting the weights right took a few weeks of tuning against real traffic distribution from our analytics.
Storing the Time Series in SQLite
We don't run a dedicated time-series database. For a health-check history at our volume — a few dozen regions probed every 30 seconds — SQLite is more than enough, and it means one fewer moving part to operate. The schema is deliberately boring:
CREATE TABLE IF NOT EXISTS probe_results (
id INTEGER PRIMARY KEY,
region TEXT NOT NULL,
edge_host TEXT NOT NULL,
checked_at INTEGER NOT NULL, -- unix epoch seconds
reachable INTEGER NOT NULL,
manifest_ok INTEGER NOT NULL,
segment_ok INTEGER NOT NULL,
segment_age_s REAL, -- nullable: unknown freshness
ttfb_ms REAL,
error TEXT
);
CREATE INDEX IF NOT EXISTS idx_probe_region_time
ON probe_results (region, checked_at DESC);
The index on (region, checked_at DESC) is what makes the windowed quorum query cheap. To build the RegionHealth objects the aggregator needs, I pull the last N probes per region and let SQLite do the counting:
-- Last 5 probes per region, collapsed into a health summary.
WITH windowed AS (
SELECT
region,
checked_at,
(reachable = 0 OR manifest_ok = 0 OR segment_ok = 0) AS failed,
segment_age_s,
ROW_NUMBER() OVER (
PARTITION BY region ORDER BY checked_at DESC
) AS rn
FROM probe_results
WHERE checked_at > unixepoch() - 600
)
SELECT
region,
SUM(failed) AS recent_failures,
COUNT(*) AS window_size,
MAX(segment_age_s) AS worst_segment_age
FROM windowed
WHERE rn <= 5
GROUP BY region;
A couple of operational notes that saved me pain. I keep the probe table in WAL mode so the Go writer and the PHP reader don't block each other — writers append, readers get a consistent snapshot. And I run a nightly DELETE FROM probe_results WHERE checked_at < unixepoch() - 1209600 to keep two weeks of history, followed by an occasional VACUUM. Fourteen days is plenty for incident forensics; anything older gets rolled up into a daily summary table before deletion. On the same box we also run our search layer on SQLite's FTS5 with a CJK-aware tokenizer, so keeping the health data in the same engine means one backup story, one connection pool, one thing to reason about.
The Concurrent Prober in Go
The Python probe is fine for a handful of edges, but once you're probing 30 regions on a 30-second cadence you want real concurrency without spawning a process per check. This is where Go shines, and it's what actually runs in production. The worker fans out to every edge concurrently, bounds the parallelism with a semaphore so we don't nuke our own network, and writes results straight into the SQLite table:
package main
import (
"context"
"database/sql"
"net/http"
"strings"
"sync"
"time"
_ "github.com/mattn/go-sqlite3"
)
type Edge struct {
Region string
Host string
ManifestURL string
}
type Result struct {
Edge Edge
Reachable bool
ManifestOK bool
SegmentOK bool
TTFBms float64
Err string
}
func probe(ctx context.Context, client *http.Client, e Edge) Result {
res := Result{Edge: e}
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, e.ManifestURL, nil)
req.Header.Set("Cache-Control", "no-cache")
start := time.Now()
resp, err := client.Do(req)
if err != nil {
res.Err = err.Error()
return res
}
defer resp.Body.Close()
res.TTFBms = float64(time.Since(start).Microseconds()) / 1000.0
res.Reachable = true
if resp.StatusCode == http.StatusOK {
res.ManifestOK = true
// Real code parses the manifest and range-fetches a segment here;
// omitted for length. Set res.SegmentOK from that fetch.
res.SegmentOK = true
}
return res
}
func runRound(ctx context.Context, db *sql.DB, edges []Edge) error {
client := &http.Client{Timeout: 6 * time.Second}
sem := make(chan struct{}, 12) // cap concurrent probes
results := make(chan Result, len(edges))
var wg sync.WaitGroup
for _, e := range edges {
wg.Add(1)
go func(e Edge) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
results <- probe(ctx, client, e)
}(e)
}
go func() { wg.Wait(); close(results) }()
tx, err := db.Begin()
if err != nil {
return err
}
stmt, err := tx.Prepare(`INSERT INTO probe_results
(region, edge_host, checked_at, reachable, manifest_ok, segment_ok, ttfb_ms, error)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
tx.Rollback()
return err
}
defer stmt.Close()
now := time.Now().Unix()
for r := range results {
_, err := stmt.Exec(r.Edge.Region, r.Edge.Host, now,
b2i(r.Reachable), b2i(r.ManifestOK), b2i(r.SegmentOK),
r.TTFBms, strings.TrimSpace(r.Err))
if err != nil {
tx.Rollback()
return err
}
}
return tx.Commit()
}
func b2i(b bool) int {
if b {
return 1
}
return 0
}
The semaphore capped at 12 is deliberate. Early on I let all 30 probes fire at once and immediately started getting rate-limited by our own CDN's edge WAF, which read the burst as a scan. Bounding concurrency to a dozen keeps the round finishing in about two seconds while looking like normal traffic. Wrapping the whole round's inserts in a single transaction turns 30 individual writes into one commit, which on SQLite is the difference between a snappy writer and a lock-contention nightmare — SQLite serializes writes, so batching them is not optional at this cadence.
Where the Probes Actually Run
Code is the easy part; probe placement is what makes the data trustworthy. A prober sitting in the same cloud region as your origin will report rosy numbers that no real user experiences. My rule is that probes must run from networks that resemble the audience. Concretely:
- Cheap VMs in target regions. A small instance in Tokyo, Singapore, and Mumbai from a couple of different providers covers most of our APAC footprint. Different providers matter because they take different transit paths.
- Cloudflare Workers as a secondary vantage. Running a lightweight version of the probe from Workers gives me hundreds of effective locations at low cost, and it checks the path through Cloudflare's own network — which is what most of our users traverse anyway.
- Never trust a single prober per region. Two independent probers per region is the minimum that lets the quorum logic distinguish "the edge is down" from "one prober's network is having a bad minute."
The aggregator doesn't care where a probe came from beyond its region label, but having two probers feed each region is what makes the failure-quorum meaningful. If both probers in sin report failures and a prober in nrt doesn't, that's a real Singapore edge problem — exactly the signal that would have caught our original eleven-minute stale-cache incident in seconds instead of minutes.
What Changed After Shipping This
The aggregator has been running for a few months now, and the concrete wins are boring in the best way:
- Mean time to detection for regional issues dropped from ~10 minutes to under 90 seconds. The quorum window is three failing probes at a 30-second cadence, so a genuine outage trips the verdict fast while transient blips stay quiet.
-
False pages went to near zero. The audience-weighted budget means a low-traffic edge flapping shows up as
degradedon the dashboard and never pages anyone at night. -
Stale-cache incidents became visible for the first time. Before, a 200-serving-old-segments failure was invisible. Now
segment_age_scatches it, and we've traced two separate origin-shield bugs we'd otherwise never have seen.
If you're delivering video across regions, the single most valuable change you can make is to stop treating health as a boolean and start treating it as a geographically-distributed, audience-weighted quorum. A 200 from one datacenter is not a health signal — it's an opinion from one vantage point. Fetch a real segment, check its age, weight the verdict by where your viewers actually are, and require a quorum before you believe the bad news. The whole thing fits in a Go worker, a SQLite table, and about a hundred lines of aggregation logic, and it will tell you the truth about your CDN far more often than your uptime monitor does.
Top comments (0)