Last month one of our eight edge regions started returning HTTP 200 for manifest requests but serving a truncated HLS playlist — the origin was up, the CDN was up, TLS was valid, and every naive uptime monitor in the world was green. Viewers in Singapore just saw a spinner. That incident is the reason I stopped trusting single-probe uptime checks and built a proper multi-region health-check aggregator for our video origins. I run TrendVidStream, a multi-region streaming-discovery platform that fans traffic across eight geographic regions, and "is the server responding" turned out to be almost useless as a health signal for video delivery. What you actually need is a correlated view: the same set of semantic checks, run from every region, folded into one status table you can reason about.
This post walks through the aggregator we run today. It's deliberately boring technology — PHP 8.4 for the reporting layer, SQLite with FTS5 for storage and searchable incident history, a Go probe binary for the actual concurrent checking, and cron plus FTP automation to distribute the probes to each region. No Kubernetes, no time-series database, no service mesh. It fits on the same LiteSpeed shared hosts that serve the site, and it has caught three real incidents that Pingdom-style monitors missed entirely.
Why HTTP 200 Is a Lie for Video
A video request is not a single round trip. A viewer hitting a playback URL triggers a chain: DNS resolution, the master .m3u8 manifest, a media playlist per rendition, then a stream of .ts or .m4s segments. Any one of those can fail while the others succeed, and the failure modes are region-specific because different regions resolve to different edge PoPs.
The checks that actually matter for us are:
-
Manifest fetch + parse. The master playlist must return 200 and parse into at least one valid variant stream with a
BANDWIDTHattribute. A byte-truncated manifest returns 200 with aContent-Lengthmismatch. -
Segment reachability. Pull the first segment referenced by the lowest-bitrate rendition and confirm it's non-empty and has a
video/*orapplication/octet-streamcontent type. - Time-to-first-byte per region. Absolute latency matters less than drift — a region whose TTFB doubled over an hour is degrading even if it's still "up."
- TLS expiry horizon. Certs that expire in under 14 days are a slow-motion outage.
-
Cross-region consistency. If seven regions see manifest revision
v=812and one seesv=799, that region is serving stale cache.
None of these are expressible as "ping the URL and check the status code." They need a probe that understands the protocol.
The Probe: A Small Go Binary
I chose Go for the probe specifically because it cross-compiles to a single static binary I can drop onto any host, and because goroutines make concurrent per-region checks trivial. Each region runs the same binary from cron; the region identity comes from an environment variable. The probe emits one JSON line per check to stdout, which the cron wrapper ships back to the aggregator.
package main
import (
"crypto/tls"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
type Result struct {
Region string `json:"region"`
Target string `json:"target"`
Check string `json:"check"`
OK bool `json:"ok"`
TTFBms int64 `json:"ttfb_ms"`
Status int `json:"status"`
Detail string `json:"detail"`
CertDays int `json:"cert_days"`
CheckedAt int64 `json:"checked_at"`
}
func probeManifest(region, url string) Result {
r := Result{Region: region, Target: url, Check: "manifest", CheckedAt: time.Now().Unix()}
client := &http.Client{Timeout: 8 * time.Second}
start := time.Now()
resp, err := client.Get(url)
if err != nil {
r.Detail = err.Error()
return r
}
defer resp.Body.Close()
r.TTFBms = time.Since(start).Milliseconds()
r.Status = resp.StatusCode
// Capture TLS expiry horizon on the same connection.
if resp.TLS != nil && len(resp.TLS.PeerCertificates) > 0 {
leaf := resp.TLS.PeerCertificates[0]
r.CertDays = int(time.Until(leaf.NotAfter).Hours() / 24)
}
body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
text := string(body)
// Semantic validation: a real master playlist advertises variants.
if resp.StatusCode != 200 {
r.Detail = fmt.Sprintf("status %d", resp.StatusCode)
return r
}
if !strings.Contains(text, "#EXTM3U") {
r.Detail = "missing #EXTM3U header"
return r
}
if !strings.Contains(text, "BANDWIDTH=") && !strings.Contains(text, "#EXTINF") {
r.Detail = "no variants or segments in manifest"
return r
}
r.OK = true
r.Detail = "ok"
return r
}
func main() {
region := os.Getenv("PROBE_REGION")
if region == "" {
region = "unknown"
}
tlsCfg := &tls.Config{MinVersion: tls.VersionTLS12}
http.DefaultTransport.(*http.Transport).TLSClientConfig = tlsCfg
enc := json.NewEncoder(os.Stdout)
for _, target := range os.Args[1:] {
enc.Encode(probeManifest(region, target))
}
}
A few deliberate choices worth calling out. The io.LimitReader cap at 1 MiB stops a misbehaving origin from streaming us a whole video during a health check. Reading the TLS peer certificate off the same response means we get cert expiry for free without a second handshake. And the probe never exits non-zero on a failed check — a failed check is data, not a probe error, so it goes into the JSON stream like everything else. The cron wrapper only cares whether the binary ran.
Getting Probes to Eight Regions Without a Fleet
Here is the part people over-engineer. You do not need agents-as-a-service or a config-management stack to run a probe in eight regions. Each of our regional hosts already runs cron for content fetch jobs. We ship the probe binary and a tiny wrapper to each host over FTP as part of the same deploy that pushes the site, then a per-host crontab entry runs it and POSTs the results back.
The deploy side is a shell loop over an lftp mirror — the same FTP automation pattern the rest of our infrastructure uses. The critical, hard-won lesson: the host list file must have Unix line endings, or a stray \r corrupts the remote path and the upload silently lands in the wrong directory.
#!/usr/bin/env bash
set -euo pipefail
# One line per region: label host user pass remote_dir
# File MUST be LF-only — verify with `cat -A regions.conf` (no ^M).
while IFS=' ' read -r label host user pass dir; do
[[ "$label" =~ ^#.*$ || -z "$label" ]] && continue
echo ">> shipping probe to ${label} (${host})"
lftp -u "${user},${pass}" "${host}" <<EOF
set ssl:verify-certificate no
set net:timeout 20
put -O "${dir}/bin" ./probe-linux-amd64 -o probe
put -O "${dir}/bin" ./run_probe.sh
chmod 755 ${dir}/bin/probe ${dir}/bin/run_probe.sh
bye
EOF
done < regions.conf
echo "all regions updated"
The per-host run_probe.sh is what cron actually invokes. It sets the region label, runs the probe against our origin URLs, and pipes the JSON lines back to the aggregator's ingest endpoint. If the POST fails, it spools the lines to a local file so the next run can retry — we never want a network blip between a region and the aggregator to look like an origin outage.
#!/usr/bin/env bash
set -uo pipefail
export PROBE_REGION="${1:-unknown}"
INGEST="https://status.trendvidstream.com/ingest"
SPOOL="$(dirname "$0")/spool.ndjson"
TOKEN_FILE="$(dirname "$0")/../.probe_token"
targets=(
"https://cdn.trendvidstream.com/live/master.m3u8"
"https://cdn.trendvidstream.com/vod/trending/master.m3u8"
)
# Append any spooled lines from previous failed runs, then this run's output.
{ [[ -f "$SPOOL" ]] && cat "$SPOOL"; ./bin/probe "${targets[@]}"; } > /tmp/probe_out.$$
if curl -fsS -m 15 \
-H "Authorization: Bearer $(cat "$TOKEN_FILE")" \
-H "Content-Type: application/x-ndjson" \
--data-binary @/tmp/probe_out.$$ "$INGEST" >/dev/null; then
: > "$SPOOL" # success — clear the spool
else
cp /tmp/probe_out.$$ "$SPOOL" # failure — keep for next run
fi
rm -f /tmp/probe_out.$$
Each region's crontab staggers the run so we don't hammer the origin from eight places simultaneously — region offsets of a couple minutes are enough:
# US-East
*/5 * * * * /home/site/monitor/bin/run_probe.sh us-east
# EU-West (offset 2 min)
2-59/5 * * * * /home/site/monitor/bin/run_probe.sh eu-west
The Aggregator: PHP 8.4 and SQLite FTS5
The ingest endpoint and the status dashboard are plain PHP 8.4 backed by SQLite. SQLite is the right call here for the same reasons it's the right call for the rest of the site: no separate database process, the whole thing is one file I can scp for a backup, and it comfortably handles the write volume of eight regions posting a handful of checks every five minutes.
The schema has two tables. checks is the raw append-only log. Then an FTS5 virtual table over the human-readable detail field so that when I'm debugging at 2 a.m. I can full-text search incident history — "show me every stale cache event in ap-southeast last week" — without writing LIKE '%...%' scans across a growing table.
<?php
declare(strict_types=1);
final class HealthStore
{
private \PDO $db;
public function __construct(string $path)
{
$this->db = new \PDO('sqlite:' . $path);
$this->db->setAttribute(\PDO::ATTR_ERRMODE, \PDO::ERRMODE_EXCEPTION);
$this->db->exec('PRAGMA journal_mode=WAL');
$this->db->exec('PRAGMA busy_timeout=5000');
$this->migrate();
}
private function migrate(): void
{
$this->db->exec(<<<SQL
CREATE TABLE IF NOT EXISTS checks (
id INTEGER PRIMARY KEY,
region TEXT NOT NULL,
target TEXT NOT NULL,
check_type TEXT NOT NULL,
ok INTEGER NOT NULL,
ttfb_ms INTEGER NOT NULL,
status INTEGER NOT NULL,
cert_days INTEGER NOT NULL,
detail TEXT NOT NULL,
checked_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_checks_recent
ON checks(target, region, checked_at DESC);
CREATE VIRTUAL TABLE IF NOT EXISTS checks_fts
USING fts5(detail, region UNINDEXED, content='checks', content_rowid='id');
CREATE TRIGGER IF NOT EXISTS checks_ai AFTER INSERT ON checks BEGIN
INSERT INTO checks_fts(rowid, detail, region)
VALUES (new.id, new.detail, new.region);
END;
SQL);
}
public function ingest(array $rows): int
{
$stmt = $this->db->prepare(<<<SQL
INSERT INTO checks
(region, target, check_type, ok, ttfb_ms, status, cert_days, detail, checked_at)
VALUES
(:region, :target, :check_type, :ok, :ttfb_ms, :status, :cert_days, :detail, :checked_at)
SQL);
$this->db->beginTransaction();
$n = 0;
foreach ($rows as $r) {
$stmt->execute([
':region' => (string)($r['region'] ?? 'unknown'),
':target' => (string)($r['target'] ?? ''),
':check_type' => (string)($r['check'] ?? 'manifest'),
':ok' => !empty($r['ok']) ? 1 : 0,
':ttfb_ms' => (int)($r['ttfb_ms'] ?? 0),
':status' => (int)($r['status'] ?? 0),
':cert_days' => (int)($r['cert_days'] ?? 0),
':detail' => (string)($r['detail'] ?? ''),
':checked_at' => (int)($r['checked_at'] ?? time()),
]);
$n++;
}
$this->db->commit();
return $n;
}
}
The ingest endpoint itself is thin: authenticate the bearer token in constant time, read the NDJSON body, decode line by line skipping malformed lines rather than rejecting the whole batch, and hand the rows to ingest().
<?php
declare(strict_types=1);
require 'HealthStore.php';
$expected = trim((string)file_get_contents(__DIR__ . '/.ingest_token'));
$auth = $_SERVER['HTTP_AUTHORIZATION'] ?? '';
$given = str_starts_with($auth, 'Bearer ') ? substr($auth, 7) : '';
if (!hash_equals($expected, $given)) {
http_response_code(401);
exit('unauthorized');
}
$raw = file_get_contents('php://input') ?: '';
$rows = [];
foreach (explode("\n", $raw) as $line) {
$line = trim($line);
if ($line === '') continue;
$decoded = json_decode($line, true);
if (is_array($decoded)) $rows[] = $decoded;
}
$store = new HealthStore(__DIR__ . '/data/health.db');
$count = $store->ingest($rows);
header('Content-Type: application/json');
echo json_encode(['ingested' => $count]);
Aggregation: Turning Rows Into a Verdict
Raw checks are noise. The value of an aggregator is the roll-up. For each target, I compute a per-region latest state and a cross-region consensus. The two questions the dashboard answers are "which regions are unhealthy right now?" and "is any single region an outlier?" — the second is how we caught the stale-cache incident.
The query that drives the dashboard pulls the most recent check per (target, region) using a window function, which SQLite 3.25+ supports natively:
public function currentStatus(int $staleAfter = 900): array
{
$sql = <<<SQL
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY target, region
ORDER BY checked_at DESC
) AS rn
FROM checks
WHERE checked_at > :floor
)
SELECT target, region, ok, ttfb_ms, cert_days, detail, checked_at
FROM ranked WHERE rn = 1
ORDER BY target, region
SQL;
$stmt = $this->db->prepare($sql);
$stmt->execute([':floor' => time() - 3600]);
$latest = $stmt->fetchAll(\PDO::FETCH_ASSOC);
$out = [];
foreach ($latest as $row) {
$t = $row['target'];
$out[$t] ??= ['regions' => [], 'healthy' => 0, 'total' => 0, 'ttfbs' => []];
$stale = (time() - (int)$row['checked_at']) > $staleAfter;
$healthy = ((int)$row['ok'] === 1) && !$stale;
$out[$t]['regions'][$row['region']] = [
'healthy' => $healthy,
'ttfb_ms' => (int)$row['ttfb_ms'],
'cert_days' => (int)$row['cert_days'],
'detail' => $stale ? 'stale (no recent probe)' : $row['detail'],
];
$out[$t]['total']++;
if ($healthy) $out[$t]['healthy']++;
if ($healthy) $out[$t]['ttfbs'][] = (int)$row['ttfb_ms'];
}
// Outlier detection: a healthy region whose TTFB is >2.5x the median.
foreach ($out as $t => &$agg) {
$ttfbs = $agg['ttfbs'];
sort($ttfbs);
$n = count($ttfbs);
$median = $n ? ($ttfbs[intdiv($n, 2)]) : 0;
foreach ($agg['regions'] as $region => &$r) {
$r['slow'] = $median > 0 && $r['ttfb_ms'] > $median * 2.5;
}
$agg['median_ttfb'] = $median;
unset($agg['ttfbs']);
}
return $out;
}
The roll-up gives me a compact verdict per target: 6/8 healthy, median TTFB 140 ms, with two regions flagged and one flagged as slow even though it's technically up. That "slow but up" flag is the single most useful output of the whole system, because it's a leading indicator. Regions almost never fail cold; they drift, TTFB creeps, then a segment fetch times out. Catching the drift buys you the window to fail over before viewers notice.
The staleness check is just as important as the OK flag. If a region's cron stops posting — the host is down, the FTP deploy corrupted the wrapper, the token rotated — the absence of data is itself an outage signal. Treating "no recent probe" as unhealthy rather than "last known good" is the difference between a monitor you trust and one that lies to you during exactly the incidents you built it for.
Alerting Without Alert Fatigue
One region blipping for one cycle is noise; two consecutive failures across a majority of regions is an incident. I gate alerts on both persistence and breadth: a target only pages when it has been unhealthy for two consecutive check cycles in at least half the regions, or when any region reports a cert horizon under 14 days. Everything else lands in the dashboard and the FTS5-searchable log, where I can review it on my own schedule.
The FTS5 table earns its keep during postmortems. After that Singapore incident I ran a single query — SELECT region, count(*) FROM checks_fts('truncated OR stale') GROUP BY region — and immediately saw the failures were concentrated in one PoP, which told me it was a cache-invalidation problem at one edge rather than an origin problem. That's a thirty-second answer that would have been a painful grep across log files in most setups.
What I'd Tell You to Steal
The architecture is intentionally unglamorous, and that's the point:
- Probe semantics, not status codes. For video, parse the manifest and pull a real segment. HTTP 200 tells you almost nothing.
- Run the same probe binary everywhere, keyed by an env var. One Go build, eight regions, zero per-region code.
- Treat missing data as unhealthy. A silent region is an outage until proven otherwise.
- Store raw, aggregate on read. SQLite with a window-function query is more than fast enough, and keeping the raw log means you can compute new roll-ups later without re-instrumenting.
- Add full-text search to your incident history early. FTS5 costs one virtual table and a trigger, and it changes postmortems from archaeology into queries.
- Watch drift, not just death. The "up but 2.5x slow" flag is your earliest warning.
We've been running this across all eight regions for a few months now on the same shared LiteSpeed hosts and FTP automation that deploy the rest of the platform. It has never needed a dedicated server, and it has caught degradations our previous external monitor rated as fully green. If you're delivering video across regions and your health check is a single curl -I from one datacenter, you don't have a health check — you have a coin flip. Build the aggregator; your viewers will notice the difference before your dashboard does.
Top comments (0)