Building a video health probe with Prometheus exporters in Go

A user in São Paulo emailed us on a Tuesday: the "Trending in Brazil" row on the homepage was blank, but everything else looked fine. I checked our uptime monitor. Green. 200 OK, 180ms, no alerts for six hours. I opened the page from a US IP and it looked perfect — full shelves, working search, thumbnails everywhere. Then I routed through a Brazilian exit node and saw exactly what the user saw: an empty br shelf, a 200 status, and no error anywhere in the logs.

That is the trap of running a multi-region video discovery site. At TrendVidStream we serve trending and streaming-platform discovery across eight regions, and the thing that breaks is almost never "the server is down." It is one region's ingest cron stalling, or a search index that silently stops answering for a single locale, while the HTTP layer keeps returning a cheerful 200. HTTP status codes describe whether a socket answered. They say nothing about whether the page is useful. This post walks through the health probe I built to close that gap: a Prometheus exporter in Go that measures video-specific health per region, backed by a cheap PHP health endpoint and cross-checked from an external Python vantage.

The failure HTTP status codes can't see

Our stack is deliberately boring: PHP 8.4 rendering pages, SQLite with an FTS5 full-text index for search, a per-region cron that pulls trending videos on a staggered schedule, and FTP-based deploys to LiteSpeed hosts. There is no Kubernetes, no service mesh, no fancy control plane to tattle on itself. That simplicity is a feature — but it means failures are quiet.

Here are the real incidents that a naive GET / -> 200 check will happily sleep through:

• A single region's shelf goes empty. The br cron hit a quota wall on the upstream API. The homepage still renders — it just renders an empty row. Status: 200.
• The whole site is stale. The cron process died three days ago. Every page loads instantly (great cache hit rate!) and serves videos from last week. Status: 200.
• Search silently degrades. An FTS5 index got corrupted during a deploy that overwrote videos.db mid-write. The MATCH query throws, the search controller catches it and returns an empty result set, and the homepage — which doesn't use search — looks totally healthy. Status: 200.
• Edge vs origin disagreement. The origin is fine, but the CDN cached a broken response for one region weeks ago and keeps serving it. From inside the datacenter everything is green.

What these have in common is that health is a property of content, not of the transport. So the probe has to look at content: how many videos are in a region's shelf, how old the newest one is, and whether search actually answers. Those become metrics, and metrics become alerts.

What a video health probe actually measures

Before writing any Go, I wrote down the four signals that would have caught every incident above. Each one maps to a Prometheus metric labeled by region:

• video_region_up — 1 if the region returned a usable payload (non-empty shelf), 0 otherwise. This is the blunt instrument.
• video_region_shelf_count — how many active videos are in the shelf right now. A slow drift toward zero is an early warning long before it hits zero.
• video_region_freshness_seconds — the age of the newest video. This is the metric that catches a stalled cron. On a healthy region it stays under our fetch interval; when it climbs past six hours, something upstream is stuck.
• video_region_search_ready — 1 if the FTS5 index answered a probe query for that region. This decouples "the page loads" from "search works."

Everything is a gauge. There are no counters here because I care about current state, not rates — and gauges let me write alert expressions that read like English. The one operational metric I add is video_region_probe_duration_seconds, because a region that is technically "up" but takes eight seconds to answer is a region I want to know about.

The Go exporter skeleton

I chose Go for the exporter for three reasons: the official client_golang library is excellent, a static binary drops onto any box with zero runtime dependencies, and goroutines make probing eight regions concurrently trivial. The main function is almost nothing — construct a registry, register one custom collector, and serve /metrics.

package main

import (
    "log"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    reg := prometheus.NewRegistry()

    regions := []string{"us", "gb", "de", "fr", "in", "br", "au", "ca"}
    collector := NewVideoHealthCollector(
        "https://trendvidstream.com",
        regions,
        8*time.Second,
    )
    reg.MustRegister(collector)

    http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))
    log.Println("video-health-probe listening on :9187")
    log.Fatal(http.ListenAndServe(":9187", nil))
}

The important design decision here is that this is a collector, not a set of pre-created gauges I mutate on a timer. With a custom collector, Prometheus drives the cadence: every time it scrapes /metrics, the collector does a fresh probe. That keeps the exporter stateless and means the metric timestamps line up exactly with the scrape. The tradeoff is that a scrape now takes as long as the slowest region probe, which is why the concurrency in the collector matters and why there's a hard timeout.

A custom collector that probes real endpoints

This is the heart of it. The collector fans out one goroutine per region, hits a lightweight health endpoint on the app, and emits const metrics from whatever comes back. MustNewConstMetric is the right tool because these values are computed fresh each scrape — there is no long-lived gauge to keep in sync.

package main

import (
    "context"
    "encoding/json"
    "net/http"
    "sync"
    "time"

    "github.com/prometheus/client_golang/prometheus"
)

type shelf struct {
    Region     string `json:"region"`
    VideoCount int    `json:"video_count"`
    NewestUnix int64  `json:"newest_unix"`
    FTS5Ready  bool   `json:"fts5_ready"`
}

type VideoHealthCollector struct {
    base    string
    regions []string
    timeout time.Duration
    client  *http.Client

    up          *prometheus.Desc
    shelfCount  *prometheus.Desc
    freshness   *prometheus.Desc
    probeTime   *prometheus.Desc
    searchReady *prometheus.Desc
}

func NewVideoHealthCollector(base string, regions []string, timeout time.Duration) *VideoHealthCollector {
    return &VideoHealthCollector{
        base:    base,
        regions: regions,
        timeout: timeout,
        client:  &http.Client{Timeout: timeout},
        up: prometheus.NewDesc("video_region_up",
            "1 if the region shelf responded with a usable payload", []string{"region"}, nil),
        shelfCount: prometheus.NewDesc("video_region_shelf_count",
            "number of videos currently in the region shelf", []string{"region"}, nil),
        freshness: prometheus.NewDesc("video_region_freshness_seconds",
            "age of the newest video in the region shelf", []string{"region"}, nil),
        probeTime: prometheus.NewDesc("video_region_probe_duration_seconds",
            "time taken to probe the region endpoint", []string{"region"}, nil),
        searchReady: prometheus.NewDesc("video_region_search_ready",
            "1 if the FTS5 search index answered for this region", []string{"region"}, nil),
    }
}

func (c *VideoHealthCollector) Describe(ch chan<- *prometheus.Desc) {
    ch <- c.up
    ch <- c.shelfCount
    ch <- c.freshness
    ch <- c.probeTime
    ch <- c.searchReady
}

func (c *VideoHealthCollector) Collect(ch chan<- prometheus.Metric) {
    var wg sync.WaitGroup
    now := time.Now()
    for _, region := range c.regions {
        wg.Add(1)
        go func(region string) {
            defer wg.Done()
            c.probeRegion(ch, region, now)
        }(region)
    }
    wg.Wait()
}

func (c *VideoHealthCollector) probeRegion(ch chan<- prometheus.Metric, region string, now time.Time) {
    start := time.Now()
    up, s := c.fetchShelf(region)
    ch <- prometheus.MustNewConstMetric(c.probeTime, prometheus.GaugeValue, time.Since(start).Seconds(), region)

    if !up {
        ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, 0, region)
        return
    }

    ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, 1, region)
    ch <- prometheus.MustNewConstMetric(c.shelfCount, prometheus.GaugeValue, float64(s.VideoCount), region)

    age := float64(now.Unix() - s.NewestUnix)
    if s.NewestUnix == 0 {
        age = -1 // sentinel: shelf has rows but no timestamp — treat as unknown
    }
    ch <- prometheus.MustNewConstMetric(c.freshness, prometheus.GaugeValue, age, region)

    ready := 0.0
    if s.FTS5Ready {
        ready = 1.0
    }
    ch <- prometheus.MustNewConstMetric(c.searchReady, prometheus.GaugeValue, ready, region)
}

func (c *VideoHealthCollector) fetchShelf(region string) (bool, shelf) {
    ctx, cancel := context.WithTimeout(context.Background(), c.timeout)
    defer cancel()

    url := c.base + "/health/shelf?region=" + region
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return false, shelf{}
    }
    req.Header.Set("User-Agent", "video-health-probe/1.0")

    resp, err := c.client.Do(req)
    if err != nil {
        return false, shelf{}
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return false, shelf{}
    }

    var s shelf
    if err := json.NewDecoder(resp.Body).Decode(&s); err != nil {
        return false, shelf{}
    }
    if s.VideoCount == 0 {
        return false, s // 200 with an empty shelf is a failure, not a success
    }
    return true, s
}

The subtle line is the last one in fetchShelf: a 200 response with video_count == 0 returns up = false. That single decision is what turns "the socket answered" into "the page is useful," and it is exactly the São Paulo incident encoded as a boolean. Note also that there is no shared mutable state — each goroutine writes only to the channel, which the Prometheus library synchronizes for us, so no mutexes are needed despite the fan-out.

Exposing per-region freshness from the app side

The exporter is only as good as the endpoint it probes. I did not want to reuse the real homepage controller — it's cached aggressively (LiteSpeed page cache in front of it), so a probe against it would measure cache health, not data health. Instead I added a dedicated /health/shelf endpoint that bypasses every cache, opens the SQLite database read-only, and reports the raw truth. Keeping it separate also means the probe can't accidentally poison the page cache.

<?php
declare(strict_types=1);

// public/health/shelf.php — cache-bypassing health surface for the Go probe.
// Reports the real per-region shelf state, never a cached render.

header('Content-Type: application/json');
header('Cache-Control: no-store');

$region = strtolower((string)($_GET['region'] ?? 'us'));
if (!preg_match('/^[a-z]{2}$/', $region)) {
    http_response_code(400);
    echo json_encode(['error' => 'bad region']);
    exit;
}

$db = new SQLite3(__DIR__ . '/../../data/videos.db', SQLITE3_OPEN_READONLY);
$db->busyTimeout(3000);

$stmt = $db->prepare(
    'SELECT COUNT(*) AS c, COALESCE(MAX(published_unix), 0) AS newest
       FROM videos WHERE region = :r AND is_active = 1'
);
$stmt->bindValue(':r', $region, SQLITE3_TEXT);
$row = $stmt->execute()->fetchArray(SQLITE3_ASSOC) ?: ['c' => 0, 'newest' => 0];

// Confirm FTS5 actually answers for this region — not just that the table exists.
$fts5Ready = false;
try {
    $probe = $db->querySingle(
        "SELECT COUNT(*) FROM videos_fts
          WHERE videos_fts MATCH 'video'"
    );
    $fts5Ready = $probe !== null;
} catch (\Throwable $e) {
    $fts5Ready = false; // a thrown MATCH means a corrupt or missing index
}

echo json_encode([
    'region'      => $region,
    'video_count' => (int)$row['c'],
    'newest_unix' => (int)$row['newest'],
    'fts5_ready'  => $fts5Ready,
], JSON_UNESCAPED_SLASHES);

Three things earn their place here. The busyTimeout(3000) matters because the ingest cron writes to the same SQLite file; without it, a probe landing mid-write would fail spuriously and page us for nothing. The regex on region keeps this endpoint from becoming an injection surface or an accidental full-table scanner. And the FTS5 MATCH probe is wrapped in a try/catch precisely because a corrupt index throws rather than returning empty — catching it is how video_region_search_ready earns its keep.

Cross-checking regions from a separate vantage

The Go exporter runs close to the origin, which is great for measuring the application but blind to anything that breaks between the CDN edge and the user. To catch the "edge cached a broken response" class of failure, I run a small Python checker from a box in a completely different network — a cheap VPS, not the FTP host the site deploys to. It hits the same public URL a real user would, so it sees whatever the edge is serving.

#!/usr/bin/env python3
"""External vantage check. Runs off-origin so it catches CDN/edge failures
the in-datacenter exporter cannot see. Exit code 1 if any region is bad."""

import concurrent.futures as cf
import json
import sys
import time
import urllib.request

BASE = "https://trendvidstream.com"
REGIONS = ["us", "gb", "de", "fr", "in", "br", "au", "ca"]
MAX_AGE = 6 * 3600  # a shelf older than 6h means the multi-region cron stalled


def probe(region: str) -> dict:
    url = f"{BASE}/health/shelf?region={region}"
    started = time.time()
    try:
        req = urllib.request.Request(url, headers={"User-Agent": "vantage-check/1.0"})
        with urllib.request.urlopen(req, timeout=8) as resp:
            payload = json.loads(resp.read())
        age = time.time() - payload["newest_unix"] if payload["newest_unix"] else None
        stale = age is not None and age > MAX_AGE
        return {
            "region": region,
            "ok": payload["video_count"] > 0 and not stale,
            "count": payload["video_count"],
            "age_h": round(age / 3600, 1) if age else None,
            "fts5": payload["fts5_ready"],
            "ms": round((time.time() - started) * 1000),
        }
    except Exception as exc:  # noqa: BLE001
        return {"region": region, "ok": False, "error": str(exc)}


def main() -> int:
    with cf.ThreadPoolExecutor(max_workers=len(REGIONS)) as pool:
        results = list(pool.map(probe, REGIONS))

    failed = [r for r in results if not r["ok"]]
    for r in sorted(results, key=lambda x: x["region"]):
        print(json.dumps(r))
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

This runs on a plain cron every ten minutes and pipes its exit code into a dead-simple alert. It intentionally duplicates some of the Go exporter's logic — that redundancy is the point. If the Go exporter says a region is healthy and the external vantage says it's stale, the disagreement itself tells me the problem is at the edge, not the origin. Two probes that can never lie the same way are worth more than one probe you trust completely.

Alerting on the metrics that matter

Metrics without alerts are just dashboards nobody looks at. Because everything is a labeled gauge, the alerting rules read almost like plain sentences. The for clauses are tuned to our staggered ingest schedule so that a single slow cron run doesn't page anyone.

groups:
  - name: video-health
    rules:
      - alert: RegionShelfDown
        expr: video_region_up == 0
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Region {{ $labels.region }} shelf is empty or unreachable"

      - alert: RegionShelfStale
        expr: video_region_freshness_seconds > 21600
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Region {{ $labels.region }} cron stalled — newest video is {{ $value | humanizeDuration }} old"

      - alert: RegionSearchDegraded
        expr: video_region_search_ready == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "FTS5 search index not answering for {{ $labels.region }}"

A few operational notes on tuning these:

• RegionShelfDown pages; the other two only warn. An empty shelf is user-visible right now. A stale-but-populated shelf or degraded search is bad, but it can wait for business hours.
• for: 10m on the down alert rides out a single failed scrape or a probe that lost a race with the ingest writer. One bad data point never pages.
• The freshness threshold is 6 hours because our slowest region's cron runs every few hours. Set this too tight and every normal cron cycle looks like an outage.
• Alert on absence too. In production I also add a rule for up{job="video-health-probe"} == 0 so that the exporter itself going dark pages someone — otherwise a dead probe reports no failures, which looks identical to perfect health.

What this caught in production

Within the first month, this probe caught three things our old "is the homepage 200" check never would have:

• A regional ingest that had been silently returning zero new videos for two days after an upstream API changed a field name. video_region_freshness_seconds climbed in a straight line and paged us on day one instead.
• An FTP deploy that overwrote videos.db while the cron was mid-write, leaving the FTS5 index inconsistent for one region. Search returned empty results and every page rendered fine — video_region_search_ready flipped to 0 and warned us in fifteen minutes.
• A CDN that had cached an empty de shelf for six hours. The Go exporter was green; the external Python vantage was red. The disagreement pointed straight at the edge, and a cache purge fixed it.

None of these were "the server is down." All of them were degraded content behind a 200. That is the entire reason this probe exists.

Conclusion

The lesson I keep relearning running a multi-region video site is that uptime is the wrong question. The right question is "is every region serving fresh, searchable content right now?" — and that is a content-level property no transport-level check can answer. A Prometheus exporter in Go is a clean way to encode that question: a stateless custom collector that fans out per region, a cache-bypassing app endpoint that reports raw database truth, an off-origin vantage that catches edge failures, and a handful of gauge-based alerts that read like English. The whole thing is a static binary and a PHP file — it fits the same boring, FTP-deployed stack it monitors, and it has turned "a user emailed us" into "we got paged first" more times than I'd like to admit. Start with the four signals that map to your actual user-visible failures, make each one a labeled gauge, and let the alert expressions fall out of that. The infrastructure is the easy part; deciding what "healthy" means is the work.