A scraper is a quiet failure machine. It does not crash loudly when a site changes; it succeeds its way into emptiness — 200 OK, valid HTML, zero items extracted, ship the empty file, sleep. My pipeline runs unattended at 07:00 every morning, so the question was never "does it work" but "how does it tell me it stopped working." Three alarms, in order of bluntness, cover the whole failure surface.
Alarm 1: volume. The day's row count must land inside a band around the trailing 7-day median. Telegram channels are human-paced; a morning with 12 messages when the median is 140 is a structural failure, not a slow day. Volume alarms catch: selector rot (page redesign → zero extractions), auth/expiry walls (200 with a login page), network egress failures. This alarm alone caught every real incident I've had. Cost: ten lines and a stats file.
Alarm 2: freshness of the input. The band check is relative to history, so it is slow on Mondays — after a quiet weekend the median drifts down and a dead pipeline sails inside the band. Fix: an absolute floor per source, checked per channel, not just in aggregate. Six channels in, at least k non-empty. Aggregate totals hide a dead source under three healthy ones; per-source floors don't.
Alarm 3: liveness of the watcher. The alarms are code; code fails too. The cron job writes a heartbeat file on completion; a second, dumber cron checks the file's mtime and screams if it's >26h old. This is the alarm that catches the alarm's own death — missed runs, disk full, the machine being off. Everything that watches something must be watched by something dumber.
The ordering matters: volume catches fast and lies occasionally (holidays, channel silences — triage by eye), the per-source floor removes the slow lies, the heartbeat removes the silent class of "the monitor never ran." Three blunt instruments, none clever, all cheap: a median, a floor, an mtime.
What I deliberately don't run: schema drift alerts on field presence (my rows have four fields and a lint step that fails the run — the lint is the schema alarm, inline not side-channel), and anything requiring a dashboard I have to open. An alarm I must look at is a chore; a message that arrives is a notification. At solo-dev scale, notifications beat dashboards every time.
The honest limit: alarms catch breakage, not drift. A site that slowly changes its format until selectors extract half-truths will sail through all three. That class is caught by shipping data people read — a buyer noticing two missing posts is my most sensitive sensor, which is why every bundle ships with row counts and channel names visible on the surface.
— 何雨禾 (heyuhe). The pipeline, its median, its floor and its heartbeat run daily; the bundles are where drift goes to die.
These three alarms guard the pipeline that feeds my data shop, which is why a silent crawler is a refund and not a shrug: the bundles ship with row counts, channel names, and the timestamps the alarms wrote — free Telegram OSINT sample (suggested $5) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9). Why the zero-events line deserves its own alarm: “No New Posts Today”. The storage it watches: append-only NDJSON.
Top comments (0)