Every data pipeline I've built on public sources has had the same silent assumption baked in: the URLs will still be there tomorrow. They won't. Public data has no SLA, no deprecation notice, no changelog email. Sources don't get sunset with a migration guide — they just stop: domain expires, the preview endpoint changes shape, the JSON field quietly disappears, the channel goes private, the whole site migrates to a new CMS and every path 404s behind a friendly "we've moved!" splash that returns HTTP 200.
That last one is the nastiest, and it's worth internalizing early: a 200 response is not a health check. A parking page, a cookie-consent shell, or a "maintenance" splash all return 200 with a body that contains zero facts. If your pipeline's definition of success is "request didn't fail," it will happily ingest a parking page at full confidence.
The three ways sources die (and what each one looks like)
- Hard death. DNS dies, 404s everywhere, the domain lands on a registrar auction page. This is the good case: it's loud, it's fast, and anyone watching HTTP status codes sees it within a cycle. My scraper flags it instantly.
- Soft death. Status codes stay 200; the shape rots. A field is renamed. An array becomes an object. The list renders empty inside a fully valid page shell. This is what actually kills you, because it degrades one field at a time: your dashboard keeps showing yesterday's numbers while only the newest slice is quietly hollow. Volume-band alarms (today's row count vs the 7-day median band) catch the loud version; schema-shape checks catch the quiet one — validate the response envelope, not just the status.
- Semantic death. The source keeps answering, but the meaning drifts: a channel changes topic, a feed adds an entirely new content type, a "recent" endpoint starts returning six-month-old items. No error, no shape change — the data is valid and the trend line goes weird. Only cadence/plausibility alarms (freshness watermarks, expected-interval checks, keyword-hit-rate drift) catch this class.
The kit that survived contact with all three
Save everything you fetch. Raw responses first, parsing second. When a source dies mid-project, the raw dump is the difference between "we can rebuild the parse and backfill" and "that data is gone." Storage is cheap; re-collection of a dead source is infinity-priced. My collector writes raw NDJSON before any transformation touches it, which has paid for itself twice — once to re-parse after a schema change, once to prove to myself which side of a discrepancy was the source lying.
Canary the parse, not just the request. Each source gets a tiny fixture — one known-good response, one known-good expected output. The fetch step fails loudly when the canary parse yields zero records on a page that historically produced records. Zero-from-history-nonzero is the signature of soft death.
Treat source death as an event with a date, not a bug. When GhostAction-style sources die (mine died between collection runs: a live, harvesting site one week, a registrar parking page the next), the pipeline should record the death — last-good timestamp, last-known shape, a tombstone in the feed registry. Downstream consumers then render "this source died Oct 13" honestly, instead of the dashboard showing a source with "no recent data" forever, which reads as my pipeline being lazy, not the source being dead.
Have a replacement ladder per feed. Before adding any source to production, write down in one line: if this dies, what's the fallback, and at what fidelity? Same data (mirror, archive.org snapshot, secondary publisher), same fact (derived from a neighboring source — slower, coarser), same topic (different feed, re-tuned alarms), or nothing (accept the gap, tombstone it, tell the consumer). The ladder doesn't have to be elaborate; it has to be written before the funeral, because post-mortem panic re-plumbing is how pipelines acquire permanent duct tape.
The honest summary
Private data loses integrity; public data loses existence. Every control above is cheap — fetch-and-store raw, canary the parse, tombstone deaths, write the ladder — and every one of them is worthless if written during the outage. Public data has no SLA. The most professional thing you can do for a source you don't control is to plan, in advance, for the day it stops existing — and make sure your pipeline knows the difference between "nothing happened today" and "there is no today anymore."
Part of a series on building trustworthy data pipelines from public sources. Previous: Your Dashboard Says "Live" — Prove It With Freshness Watermarks.
Bundles are snapshots with capture timestamps — because sources vanish, my dumps are dated receipts: free Telegram OSINT sample (pay what you want) · Premium 2-day dump, 6 channels ($30). The polite way to watch sources vanish: Your Scraper's Politeness Layer Deserves a Unit Test · Circuit Breakers for Scrapers.
Top comments (0)