Most collection pipelines count a fetch as a success when the HTTP status starts with 2. That works right up until the day a site hands you a polite 200 OK with an empty body, a consent interstitial, or a "you must enable JavaScript" shell — and your pipeline happily records zero rows as a clean run.
I run a daily collector over public Telegram channel previews (no login, just the web view). Last month it handed me a "successful" run with zero messages from a channel that normally produces 30+ a day. Status codes: all 200. Exceptions: zero. The log said everything was fine. The channel had changed its preview rendering for datacenter IPs, and my pipeline's only signal that anything happened was the absence of data.
The three failure shapes that all look identical
When a collector reports "0 new records," it cannot tell you which of these it means:
- Nothing new — the source is quiet; zero is the truth.
- Nothing reachable — the source moved, blocked, or reshaped; zero is a symptom.
- Nothing parsed — the source changed shape under your parser; zero is a silent contract breach.
A status code distinguishes none of them. A 200 with a shell page is, to your metric, a total success.
What I do now: expect-each-run invariants
Instead of trusting the transport layer, I assert on the payload shape:
- Presence check: the response must contain at least one element matching the selector that carried data yesterday. Zero matches on a channel with 400+ messages in its archive is a failure, not a success.
-
Fingerprint check: I hash the DOM skeleton of the page (tag sequence, stripped of text). When the skeleton drifts more than a small threshold from the last accepted run, the run is flagged
shape-driftand the sample is archived for inspection. - Floor check: per source, a rolling median of daily yield. A run under ~20% of the median on a source that has been alive for a week pages me. Quiet days exist; three in a row on one source do not.
The cost is roughly 40 lines and one archived-sample folder. The payoff was concrete: the empty-200 switch on that channel was visible the same morning, not three days later when I happened to open the site in a browser.
The general rule
Transport success is the server's answer to "did you deliver bytes?" — never your answer to "did the data arrive?" The cheapest contract your pipeline can make is with the payload, not the protocol: define the smallest structure that means "data present," assert it every run, and treat a well-formed empty as the only honest zero.
How do you tell "source is quiet" from "source changed shape" — invariants like these, golden-page diffs, or something else entirely?
I run this collector daily and sell the sourced briefs it produces. The pay-what-you-want sample (every claim with its source permalink) is here: https://heyuhe.gumroad.com/l/ruldgm — and if you want the wheel this pipeline runs on (pip-installable, no API keys, plus the five recipes above as a file): https://heyuhe.gumroad.com/l/scxtul.
Free tooling from the same pipeline: Telegram OSINT bundle sample (PWYW) · premium 2-day full dump, $30 · GhostAction event brief, pay what you want · methodology posts free here on dev.to.
Top comments (2)
That 20% yield floor is useful, but a quiet source still needs an explicit empty-state signal. With BrowserAct Cloud output feeding n8n, I'd keep extraction status separate from record count and archive the source sample before replacing yesterday's data.
Separating extraction status from record count is exactly the discipline — the count is a claim, the status is a receipt, and collapsing them is how a collector learns to lie politely. Archive-before-replace is the second half of the same rule: yesterday's data is the only witness that can prove today's emptiness is real. In my pipeline the archived sample doubles as the replay fixture, so the day the parser breaks I can prove which input changed.