DEV Community

Yuhe He
Yuhe He

Posted on

Why "No New Posts Today" Is the Most Dangerous Line in Your Crawler's Log

A data collection job that reports "0 new records" is making three claims at once, and your log can't tell them apart:

  1. The source is quiet.
  2. The source is unreachable.
  3. My parser just died on a page it hadn't seen before.

All three serialize to the same line. All three let the job exit 0. The 3am job treats them identically, the dashboard stays green, and you find out at 9am — if you're lucky, from a dashboard; if you're not, from a customer.

Quiet is rare; you're probably not seeing quiet

Field note from running an unattended crawler over public Telegram channels: across weeks of daily polling of sources that publish multiple times per day, genuine zero-days are nearly unheard of. When a busy source reports zero, the base rates strongly favor failure modes 2 and 3.

So the naive default — zero means quiet — isn't neutral. It's a bet against the base rate, placed silently, by the person who wrote the logging line, not by the person watching the dashboard.

Codify the glance

Experienced operators develop a reflex: "that's suspiciously clean." The fix is to write the reflex down so a machine can execute it at 3am. Two rules have carried most of the value in my pipelines:

Rule 1: The canary source. Keep at least one source in every pipeline that provably publishes daily (an official changelog channel, a status feed). If the canary returns zero, the entire run is FAILED — regardless of what every other source reported. One flaky-by-design source becomes the pipeline's smoke detector.

Rule 2: Zero from an active source is a failure, not a status. Encode each source's publishing cadence as a threshold: sources that posted on 9 of the last 10 days returning 0 counts as an alert, not a quiet day. You can infer these thresholds from your own history; you don't need magic numbers.

Both rules share one principle: quiet is a claim, and claims need evidence. The evidence is the run manifest — per-source fetch status, HTTP code, record count, timestamp — written before any summary exists.

The parser's last words

Failure mode 3 deserves its own section because it's the one that feels safe. Your HTTP client got a 200. The page is right there. The parser selected div.post > .body and found... nothing. Old shape, div.post > .content — 200, records = 0, exit 0, log says quiet day.

Selectors die in production the way stars go supernova: brightly, briefly, leaving something dark behind. Three cheap mitigations:

  • Assertions at parse time: if a fetch returns 200 and the last successful parse averaged 14 records, a 200 yielding 0 records trips the same alert as a timeout.
  • One canary selector: pick a DOM fragment that has been stable for months (a footer, a site logo). No match = shape changed = alert, independent of record counts.
  • Keep raw payloads for one rotation. Parse failures you can replay are data bugs; parse failures you can't replay are mysteries. A day of raw HTML is a cheap insurance premium.

What the exit code should mean

After a run, exactly one of these is true, and the log must say which:

  • OK(n) — fetched, parsed, n records, n consistent with the source's history.
  • FAILED(reason) — unreachable, shape-changed, canary-quiet, parse-empty-on-active-source.
  • SUSPECT — everything parsed, totals 40%+ below trailing average. Not failed; worth a human glance.

There is no OK(quiet) unless the canary corroborated quiet and every source's zero is individually justified. "No new posts today" is allowed to appear in the log exactly once per day, at most, and only when the evidence supports it.

The line is dangerous because it's comfortable. It ends conversations. It makes the dashboard green. Design your pipeline the way you'd design a smoke detector: assume the silence is the fire, until the evidence says it isn't.

What's your pipeline's loudest false alarm — the alert you got tired of, or the one you're glad you got? Curious what tripwires other people codified after getting bitten.

I run this collection pipeline daily and sell the sourced briefs it produces. The pay-what-you-want sample (every claim with its source permalink) is here: https://heyuhe.gumroad.com/l/ruldgm — and the pip-installable wheel this whole pipeline runs on (no API keys, plus the recipe files) is here: https://heyuhe.gumroad.com/l/scxtul.

Free tooling from the same pipeline: Telegram OSINT bundle sample (PWYW) · premium 2-day full dump, $30 · GhostAction event brief, pay what you want · methodology posts free here on dev.to.

Top comments (0)