Every guide to scraping t.me/s/<channel> says the same thing: you can get the latest ~20 posts per fetch, no login. Nobody warns you about what the number 20 hides. The public preview layer is a window, not a firehose, and the shape of that window is the first thing your ingestion design has to respect.
The window slides with new posts, not with your polling. Fetching the preview page repeatedly within an hour returns overlapping content; the page is a function of the channel's newest posts, not of wall-clock time. Poll cadence should be event-triggered (new post detected at the top of the page) with a low-frequency heartbeat as backstop - not a fixed 1-minute loop that re-ingests 95% of the same HTML.
Post IDs are the ingestion cursor. Every preview page carries data-post="<channel>/<id>" per post. IDs are sequential within a channel. Store the max ID seen per channel; on the next fetch, anything with ID > last_seen is new. This is the cheapest correct dedupe, and it makes out-of-order fetches idempotent.
The coverage ceiling is structural. The preview exposes a contiguous slice of recent history. The gap between the oldest ID visible and the newest is your coverage window, and it is not the channel's full history - deleted posts, and posts beyond the window, are invisible. Two consequences your pipeline must encode:
- Ingested corpus = population, not census. Every downstream "absence" claim ("the channel never mentioned X") is only valid within the observed ID range. Store the range with every export; analysts reading a corpus without ID ranges make false-negative claims within a week.
- If your poller sleeps too long between fetches on a busy channel (warzone channels post 100+/day), posts fall off the window unobserved. The fix is cadence sized to the channel's posting rate - measure posts/day, poll at a fraction of the mean inter-post interval. A 500-channel set with one fixed cadence is a pipeline of silent coverage holes.
The free lunch in the gap. The visible window shrinks relative to total history on busy channels, which means the visible slice is biased toward the recent topic mix. Trend claims over "the channel's posts" from a preview crawl are recency-weighted by construction. Say so in the export, because the data does.
The ingestion cadence rules, cursor logic, and coverage-range bookkeeping are packaged in the Telegram & Web OSINT Bundle ($5). Free sample brief shows the export format with ID ranges included.
Runs free on GitHub Actions - no server, no paid APIs.
Top comments (1)
tr.ee/dev-to