DEV Community

Yuhe He
Yuhe He

Posted on

Telegram Scraping in Python Without an API Key: The Public-Preview Collector That Survives

#python #telegram #scraping #osint

Most "Telegram scraping in Python" tutorials tell you to install Telethon, register an API ID with your phone number, and log in as a user bot. Fine — until you're monitoring 200 public channels and your collection dies the day Telegram bans the session, rate-limits the account, or the phone that owns the number changes SIMs.

For public channels, you don't need an account at all. The web preview at t.me/s/<channel> returns the last ~20 posts as plain HTML. Scrape that, and your "API" is a GET request with zero credentials.

The shape of a collector that survives

import requests, re, json, time, random
from pathlib import Path

CH = Path("channels.txt").read_text().split()
seen = json.loads(Path("seen.json").read_text()) if Path("seen.json").exists() else {}
raw = Path("raw"); raw.mkdir(exist_ok=True)

for ch in CH:
    r = requests.get(f"https://t.me/s/{ch}", timeout=15,
                     headers={"User-Agent": "Mozilla/5.0"})
    if r.status_code != 200:
        continue                      # degrade: miss, don't crash
    html = r.text
    (raw / f"{ch}-{int(time.time())}.html").write_text(html)
    ids = re.findall(r'data-post="[^"]*-(\d+)"', html)
    new = [i for i in ids if i not in set(seen.get(ch, []))]
    seen.setdefault(ch, []).extend(new)
    time.sleep(3 + random.random() * 4)   # jitter between channels
json.dump(seen, open("seen.json", "w"))
Enter fullscreen mode Exit fullscreen mode

The four rules that separate a toy from a collector

  1. Store raw first, parse later. Raw HTML per fetch, timestamped on disk. Parsers improve; raw never changes. Re-parse free, re-fetch forever.
  2. IDs, not text. data-post IDs are per-channel and sequential. Diff on IDs and you get exact new-post detection; diff on text and you get duplicate nightmares.
  3. Persist seen. In-memory dedupe resets on every restart — which is exactly when collectors restart.
  4. Degrade, don't crash. A 429, a 503, a network blip: skip the cycle, catch up against the preview window. Unattended means the loop assumes it will be punched.

When you do want the real API

Private channels, media downloads at scale, posting, search — that's Telethon/MTProto territory, with account risk attached. Public monitoring is a solved problem at the preview layer; pay the account-risk tax only for capabilities you actually use.

I run the pattern above across ~200 channels for market and OSINT signals. The playbook — cadence design, ?before= cursor paging for backfill, repost-ring flags, wording-diff alerting — is the $5 bundle: https://heyuhe.gumroad.com/l/poddr. Free sample brief, pay what you want, $0 fine: https://heyuhe.gumroad.com/l/ruldgm.

Top comments (0)