DEV Community: Esteban Ortega

How to Scrape Telegram Channels Without the API (No Login, No Phone Number)

Esteban Ortega — Fri, 31 Jul 2026 05:34:32 +0000

Every guide to scraping Telegram starts the same way: register an application at my.telegram.org, get an api_id and api_hash, log in with your phone number, manage a session string with Telethon or Pyrogram, and hope your account doesn't get limited.

For public channels, none of that is necessary. Telegram publishes a full web preview of every public channel that anyone — including a plain HTTP client — can read. Here's how it works, and where its limits are.

The t.me/s/ trick

Every public channel has a server-rendered preview at:

https://t.me/s/{channel_name}

Open https://t.me/s/telegram in a browser with JavaScript disabled — the messages are still there. It's plain HTML, served to any User-Agent, with no login, no cookies, no API key, and no bot token.

Even better: it paginates. Each page shows ~20 messages, and appending ?before={message_id} walks backward through history:

import re, requests

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ..."
s = requests.Session(); s.headers["User-Agent"] = UA

html = s.get("https://t.me/s/durov").text
ids = [int(x) for x in re.findall(r'data-post="[^/]+/(\d+)"', html)]
older = s.get(f"https://t.me/s/durov?before={min(ids)}").text  # next 20, older

What's actually in the preview

More than most people assume. Each tgme_widget_message block contains:

Full message text (with formatting markup you can strip or keep)
Timestamp — a proper ISO 8601 datetime attribute
View count ("1.1M" style, abbreviated)
Reactions — per-emoji counts, including paid Star reactions. This surprises people; most scrapers never parse the reactions footer at all
Media — direct CDN URLs for photos and video files (cdn*.telesco.pe), plus thumbnails
Links and link previews, forwarded-from attribution, reply context, post author
Channel-level: title, description, subscriber count, avatar

What's not there: discussion-group comments, poll internals, subscriber-only content, and anything from private channels or groups — those genuinely require the real API and an account. Individual user accounts and bots have no preview either.

The gotchas (why "just regex it" grows teeth)

Non-channels return HTTP 200 with an empty preview, not a 404 — so "no messages" and "not a public channel" look identical unless you also check for channel metadata.
The message text div ends at the first </div> — greedy patterns swallow the footer, reactions, and "VIEW IN TELEGRAM" boilerplate into your text.
Counts come abbreviated (10.2K, 1.1M) and need normalizing; custom emoji reactions are <tg-emoji> elements with IDs, not characters.
Albums, voice notes, stickers, and documents each have their own markup variants.
At volume you'll want polite pacing and IP rotation — Telegram is tolerant, but not infinitely.

The maintained shortcut

I packaged all of the above as an Apify Actor: Telegram Channel Scraper. Give it channel names in any form (durov, @durov, t.me/durov), get structured JSON/CSV:

{
  "channel": "durov",
  "message_id": 527,
  "datetime": "2026-06-15T12:04:11+00:00",
  "text": "🏆 Telegram is launching a $200,000 contest for content creators.",
  "views": 21900000,
  "total_reactions": 514000,
  "reactions": [{ "emoji": "⭐paid", "count": 24900 }],
  "media_type": "video",
  "videos": ["https://cdn4.telesco.pe/file/...mp4"],
  "url": "https://t.me/durov/527"
}

It handles the pagination, the empty-preview trap, reaction parsing, media extraction, and keyword filtering — priced per message (about a tenth of a cent), runs on a schedule for monitoring, and plugs into Sheets/webhooks/LLM pipelines via the Apify API and MCP. A 50-message pull across two channels takes about six seconds.

Typical uses: monitoring crypto/announcement channels, OSINT with timestamps and reach metrics, competitor watching, and turning channels into clean text datasets for RAG.

FAQ

Is this against Telegram's rules? It reads content Telegram itself publishes to any anonymous web visitor. No authentication is bypassed and no private data is touched.

Can I get comments or poll results? No — those aren't in the public preview. That's real-API territory (Telethon/Pyrogram with an account).

How far back can I scrape? The ?before= pagination walks to the channel's first message if you let it.

Why not just use the Bot API? Bots must be added to a channel by an admin. The preview needs nothing.

If you hit a channel that parses oddly, open an issue on the actor — real examples are how coverage improves.

How to Download Every Ad a Competitor Runs on Google (and Why Most Scrapers Return Empty)

Esteban Ortega — Wed, 29 Jul 2026 22:00:24 +0000

Google publishes every ad an advertiser is currently running. Not leaked, not scraped from private data — published, on purpose, at adstransparency.google.com, as part of its ads-accountability program. Search any brand and you'll see their whole live creative library: every image ad, every HTML5 unit, when each first and last ran, and in which countries.

For anyone doing competitive ad research, that's the primary source. The catch is what's not there: an export button. You can look at ads one at a time in a web UI, and that's it.

What the Transparency Center actually gives you

Three ways to find an advertiser:

By name — resolved through a suggestion service (type "Nike", get candidate advertisers).
By advertiser ID — every advertiser has a stable AR… identifier, visible in the page URL. This is the deterministic option.
By domain — with a quirk: Google exposes match counts for a domain but not a clean domain→advertiser mapping, so some domains only resolve to candidates.

Per creative, the underlying data includes the format (image / HTML5), preview URLs, first-shown and last-shown timestamps, and — one level deeper — per-region last-shown dates and every render variant of the creative.

Why DIY scrapers (and several paid ones) silently return nothing

The site is a JavaScript app talking to internal RPC endpoints, and if you build against them yourself, four landmines are waiting:

The payload shape drifts. Google periodically reshuffles the request format. A scraper built on last quarter's payload doesn't error — it receives an empty {} and happily reports "this advertiser runs zero ads." That's the worst possible failure mode for research data: it looks like an answer. It's the main reason the most-installed Transparency scraper on Apify sits at ~2.2★ — check its reviews and you'll see "returns empty" over and over.
Pagination is a token chain. Results come in pages linked by continuation tokens. Stop after the first page — which many tools quietly do — and a 900-creative advertiser looks like a 100-creative advertiser. No error, just silently wrong.
An empty response is genuinely ambiguous. Zero results can mean a dormant advertiser, a soft-block on your IP, or payload drift. Telling those apart requires cross-checking Google's own advertiser-level ad counts — most tools don't bother.
The asset URLs expire. The simgad image links die quickly. A swipe file of URLs rots in weeks unless you persist the actual bytes at collection time.

None of this is hard to handle once — it's hard to keep handled, because the drift is ongoing.

The maintained option

I run enough of this research that I packaged the whole thing as an Apify actor: Google Ads Transparency Scraper. Design goals, in order:

Complete captures or a loud error — never a silent empty. Payload drift, soft-blocks, and mid-pagination truncation all produce explicit error/notice records, and a truncated run is marked partial: true on the advertiser summary. A genuine zero (Google's own counts confirm no ads) is reported as a clean zero.
Full token pagination. Every creative, not the first page. Each record carries the creative ID, format, preview/image URL, first/last shown, and a deep link back to the Transparency Center so anyone can verify.
Optional depth. Per-region last-shown dates and render variants (one enrichment call per creative), and optional asset rehosting that stores the actual image bytes before the URLs expire.
The advertiser summary tells you if you got everything. It carries both the count you scraped and Google's own reported totals, so a capped or truncated run is visible instead of invisible.

curl -X POST "https://api.apify.com/v2/acts/silentshadow55~google-ads-transparency-actor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"advertiserName": "Nike", "region": "US", "maxCreativesPerAdvertiser": 300}'

Pricing is per creative (about a tenth of a cent), which for context is a different shape than SerpApi's metered Transparency API (~$25/mo floor) or ad-spy dashboards like AdSpy/BigSpy ($9–459/mo seats): you pay for the advertisers you actually check, and the raw JSON is yours to pipe into Sheets, a dashboard, n8n, or an AI agent.

Is this okay to do?

The Ads Transparency Center is Google's own public accountability database — the same pages it shows any visitor, with no login, no gate, and no terms click-through. Reading it politely (rate-limited, no CAPTCHA or WAF circumvention — this actor does neither) is reading public data that exists specifically so ads can be scrutinized. Use it for competitive research, not for anything the transparency program wasn't built for.

I build data tools on Apify. If an advertiser, region, or format misbehaves, open an issue on the actor — loud errors are the product, so bugs get fixed fast.

pytrends Is Dead. Here's How to Get Google Trends Data in 2026

Esteban Ortega — Tue, 28 Jul 2026 23:25:38 +0000

If you've landed here, you probably just discovered one of two things: pytrends — the unofficial Google Trends library the entire Python ecosystem relied on for a decade — was archived in April 2025 and no longer works reliably, or Google's long-promised official Trends API is still in limited-access alpha and you can't get in.

Meanwhile you have a dashboard, a research pipeline, or an SEO report that needs interest-over-time data. Here's the honest state of your options in 2026.

What actually powers Google Trends

There's no public API, but the Trends website itself is a JavaScript app fed by JSON endpoints — the same data you see in the charts, served as structured JSON. The flow is a two-step "token dance":

Call /trends/api/explore with your keywords, geo, and timeframe. It returns a set of widgets, each with a token — one for the timeline, one for the region map, one each for related queries and topics.
Call the matching widgetdata endpoint (multiline, comparedgeo, relatedsearches) with that token to get the actual data.

Two quirks to know: every response starts with an anti-XSSI prefix ()]}') you must strip before parsing, and you need a session cookie from an initial page visit — amusingly, Google can answer that first HTML request with HTTP 429 while still setting the cookie and serving the JSON API just fine.

Option 1: Build it yourself

A minimal working example (Python, plain requests, no browser):

import json, requests

s = requests.Session()
s.headers.update({"User-Agent": "Mozilla/5.0 ...", "Accept-Language": "en-US,en;q=0.9"})
s.get("https://trends.google.com/trends/explore")  # cookie warm-up (ignore the status)

req = {"comparisonItem": [{"keyword": "standing desk", "geo": "US", "time": "today 12-m"}],
       "category": 0, "property": ""}
r = s.get("https://trends.google.com/trends/api/explore",
          params={"hl": "en-US", "tz": "360", "req": json.dumps(req)})
widgets = json.loads(r.text.split("\n", 1)[1])["widgets"]  # strip )]}' prefix

w = next(x for x in widgets if x["id"] == "TIMESERIES")
r = s.get("https://trends.google.com/trends/api/widgetdata/multiline",
          params={"hl": "en-US", "tz": "360",
                  "req": json.dumps(w["request"]), "token": w["token"]})
timeline = json.loads(r.text.split("\n", 1)[1])["default"]["timelineData"]

This works today. The part that stops being fun is everything around it:

Rate limits. Google 429s aggressively and unpredictably per IP. Any real workload needs pacing, exponential backoff, cookie-session rotation, and a proxy pool.
Maintenance. These are internal endpoints; Google reshuffles them occasionally (that's part of what killed pytrends once its maintainers stopped chasing).
Edge cases. Example: the region-map widget accepts a resolution override, but forcing a resolution that doesn't fit the geo (say, REGION on a worldwide query) makes Google return HTTP 500. There are a dozen little landmines like this.

If you need Trends data once, copy the snippet above and enjoy. If you need it every day, you've just adopted a small maintenance hobby.

Option 2: Use it as a maintained API

I got tired of maintaining exactly that hobby, so I packaged the whole thing as an Apify Actor: Google Trends Scraper. One call gets you, per term:

Interest over time at Google's finest granularity for your window, with partial-period flags plus computed average and peak
Interest by region (countries, states, cities, or US metro areas)
Related queries and topics — top and rising, with "Breakout" markers (the early-signal goldmine for SEO and product research)
True compare mode — up to 5 terms normalized on one shared 0–100 scale, like the Trends UI comparison
All five properties: Web, YouTube, News, Images, Shopping

No browser anywhere in the stack — it's the same plain-HTTP flow described above, plus the unglamorous production parts: pacing, backoff, session rotation, per-session proxy identities, and per-term error reporting so one flaky keyword doesn't kill a 50-term run.

curl -X POST "https://api.apify.com/v2/acts/silentshadow55~google-trends-actor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchTerms": ["standing desk", "walking pad"], "geo": "US", "compareAllTerms": true}'

Results export as JSON/CSV/Excel, run on a schedule for daily tracking, and plug into Google Sheets, Make, Zapier, LangChain — or AI agents via Apify's MCP server ("is interest in X growing?" answered with real data). Pricing is per term (about half a cent), with free platform credits covering a trial.

FAQ

What do the 0–100 numbers mean? Relative popularity within your request's timeframe/geo/term set — 100 is the peak of that window, not an absolute volume. To compare terms against each other, they must be requested together (that's what compare mode does).

Which timeframes work? now 1-H through today 5-y, all, and custom ranges like 2025-01-01 2025-06-30. Shorter windows return finer granularity (hourly/daily vs weekly).

Is Google's official API coming? It was announced in 2025 and remains limited-access alpha. When it ships broadly, use it — until then, the website's endpoints are the only public source.

Is scraping Trends legal? It reads the same public, aggregated, anonymized statistics Google shows every visitor at trends.google.com.

I build data tools on Apify — this one exists because I needed it. If a term, geo, or timeframe misbehaves, open an issue on the actor and it'll get fixed fast.