DEV Community: Kyungmin Lee

pytrends is dead. Here is how Google Trends actually loads data in 2026 (including Trending Now)

Kyungmin Lee — Tue, 15 Sep 2026 04:41:15 +0000

If you have tried pytrends recently you know the story: TooManyRequestsError on the first call, dailytrends returning 404, and an unmaintained repo. Google redesigned Trends in 2024, moved the "Trending Now" feed to a different backend, and started returning 429 to anything that looks like a script.

The good news is that the site is still a plain JSON API underneath. Here is the map I put together while building a browserless client with nothing but httpx.

The two-step token dance (explore → widgetdata)

Every Trends chart is loaded in two requests:

GET /trends/api/explore?hl=en-US&tz=-540&req=<JSON> — you describe what you want (keywords, geo, timeframe, property) and Google answers with a list of widgets, each carrying a token and a request object.
GET /trends/api/widgetdata/<kind>?token=<token>&req=<JSON(widget.request)> — you send the widget's own request back with its token and get the data.

The req for explore looks like this:

import json

def explore_req(keywords, geo="", timeframe="today 12-m", category=0, prop=""):
    return json.dumps({
        "comparisonItem": [{"keyword": k, "geo": geo, "time": timeframe} for k in keywords],
        "category": category,
        "property": prop,          # "" web, "images", "news", "youtube", "froogle"
    }, separators=(",", ":"))

Widget IDs tell you which endpoint to call:

Widget id	Endpoint	What you get
`TIMESERIES`	`/trends/api/widgetdata/multiline`	`timelineData[]` with `time`, `formattedTime`, `value[]`, `isPartial`
`GEO_MAP` (or `GEO_MAP_0` … per keyword)	`/trends/api/widgetdata/comparedgeo`	`geoMapData[]` with `geoCode`, `geoName`, `value[]`
`RELATED_QUERIES_0`	`/trends/api/widgetdata/relatedsearches`	two `rankedList` entries: TOP then RISING
`RELATED_TOPICS_0`	same endpoint, `keywordType: ENTITY`	topics — empty for logged-out sessions (see below)

Every response starts with the anti-JSON-hijacking prefix )]}'\n — strip it before json.loads.

Two details that cost me an evening:

The token is bound to the exact request JSON you send back. Changing userConfig → 401. The only fields you may edit are resolution (COUNTRY/REGION/CITY/DMA) and includeLowVolumeGeos on the geo widget.
Explore stamps userConfig.userType. Logged-out HTTP sessions get USER_TYPE_SCRAPER; a real Google login gets USER_TYPE_LEGIT_USER. Time series, geo and related queries work fine for scrapers, but related topics return {"rankedList": []}. If you need topics you need your own cookies — I don't ship that.

Where Trending Now went

/trends/api/dailytrends and /trends/api/realtimetrends are gone (404). The new Trending Now page is a Google "batchexecute" app:

POST https://trends.google.com/_/TrendsUi/data/batchexecute?rpcids=i0OFE&source-path=/trending&hl=en-US&rt=c
Content-Type: application/x-www-form-urlencoded
f.req=[[["i0OFE","[null,null,\"US\",0,\"en-US\",24]",null,"generic"]]]

The inner argument list is [null, null, geo, 0, hl, hours]. The response is chunked: )]}'\n\n<length>\n[["wrb.fr","i0OFE","<json string>",...]] — read the length prefix, parse the chunk, then parse the JSON string inside it. Each trend item is a positional array:

[title, null, geo, [start_ts], [end_ts] | null, null, search_volume, null,
 pct_increase, [breakdown_keywords], [category_ids], [[article_id, lang, geo], ...],
 normalized_title]

I keep the mapping in one parse_trending() function and treat it as "will break someday". Note that the category argument in the request does not filter server-side (I got 566 vs 567 items with and without it), so filter on category_ids client-side.

There is also /trending/rss?geo=US, a plain RSS feed with ht:approx_traffic and ht:news_item, which makes a nice fallback when the batchexecute format changes.

Staying under the 429 radar

Google Trends rate-limits per IP and, more subtly, per session:

Warm up first: GET https://trends.google.com/ (302 → /trends/) sets the NID cookie. Endpoints work without it on a fresh IP, but sessions without it hit 429 far sooner.
relatedsearches is the touchiest endpoint — it 429s on the first call in a burst but is fine with ~5 s spacing.
On 429 (an HTML "Error 429" page, no Retry-After): back off 5 → 10 → 20 s with jitter, drop the cookie jar and re-warm, and if you have a proxy pool, build a new client on a new IP.
Cache explore results per (keywords, geo, timeframe): the tokens stay valid for a while, so three widgets cost one explore call, not three.

In production I keep ≥ 2 s between calls and rotate the proxy session every handful of requests. A keyword with 12 months of history, regions and related queries takes about 15 seconds end to end.

What you get

A keyword becomes rows like these:

{"type": "interestOverTime", "keyword": "ai agents", "geo": "US", "date": "2026-09-07", "value": 83, "isPartial": true}
{"type": "interestByRegion",  "keyword": "ai agents", "geo": "US", "geoCode": "US-CA", "geoName": "California", "value": 100}
{"type": "relatedQuery",      "keyword": "ai agents", "kind": "rising", "query": "ai agents framework", "value": "Breakout"}
{"type": "trending", "geo": "US", "rank": 1, "title": "…", "searchVolume": 500000, "startedAt": "2026-09-14T02:10:00Z", "relatedQueries": ["…"], "newsArticles": [...]}

That is enough to rebuild everything pytrends used to give you, plus the Trending Now feed it never had.

Source: https://github.com/m2kyungmin/apify-actors/tree/main/google-trends-scraper (the endpoint table at the top of trends_client.py is the living version of this post). Hosted, pay-per-row version: https://apify.com/kyungminlee/google-trends-scraper.

Scraping the Google Ads Transparency Center without a browser (and reading image ads with OCR)

Kyungmin Lee — Tue, 15 Sep 2026 04:39:19 +0000

The Google Ads Transparency Center shows every ad an advertiser has run on Search, YouTube, Display, Maps and Shopping. Marketers love it and hate it in equal measure: the data is all there, but the UI is an infinite scroll of cards, there is no export, and the text of image ads is baked into pixels.

This post walks through how the site actually loads its data, how to call those endpoints from Python with httpx, and how to get the copy out of image creatives with Tesseract. No Selenium, no Playwright.

1. The page is a thin shell over an RPC API

Open DevTools on an advertiser page and filter for rpc. Everything interesting is a POST to

https://adstransparency.google.com/anji/_/rpc/<Service>/<Method>?authuser=

with a form body of exactly one field, f.req, that contains JSON. Three methods do all the work:

Method	Purpose
`SearchService/SearchSuggestions`	advertiser search by name (returns advertiser IDs, regions, ad-count ranges, and the domains they advertise)
`SearchService/SearchCreatives`	the ad list for an advertiser, with filters and a cursor
`LookupService/GetCreativeById`	one ad's detail: regions, first/last shown, formats, video ID

The JSON is protobuf-as-JSON: keys are field numbers, not names. That is the only really annoying part, because you have to work out what "3": {"4": 2, "8": ["US"]} means by changing filters in the UI and diffing the requests. After an hour of that you end up with a payload builder like this:

FORMAT_BY_NAME = {"TEXT": 1, "IMAGE": 2, "VIDEO": 3}

def creatives_payload(advertiser_id, *, page_size=40, fmt=None, regions=None,
                      start=None, end=None, cursor=None, topic_id=1, viewer_geo="US"):
    f = {"1": advertiser_id}
    if fmt:      f["4"] = FORMAT_BY_NAME[fmt.upper()]
    if regions:  f["8"] = regions          # ISO alpha-2 codes
    if start:    f["6"] = start            # {"1": year, "2": month, "3": day}
    if end:      f["7"] = end
    payload = {"2": page_size, "3": f, "7": {"1": topic_id, "2": 0, "3": viewer_geo}}
    if cursor:
        payload["4"] = cursor              # opaque, returned as "2" in the previous response
    return payload

Calling it is a one-liner:

import json, httpx

RPC = "https://adstransparency.google.com/anji/_/rpc/"

async def rpc(client: httpx.AsyncClient, method: str, payload: dict) -> dict:
    r = await client.post(RPC + method + "?authuser=",
                          data={"f.req": json.dumps(payload, separators=(",", ":"))})
    r.raise_for_status()
    return r.json()

The response for SearchCreatives has the creatives in "1", the next cursor in "2", and the total count range in "4"/"5". Each creative carries its ID, format, a preview URL (image or a YouTube video ID) and the advertiser ID. Regions and run dates come from the per-creative lookup, so a full record costs two requests.

2. Blocking: it is the IP, not the headers

The endpoints do not need cookies or special headers beyond a normal User-Agent. What they do have is a per-IP budget. From a single datacenter IP you get somewhere between a few dozen and a few hundred RPC calls before every response becomes a 302 to google.com/sorry/…. There is no Retry-After; you just get bounced.

Two things fix it:

Rotate IPs every N calls. On Apify I create a proxy configuration once and derive a new session (= new IP) every 8 calls by changing the session-… part of the proxy URL. Do the derivation synchronously if your HTTP calls run in a worker thread — awaiting new_url() from a thread deadlocks.
Back off on 302/429/503 with exponential delay (2, 4, 8, 16 s) and rotate before retrying. Four attempts is enough; if all four are blocked, stop the run instead of burning money.

blocked = r.status_code in (301, 302) and "google.com/sorry" in r.headers.get("location", "")
if blocked or r.status_code in (429, 403, 503):
    await self._rotate()
    await asyncio.sleep(delay); delay *= 2
    continue

One honest caveat: when I ran seven advertiser exports in parallel from the same proxy pool, Google started bouncing every request for about an hour, including ones that had worked minutes earlier, while the container's own egress IP was still fine. Space your runs, and if your proxy pool is shared, keep a direct-connection fallback for the moments the whole pool is bounced.

3. Image ads: the copy is in the pixels

Text ads come back as text. Image ads come back as a preview PNG/JPEG, and for most brands that is 60–80 % of the creatives. The headline you want is inside the image.

Tesseract handles ad creatives surprisingly well because ad copy is large, high-contrast and horizontal. The whole OCR step:

from io import BytesIO
from PIL import Image, ImageOps
import pytesseract

async def ocr_image(client, url: str) -> str | None:
    r = await client.get(url, timeout=20)
    if r.status_code != 200:
        return None
    img = Image.open(BytesIO(r.content)).convert("L")     # greyscale
    if img.width < 600:                                    # upscale small creatives
        img = img.resize((img.width * 2, img.height * 2))
    img = ImageOps.autocontrast(img)
    text = pytesseract.image_to_string(img, lang="eng", config="--psm 6")
    text = " ".join(text.split())
    return text if len(text) >= 3 else None

--psm 6 ("assume a single uniform block of text") beats the default for ads; autocontrast and 2× upscaling recover most of the small legal lines. Install tesseract-ocr and tesseract-ocr-eng in your Dockerfile; on the apify/actor-python image that is one apt-get line.

Expect OCR to add roughly a second per image ad. In the hosted Actor I mark OCR-derived copy with adTextSource: "ocr" so downstream users can treat it as noisy, and I charge the OCR event only when Tesseract actually returned text.

4. Putting it together

The final record per ad looks like this:

{
  "adId": "CR1234…",
  "advertiserId": "AR0987…",
  "advertiserName": "Shopify",
  "format": "IMAGE",
  "adText": "Start selling online today. Free trial.",
  "adTextSource": "ocr",
  "imageUrl": "https://tpc.googlesyndication.com/…",
  "regions": ["US", "CA", "GB"],
  "firstShown": "2026-05-02",
  "lastShown": "2026-09-13",
  "detailUrl": "https://adstransparency.google.com/advertiser/AR0987…/creative/CR1234…"
}

From here it is ordinary data work: diff two weekly exports to see which creatives a competitor killed, group by regions to see where they are expanding, or feed adText to an LLM for messaging analysis.

Takeaways

The Transparency Center is an RPC API with protobuf-style JSON; three methods cover advertiser search, ad listing and ad detail.
The only real defence is per-IP rate limiting — rotate sessions every few calls and back off on 302 → /sorry.
Tesseract with --psm 6, greyscale and 2× upscaling turns image ads into searchable copy.

Source: https://github.com/m2kyungmin/apify-actors/tree/main/ads-transparency-scraper. Hosted, pay-per-ad version: https://apify.com/kyungminlee/ads-transparency-scraper.