DEV Community

Kyungmin Lee
Kyungmin Lee

Posted on Fully Autonomous

pytrends is dead. Here is how Google Trends actually loads data in 2026 (including Trending Now)

#python #webscraping #googletrends #data

If you have tried pytrends recently you know the story: TooManyRequestsError on the first call, dailytrends returning 404, and an unmaintained repo. Google redesigned Trends in 2024, moved the "Trending Now" feed to a different backend, and started returning 429 to anything that looks like a script.

The good news is that the site is still a plain JSON API underneath. Here is the map I put together while building a browserless client with nothing but httpx.

The two-step token dance (explore → widgetdata)

Every Trends chart is loaded in two requests:

  1. GET /trends/api/explore?hl=en-US&tz=-540&req=<JSON> — you describe what you want (keywords, geo, timeframe, property) and Google answers with a list of widgets, each carrying a token and a request object.
  2. GET /trends/api/widgetdata/<kind>?token=<token>&req=<JSON(widget.request)> — you send the widget's own request back with its token and get the data.

The req for explore looks like this:

import json

def explore_req(keywords, geo="", timeframe="today 12-m", category=0, prop=""):
    return json.dumps({
        "comparisonItem": [{"keyword": k, "geo": geo, "time": timeframe} for k in keywords],
        "category": category,
        "property": prop,          # "" web, "images", "news", "youtube", "froogle"
    }, separators=(",", ":"))
Enter fullscreen mode Exit fullscreen mode

Widget IDs tell you which endpoint to call:

Widget id Endpoint What you get
TIMESERIES /trends/api/widgetdata/multiline timelineData[] with time, formattedTime, value[], isPartial
GEO_MAP (or GEO_MAP_0 … per keyword) /trends/api/widgetdata/comparedgeo geoMapData[] with geoCode, geoName, value[]
RELATED_QUERIES_0 /trends/api/widgetdata/relatedsearches two rankedList entries: TOP then RISING
RELATED_TOPICS_0 same endpoint, keywordType: ENTITY topics — empty for logged-out sessions (see below)

Every response starts with the anti-JSON-hijacking prefix )]}'\n — strip it before json.loads.

Two details that cost me an evening:

  • The token is bound to the exact request JSON you send back. Changing userConfig → 401. The only fields you may edit are resolution (COUNTRY/REGION/CITY/DMA) and includeLowVolumeGeos on the geo widget.
  • Explore stamps userConfig.userType. Logged-out HTTP sessions get USER_TYPE_SCRAPER; a real Google login gets USER_TYPE_LEGIT_USER. Time series, geo and related queries work fine for scrapers, but related topics return {"rankedList": []}. If you need topics you need your own cookies — I don't ship that.

Where Trending Now went

/trends/api/dailytrends and /trends/api/realtimetrends are gone (404). The new Trending Now page is a Google "batchexecute" app:

POST https://trends.google.com/_/TrendsUi/data/batchexecute?rpcids=i0OFE&source-path=/trending&hl=en-US&rt=c
Content-Type: application/x-www-form-urlencoded
f.req=[[["i0OFE","[null,null,\"US\",0,\"en-US\",24]",null,"generic"]]]
Enter fullscreen mode Exit fullscreen mode

The inner argument list is [null, null, geo, 0, hl, hours]. The response is chunked: )]}'\n\n<length>\n[["wrb.fr","i0OFE","<json string>",...]] — read the length prefix, parse the chunk, then parse the JSON string inside it. Each trend item is a positional array:

[title, null, geo, [start_ts], [end_ts] | null, null, search_volume, null,
 pct_increase, [breakdown_keywords], [category_ids], [[article_id, lang, geo], ...],
 normalized_title]
Enter fullscreen mode Exit fullscreen mode

I keep the mapping in one parse_trending() function and treat it as "will break someday". Note that the category argument in the request does not filter server-side (I got 566 vs 567 items with and without it), so filter on category_ids client-side.

There is also /trending/rss?geo=US, a plain RSS feed with ht:approx_traffic and ht:news_item, which makes a nice fallback when the batchexecute format changes.

Staying under the 429 radar

Google Trends rate-limits per IP and, more subtly, per session:

  • Warm up first: GET https://trends.google.com/ (302 → /trends/) sets the NID cookie. Endpoints work without it on a fresh IP, but sessions without it hit 429 far sooner.
  • relatedsearches is the touchiest endpoint — it 429s on the first call in a burst but is fine with ~5 s spacing.
  • On 429 (an HTML "Error 429" page, no Retry-After): back off 5 → 10 → 20 s with jitter, drop the cookie jar and re-warm, and if you have a proxy pool, build a new client on a new IP.
  • Cache explore results per (keywords, geo, timeframe): the tokens stay valid for a while, so three widgets cost one explore call, not three.

In production I keep ≥ 2 s between calls and rotate the proxy session every handful of requests. A keyword with 12 months of history, regions and related queries takes about 15 seconds end to end.

What you get

A keyword becomes rows like these:

{"type": "interestOverTime", "keyword": "ai agents", "geo": "US", "date": "2026-09-07", "value": 83, "isPartial": true}
{"type": "interestByRegion",  "keyword": "ai agents", "geo": "US", "geoCode": "US-CA", "geoName": "California", "value": 100}
{"type": "relatedQuery",      "keyword": "ai agents", "kind": "rising", "query": "ai agents framework", "value": "Breakout"}
{"type": "trending", "geo": "US", "rank": 1, "title": "…", "searchVolume": 500000, "startedAt": "2026-09-14T02:10:00Z", "relatedQueries": ["…"], "newsArticles": [...]}
Enter fullscreen mode Exit fullscreen mode

That is enough to rebuild everything pytrends used to give you, plus the Trending Now feed it never had.

Source: https://github.com/m2kyungmin/apify-actors/tree/main/google-trends-scraper (the endpoint table at the top of trends_client.py is the living version of this post). Hosted, pay-per-row version: https://apify.com/kyungminlee/google-trends-scraper.

Top comments (0)