DEV Community

member_924bafb1
member_924bafb1 Subscriber

Posted on

How to Get Google Trends Data in Python Without 429 Errors (a pytrends Alternative for 2026)

If you've ever written pytrends.build_payload(...) and seen TooManyRequestsError: The request failed: Google returned a response with code 429, you're not alone. pytrends was archived, Google's official Trends API is still invite-only alpha, and Trends now rate-limits automated traffic aggressively. This tutorial shows a dependable way to pull interest over time, interest by region and related queries for hundreds of keywords from Python, and explains why the usual fixes stop working.

Why pytrends gets 429s

The Google Trends front end calls a handful of internal endpoints:

  1. /trends/api/explore returns a list of "widgets", each with a short-lived token.
  2. /trends/api/widgetdata/multiline returns the timeline.
  3. /trends/api/widgetdata/comparedgeo returns regions.
  4. /trends/api/widgetdata/relatedsearches returns related queries and topics.

Every response starts with an anti-JSON-hijacking prefix ()]}'), and every call needs a valid NID cookie. When you send many requests from one IP, Google returns 429 or redirects you to /sorry/. The common advice of time.sleep(60) plus retries=3 works for ten keywords. It doesn't work for a thousand, and it doesn't work from cloud IPs at all, because those are often blocked from the very first request.

What actually works is:

  • a fresh session (cookie plus new tokens) after each block,
  • rotating egress IPs,
  • per-widget retries, so one failed section doesn't throw away the whole query,
  • honest status reporting, so you know which rows are complete.

You can build that yourself, or use a hosted Actor that already does it.

Option A: a hosted API (about 5 minutes)

Reliable Google Trends API on Apify handles the warmup, proxy rotation, retries and residential fallback. It charges $0.002 per successful query, and failed queries are free. In our 34-query gate runs on Apify, it returned 34 of 34 results in every proxy mode we tested.

Install the client:

pip install apify-client pandas
Enter fullscreen mode Exit fullscreen mode

Run a comparison:

import os
import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("eligible_fame/reliable-google-trends").call(run_input={
    "searchTerms": [
        "chatgpt, claude, gemini",     # one query comparing 3 terms
        "air fryer",                   # a second query
    ],
    "geo": "US",
    "timeframe": "today 12-m",
    "interestOverTime": True,
    "interestByRegion": True,
    "relatedQueries": True,
})

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for r in rows:
    print(r["keywords"], r["status"], r["errors"])
Enter fullscreen mode Exit fullscreen mode

Turn the timeline into a DataFrame

def timeline_df(row):
    recs = [{"date": p["time"], **p["values"]} for p in row["interest_over_time"]]
    return pd.DataFrame(recs).set_index("date")

df = timeline_df(rows[0])
print(df.tail())
df.plot(title="ChatGPT vs Claude vs Gemini, US, 12 months")
Enter fullscreen mode Exit fullscreen mode

Rising queries (breakouts)

for kw, rq in rows[1]["related_queries"].items():
    for q in rq["rising"][:10]:
        print(kw, "->", q["query"], q["formatted_value"])
Enter fullscreen mode Exit fullscreen mode

Bulk keyword lists

Each searchTerms entry is its own query, so you can pass 500 entries in one run. To set different options for each query, use customQueries:

run_input = {"customQueries": [
    {"keywords": "bitcoin, ethereum", "geo": "GB", "timeframe": "today 5-y", "property": "news"},
    {"keywords": "lofi music", "property": "youtube"},
]}
Enter fullscreen mode Exit fullscreen mode

Every run also writes a RUN_SUMMARY record to the key-value store with the success rate and block counts:

summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("RUN_SUMMARY")["value"]
print(summary)
Enter fullscreen mode Exit fullscreen mode

Option B: roll your own (what you need to handle)

If you'd rather run it yourself, here's the minimum:

import httpx, json, time, random

def warm(client):
    # A headers-only request is enough to obtain an NID cookie
    with client.stream("GET", "https://trends.google.com/trends/explore?geo=US") as r:
        pass

def explore(client, kw, geo="US", tf="today 12-m"):
    req = {"comparisonItem": [{"keyword": kw, "geo": geo, "time": tf}],
           "category": 0, "property": ""}
    r = client.get("https://trends.google.com/trends/api/explore",
                   params={"hl": "en-US", "tz": 0, "req": json.dumps(req)})
    if r.status_code == 429:
        raise RuntimeError("blocked")
    return json.loads(r.text[r.text.index("{"):])["widgets"]
Enter fullscreen mode Exit fullscreen mode

From here you still need to: call each widget endpoint with its token; rotate the session and the IP whenever you get a 429 or /sorry/; back off with jitter; and treat related topics as optional (Google often withholds them from sessions it labels USER_TYPE_SCRAPER). In practice the proxy bill and the maintenance cost far more than the code.

Gotchas that apply either way

  • Values are relative (0–100), not search volume. Only compare keywords that were in the same query.
  • More than 5 keywords? Use a shared anchor term in every batch (for example "weather") and rescale each batch against it.
  • Timeframe strings follow the Trends URL format: now 7-d, today 3-m, today 5-y, all, 2024-01-01 2024-06-30.
  • Last data point is often is_partial: true. Drop it before you fit a forecast.

Schedule it

In the Apify console, open the Actor, save your input as a Task, and add a Schedule (for example, daily at 06:00). Then use a webhook or the n8n/Make/Zapier Apify app to push new rows into Google Sheets or your warehouse.

Wrap-up

pytrends taught a generation of analysts to love Trends data, but the 429 wall is real. If you need a few queries a week, a careful DIY script with backoff can be enough. If you need hundreds of keywords on a schedule, without babysitting proxies, try Reliable Google Trends API. It's pay-per-result, and failed queries are free.

Top comments (0)