Google Trends ties its data tokens to your IP and it broke my scraper in a way I didn't expect

#api #webscraping #seo #showdev

I just shipped a Google Trends scraper as an Apify Actor (https://apify.com/swiftscrape/google-trends-fast-scraper), and the most interesting part wasn't building it — it was a blocking bug that only showed up in production, on the platform, that I couldn't reproduce locally no matter what I tried.

If you've ever tried to scrape Google Trends and had it work on your laptop but mysteriously fail behind proxies, this one's for you. The root cause is non-obvious and I couldn't find it documented anywhere, so here's the whole thing.

Why build another Google Trends scraper?

Google Trends has no official API. There are scrapers out there, but the most-used one on the market is slow (runs take minutes), it charges you even when it returns nothing, and its multi-term comparison breaks when called from AI agents. Its reviews are full of "ran for 20 minutes, returned nothing, cost me money."

That's a wedge. Same data, but: fast, and you only pay for terms that actually return data — failed, blocked or empty terms are free. If a scraper is unreliable, that's the developer's problem, not the user's bill.

The architecture (no browser)

Google Trends' UI is backed by internal JSON endpoints. You don't need a headless browser — you can talk to them directly:

GET /trends/api/explore with your terms → returns widget tokens (one for the timeline, and related-search widgets per term).
GET /trends/api/widgetdata/multiline?token=… → interest over time.
GET /trends/api/widgetdata/relatedsearches?token=… → related queries.

No Chrome, no Puppeteer. Direct HTTP is why it runs in ~1.5s locally instead of minutes.

The bug: it worked locally, failed on the platform

Locally (running from my home IP, no proxy): flawless. Timeline, related queries, everything.

On Apify (behind a datacenter proxy): explore succeeded, timeline succeeded... but related queries silently failed for every term. Then when I switched to residential proxies to "fix" it, it got worse — now even the timeline failed with blocked.

That combination made no sense to me at first:

	explore	timeline	related
Home IP (no proxy)	✅	✅	✅
Datacenter proxy	✅	✅	❌
Residential proxy	✅	❌	❌

If residential IPs are "better," why did they break more?

The root cause: widget tokens are bound to the IP that requested them

Here's the thing nobody tells you: the widget tokens Google hands you in step 1 are tied to the IP address that made the explore call. When you then call the multiline / relatedsearches endpoints with those tokens, Google checks that the request comes from the same IP. Different IP → token rejected → blocked.

Now the table makes sense:

No proxy: every request comes from one IP (mine). Everything validates.
Datacenter proxy: small IP pool, so explore and timeline sometimes landed on the same IP by luck — but the parallel related calls hit fresh IPs and got rejected.
Residential proxy: huge rotating IP pool, so even the very next request after explore came from a different IP. Even the timeline broke.

The "better" proxy broke things precisely because it rotated IPs more aggressively.

The fix: pin one sticky session per run

The fix is to force every request in a single run through the same IP by pinning a proxy session:

const proxy = await Actor.createProxyConfiguration({ groups: ['RESIDENTIAL'] });
const sessionId = `gt${Date.now()}`;

// Every request in this run reuses the same IP:
const proxyUrl = await proxy.newUrl(sessionId);

One sticky session → explore, timeline, and related all come from the same IP → tokens stay valid. Result after the fix, on the platform:

chatgpt: 53 weekly data points, 16 top related queries, 3 rising
gemini: 53 data points, 25 top, 18 rising
~16s end to end for a 2-term comparison

The lesson generalizes beyond Google Trends: any two-step "get a token, then use the token" API can be IP-bound. If step 2 fails behind a proxy but step 1 succeeds, suspect IP-pinned tokens before you suspect rate limiting.

The pricing angle

Because I only charge for terms that return validated data, I had to make the billing code honest: a term with no search volume comes back from Google as all-zeros with a hasData: false flag, not an empty array. If you don't check hasData, you'll "successfully" return a series of zeros and charge for a term that had no data. That check is the difference between "no result = no charge" being a slogan and being true.

Takeaways

Google Trends widget tokens are IP-bound — pin a sticky proxy session for the whole run.
Test scrapers on the actual deployment target, not just locally. This bug was invisible on my home IP.
Residential ≠ automatically better. Rotation can break session-bound flows.
If you charge per result, respect the provider's hasData flags or your "free on failure" promise is a lie.

The scraper is live here if you want to try it (pay-per-result, failed terms are free): Fast Google Trends Scraper on Apify.

Happy to answer questions about the internal endpoints or the token behavior in the comments.