DEV Community

Ken-Mutisya
Ken-Mutisya

Posted on

Scrape Google Trends Without an API Key (Including the Scraper Flag Google Hands You)

#ai #api #javascript #showdev

Google Trends has no official API, and most wrapper libraries rot within months. But the Trends site itself runs on a keyless JSON API that anyone can call, and it serves the exact numbers you see in the UI. Here is the full recipe, including one gotcha where Google quietly labels your session a scraper.

The two step flow

Trends works in two steps. First you call explore, which returns a list of widgets, one per chart on the page, each with a signed token:

GET https://trends.google.com/trends/api/explore
    ?hl=en-US&tz=0
    &req={"comparisonItem":[{"keyword":"web scraping","geo":"US","time":"today 12-m"}],"category":0,"property":""}
Enter fullscreen mode Exit fullscreen mode

Then you call a widget data endpoint with that widget's request and token:

GET https://trends.google.com/trends/api/widgetdata/multiline?hl=en-US&tz=0&req=<widget.request>&token=<widget.token>
Enter fullscreen mode Exit fullscreen mode

The widget kinds map to endpoints: TIMESERIES uses multiline, GEO_MAP uses comparedgeo, and both RELATED_QUERIES and RELATED_TOPICS use relatedsearches.

The cookie trick

Call explore cold and you get a 429. The API wants a NID cookie, and here is the counterintuitive part: you get it by requesting the public explore page first, and that page may itself respond 429 while still setting the cookie you need.

const res = await fetch('https://trends.google.com/trends/explore?geo=US&q=test');
// res.status may be 429. The Set-Cookie header is still there.
const cookies = res.headers.getSetCookie();
Enter fullscreen mode Exit fullscreen mode

Grab the cookie from the 429 response, retry explore, and everything works.

Strip the anti JSON prefix

Every Trends response starts with a junk line like )]}' to break naive JSON.parse calls. Drop everything up to the first newline:

const body = await res.text();
const data = JSON.parse(body.slice(body.indexOf('\n') + 1));
Enter fullscreen mode Exit fullscreen mode

The scraper flag

Here is the part I have not seen documented. Look inside the widget request object that explore returns to a keyless session:

"userConfig": { "userType": "USER_TYPE_SCRAPER" }
Enter fullscreen mode Exit fullscreen mode

Google knows. And it is mostly fine with it: the timeline, regions and related queries widgets all return full data anyway. But the related topics widget returns an empty list for scraper sessions. You cannot cheat it either, because the token is signed over the request. Edit userType to USER_TYPE_LEGIT_USER and the endpoint answers 401.

So the practical rule: related queries yes, related topics no. Plan your product around that.

What you get

The multiline response is a dated timeline of 0 to 100 values, comparedgeo gives a value per region, and relatedsearches gives two ranked lists, top and rising, where rising entries carry growth like +250% or Breakout. That rising list is the most useful output in the whole API: it shows what people are starting to search around a niche before it peaks.

Pace your calls (a second or two between keywords) and refresh the cookie when a 429 slips through, and the whole thing runs reliably from a datacenter IP with no proxy.

If you want it as a service

I packaged this recipe as a pay per use actor: keywords in, JSON rows out (interest over time, interest by region, related rising and top queries), with the first rows of every run free: https://apify.com/scrapemint/google-trends-scraper

Part of a growing set of keyless scrapers I build in public. The theme stays the same: find the JSON endpoint a site already uses, skip the browser, and the data costs almost nothing.

Top comments (0)