Google Trends 429s your first request. The fix is 106 bytes.

Vladimir Triphonov — Fri, 07 Aug 2026 14:15:08 +0000

If you have ever tried to pull data from Google Trends programmatically, you have met this:

HTTP 429 Too Many Requests

Not after a hundred requests. On the first one. From a clean IP that has never touched Google Trends before.

That looks like a block, so most people reach for proxies, back off aggressively, add random sleeps — and it still fails. Because it is not a block. It is a missing cookie.

Here is what is actually going on, measured rather than guessed. The whole working flow, up front:

┌─ per keyword, on its own fresh IP ──────────────────────────────┐
│                                                                 │
│  GET /_/TrendsUi/data/batchexecute  → 405, sets NID      106 B  │
│  GET /trends/api/explore            → 200, 4 tokens        5 KB │
│                                                                 │
│      ├─ widgetdata/multiline        → interest over time        │
│      ├─ widgetdata/comparedgeo      → interest by region        │
│      └─ widgetdata/relatedsearches  → top + rising queries      │
│         └─ ~800 ms apart, or the limiter cuts you off           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Every element of that has a reason behind it, and most of them cost me a failed build to learn.

Step 1: reproduce it

A bare request to the Trends API, no session, no cookies:

const url = 'https://trends.google.com/trends/api/explore?hl=en-US&tz=-360&req='
  + encodeURIComponent(JSON.stringify({
      comparisonItem: [{ keyword: 'bitcoin', geo: 'US', time: 'today 12-m' }],
      category: 0, property: '',
    }));

const res = await fetch(url);
console.log(res.status); // 429

Every time. Fresh IP, fresh process, does not matter.

Step 2: the cookie

Google Trends issues an NID cookie on its own pages, and /trends/api/* refuses to serve anyone who does not present one. So warm the session first:

const jar = new Map();

async function call(url) {
  const res = await fetch(url, {
    headers: {
      'user-agent': UA,
      'accept-language': 'en-US,en;q=0.9',
      ...(jar.size ? { cookie: [...jar].map(([k, v]) => `${k}=${v}`).join('; ') } : {}),
    },
  });
  for (const raw of res.headers.getSetCookie?.() ?? []) {
    const [pair] = raw.split(';');
    const i = pair.indexOf('=');
    if (i > 0) jar.set(pair.slice(0, i).trim(), pair.slice(i + 1).trim());
  }
  return res;
}

await call('https://trends.google.com/trends/');   // 200, sets NID
const res = await call(url);                        // 200 ✅

Measured on a clean IP:

explore (no cookie)  → 429       1,697 bytes
GET /trends/         → 200     822,038 bytes   ← sets NID
explore (with cookie)→ 200       5,583 bytes   ✅

The response is prefixed with )]}' before the JSON, which you have to strip:

const json = JSON.parse(text.replace(/^\)\]\}'?,?\s*/, ''));

You get back tokens for four widgets: TIMESERIES, GEO_MAP, RELATED_TOPICS, RELATED_QUERIES. Each one is a second call to /trends/api/widgetdata/... carrying that token.

Why so many tools are broken right now

pytrends, the library most tutorials still recommend, was archived in April 2025. Its session bootstrap no longer acquires the cookie Google's current flow expects, so the first data call returns 429 — not because you are throttled, but because the handshake is stale.

If you are debugging a Trends scraper that "suddenly stopped working," this is usually why.

Step 3: the part that surprised me — warming costs 822 KB

Look at that warm-up line again: 822 KB to fetch a cookie. On residential proxies at $8/GB that is $0.0066 every time you rotate an IP. And as we will see in a moment, you rotate constantly. That single line was going to dominate my costs.

So I went looking for the cheapest URL on the domain that still hands out an NID:

URL	Status	Bytes	Sets NID
`/trends/`	200	822,057	✅
`/trends/explore`	429	1,701	✅
`/_/TrendsUi/data/batchexecute`	405	106	✅
`/robots.txt`	200	93	❌

batchexecute rejects a GET with 405 Method Not Allowed — and sets the cookie anyway.

The obvious question: does a cookie handed out by a rejected request actually authenticate API calls? I assumed no. I was wrong:

warm-up: batchexecute (106 B)  → 8/8 keywords succeeded, 14 KB/keyword
warm-up: /trends/    (822 KB)  → 8/8 keywords succeeded, 836 KB/keyword

Identical success rate, 58x less traffic. Cost per keyword dropped from $0.00623 to $0.00011.

Keep the full page as a fallback in case Google stops setting cookies on rejected requests:

const WARMUP_URLS = [
  'https://trends.google.com/_/TrendsUi/data/batchexecute',  // 106 bytes
  'https://trends.google.com/trends/',                        // 822 KB fallback
];

for (const url of WARMUP_URLS) {
  await call(url);
  if (jar.has('NID')) break;   // the status does not matter, only the cookie
}

Step 4: the cookie is necessary but not sufficient

Warm the cookie, run ten keywords in a row from one IP, and watch:

Success rate: 0/10 = 0.0%

All 429, despite correct warm-up and re-warming on every failure. My earlier single-keyword test had only passed because the IP was fresh.

The limit is bound to the IP, not the session. An address survives roughly 10-15 requests and then it is done, whatever cookies you present. One keyword with all four widgets costs about five requests. So you get two, maybe three keywords per address.

Rotating the IP per keyword is therefore mandatory, not an optimisation.

Step 5: the mistake that cost me every result

Here is the bug I want you to avoid, because it is subtle and it silently destroys your success rate.

My first implementation looked like this:

for (let attempt = 1; attempt <= maxAttempts; attempt++) {
  try {
    await warmup();
    const widgets = await explore(keyword);
    const rows = [
      ...await fetchTimeline(widgets),
      ...await fetchGeo(widgets),      // ← 429 here
      ...await fetchRelated(widgets),
    ];
    return rows;
  } catch {
    // retry everything from scratch
  }
}

Reasonable-looking. And completely wrong.

One keyword needs ~5 requests. An IP tolerates a handful. So getting cut off mid-keyword is the normal case, not the exception. When fetchGeo threw, this code discarded the timeline data it had already successfully fetched — data it had already spent requests to obtain — and then re-spent that budget fetching the same thing again on the next attempt. Every attempt burned its IP at the same point and threw away the same good data.

The fix is to accumulate across attempts and only retry what is still missing:

const collected = new Map();

while (attempt < maxAttempts && collected.size < wanted.length) {
  attempt++;
  const missing = wanted.filter((t) => !collected.has(t));
  const session = await newSession();   // fresh IP

  try {
    await warmup(session);
    const widgets = await explore(session, keyword);

    for (const type of missing) {
      try {
        collected.set(type, await fetchWidget(session, widgets[type]));
      } catch (err) {
        // this dataset alone failed — keep the others, retry it on the next IP
      }
    }
  } catch { /* whole session died, next attempt gets a new IP */ }
}

Same target, same rate limits, same number of attempts. The difference:

before: 0 rows,   0% success
after:  154 rows, 100% success

That is on a rate-limited IP with no proxy at all. With residential rotation it held at 10/10 keywords, every one on the first attempt.

The general lesson has nothing to do with Google: when a unit of work costs more requests than your budget per IP, partial progress is the thing you must not throw away. Retry the gap, not the whole job.

Step 6: pace your calls, or the limiter finds you anyway

A day after everything worked, it stopped. Same code, same proxies, 429 on widgetdata from five different IP sources — my own address, two fresh residential IPs, two datacenter IPs. explore still returned 200 and handed over widget tokens; only the data calls failed.

My first conclusion was that Google had killed the legacy endpoint. That was wrong, and it is worth saying plainly because it is the kind of wrong conclusion that sends you rewriting a working client for two days.

What actually happened: Google tightened the limits, and my client was firing the four widget calls back-to-back with no gap. That pattern had been getting away with it. Now it wasn't.

The tell was in my own two test scripts. The one that failed hit the widgets immediately. The one that passed had an await sleep(800) between them — I had put it there for readability of the log output, not for any principled reason. Same target, same proxies, minutes apart:

no pacing:    TIMESERIES 429
800ms pacing: TIMESERIES 200 · GEO_MAP 200 · RELATED_TOPICS 200 · RELATED_QUERIES 200

Adding that one sleep, plus a referer header matching what the real UI sends, took the actor from intermittent total failures back to full results on every run.

let first = true;
for (const type of missing) {
  if (!first) await sleep(800);   // ← this line
  first = false;
  // ...fetch widget
}

Three seconds per keyword. Cheap insurance.

A debugging note that cost me half a day

When this broke, my logs said exactly this:

[1/1] "bitcoin" failed after 4 attempts: fetch failed

fetch failed is undici's generic transport error. The actual reason — the status code, the socket error, whatever it was — lives in err.cause, and I was only logging err.message. So I went hunting through Google's protocol for a problem that a single log line would have pointed at.

const describe = (err) => {
  const cause = err.cause?.code ?? err.cause?.message;
  return cause ? `${err.message} (${cause})` : err.message;
};

If you write anything on top of fetch, log cause. It is the difference between a diagnosable failure and a guessing game.

Gotchas worth knowing

rising related queries are co-searches, not synonyms. For yoga mat the top list is sensible — black yoga mat, yoga mat bag, travel yoga mat. The rising list returned robot vacuum and wireless earbuds. That is not a bug: rising shows what is growing fastest among people who also searched your term, which during a sale season means unrelated products. top for keyword research, rising for momentum. formattedValue: "Breakout" means growth above 5000%.

Values are relative and scaled per request. The 0-100 numbers are normalised within one keyword's own results. You cannot compare 73 for one keyword against 73 for another.

Empty results are a real answer. If Google has too little volume for a term it returns nothing. Do not retry that forever.

Google labels you. The widget request payload contains "userConfig": {"userType": "USER_TYPE_SCRAPER"}. It knows. It serves the data anyway.

Summary

/trends/api/* returns 429 without a warmed NID cookie. Warm the session first.
pytrends is archived and no longer does this — that is why so much code broke.
Warm via /_/TrendsUi/data/batchexecute (106 bytes) instead of /trends/ (822 KB). It returns 405 and sets the cookie anyway.
The rate limit follows the IP, not the session. Rotate per keyword.
Accumulate partial results across retries. This one change took me from 0% to 100%.
Space the widget calls ~800ms apart and send a referer. Back-to-back requests trip the limiter even when everything else is right.
Log err.cause. fetch failed on its own tells you nothing.

And one meta-lesson. Twice in this project I jumped to "the platform changed the protocol" when the real answer was in my own code — first when partial results were being discarded, then when the calls were not paced. Both times the evidence was already sitting in a diff between two of my own test runs. Before you conclude the target changed, check what changed on your side.

I packaged all of this as a Google Trends Scraper on Apify if you would rather not maintain it yourself — it returns interest over time, interest by region, and top/rising related queries as flat rows. But everything above is what it does under the hood, and it is enough to build your own.

DEV Community: Vladimir Triphonov