<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: cleanrows</title>
    <description>The latest articles on DEV Community by cleanrows (@cleanrowsdev).</description>
    <link>https://dev.to/cleanrowsdev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254049%2Fbb77e0e3-5e89-473a-b4a2-1ef96b27e0fd.png</url>
      <title>DEV Community: cleanrows</title>
      <link>https://dev.to/cleanrowsdev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/cleanrowsdev"/>
    <language>en</language>
    <item>
      <title>Why your Google Trends scraper breaks after 40 keywords (and what fixed it for me)</title>
      <dc:creator>cleanrows</dc:creator>
      <pubDate>Thu, 19 Nov 2026 14:46:12 +0000</pubDate>
      <link>https://dev.to/cleanrowsdev/why-your-google-trends-scraper-breaks-after-40-keywords-and-what-fixed-it-for-me-4j1j</link>
      <guid>https://dev.to/cleanrowsdev/why-your-google-trends-scraper-breaks-after-40-keywords-and-what-fixed-it-for-me-4j1j</guid>
      <description>&lt;p&gt;Google Trends has no official public API, so most people scrape it, either with pytrends or with a hosted scraper. It works fine for the first few dozen keywords. Then everything starts coming back as &lt;code&gt;429 Too Many Requests&lt;/code&gt;, and it stays that way.&lt;/p&gt;

&lt;p&gt;I hit exactly this while building a Google Trends tool, so here's what I measured.&lt;/p&gt;

&lt;h3&gt;
  
  
  The test
&lt;/h3&gt;

&lt;p&gt;I ran the same job, 50 everyday keywords with 12 months of interest over time each, in a few different setups:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Where the requests came from&lt;/th&gt;
&lt;th&gt;Keywords that returned data&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A cloud server, no proxy, one keyword at a time&lt;/td&gt;
&lt;td&gt;50 / 50 (2 needed a retry)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The same server, 3 keywords in parallel&lt;/td&gt;
&lt;td&gt;blocked almost immediately&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A shared datacenter proxy pool (5 IPs)&lt;/td&gt;
&lt;td&gt;44 / 50, then blocked&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Residential proxies&lt;/td&gt;
&lt;td&gt;50 / 50, zero 429s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The datacenter run is the interesting one. The first 44 keywords went through. After that, every request got a 429, including retries from fresh IPs in the same pool, and the pool was still blocked 13 minutes later. It isn't a per-request limit you can wait out. Google throttles the whole range of addresses.&lt;/p&gt;

&lt;h3&gt;
  
  
  What actually helped
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Residential IPs.&lt;/strong&gt; This was the biggest single fix. Google treats residential traffic like normal users, and the run went from 88% to 100% of keywords returned.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reuse the session.&lt;/strong&gt; The explore endpoint needs a cookie from the Trends page. Loading that page for every keyword almost triples your traffic. One cookie jar per proxy session cut my traffic by about 64%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retry on a new IP, not the same one.&lt;/strong&gt; A 429 on one IP rarely clears in seconds. Rotate the session, back off, and try again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never return half a result.&lt;/strong&gt; A keyword needs several requests: the timeline, regions and related queries. If one of them fails, retry the whole keyword instead of saving a row with a silently empty section.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  The data you get
&lt;/h3&gt;

&lt;p&gt;For each keyword:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;interest over time&lt;/strong&gt;, 0 to 100 per week or day;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;interest by country&lt;/strong&gt;, or by subregion, metro and city when you pick a country;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;related queries&lt;/strong&gt;, top and rising, including "Breakout" ones.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One honest caveat: in my tests Google returned related topics empty for these automated requests.&lt;/p&gt;

&lt;h3&gt;
  
  
  If you just want the data
&lt;/h3&gt;

&lt;p&gt;I packaged all of this as an Apify Actor: search "Google Trends Scraper &amp;amp; API (Reliable)" on the Apify Store, by cleanrows. It costs $2 per 1,000 keywords with the proxies included, and keywords that fail aren't charged. If you'd rather build your own, the four points above are what matter.&lt;/p&gt;

&lt;p&gt;Happy to answer questions about the endpoints in the comments.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>javascript</category>
      <category>seo</category>
      <category>datascience</category>
    </item>
  </channel>
</rss>
